The Best Ephemeral CI Runner Platforms in 2026

A CI runner is not a build machine. That's the marketing description. What a long-lived, shared runner actually is: a writable filesystem, a warm package cache, a Docker socket, cached registry credentials, an SSH agent, a kubeconfig, and a cloud instance profile — all sitting in the environment of a process whose entire job is to execute a script chosen by whoever opened the pull request. Every job is a stranger's Makefile with your deploy keys in the environment. The supply-chain incidents of the last few years that made everyone update their runbooks mostly rode that exact pattern: not an exotic zero-day, just a build step doing something ordinary in a place that remembered things.

I'm Ajay; I built PandaStack. This post is a roundup of the ways to stop doing that — the platforms and patterns for ephemeral CI runners, where a job gets a fresh machine, runs once, and is destroyed. I'll set out the criteria that actually separate them (isolation boundary, provisioning latency, single-use versus 'reset', cache strategy, secrets and OIDC, egress control, cost model, operational burden), walk the field option by option, and then spend real time on the two objections that decide most of these projects: the cache problem, and how you hold a credential in a job you don't trust. Nobody argues about whether ephemeral is better any more. They argue about what it costs.

What you are actually buying when you buy a CI runner

Start with the boring inventory, because the argument for ephemerality is entirely contained in it. When a job finishes on a persistent runner, the process exits. Almost nothing else does. The next job — possibly from a different repository, possibly from a fork opened by an account created eleven minutes ago — starts on top of everything the previous one left behind.

• Package caches, which are directories a job can write to — `~/.npm`, `~/.cache/pip`, `~/.m2`, `~/.gradle`, `ccache`. A job that writes a poisoned artifact into one has published a dependency to every future job on that machine, with no registry involved and no lockfile change to review.
• Credentials at rest, accumulated as a side effect of convenience — `~/.docker/config.json` after a registry login, an `~/.npmrc` with a publish token, a `~/.kube/config`, a leftover `~/.aws/credentials`. None were granted to the current job. They're just there.
• The checkout itself. `_work/<repo>` is reused between runs on most self-hosted setups and is by construction attacker-writable — a `.git/hooks/post-checkout` planted by one job runs during the next job's clone, before any of your pipeline logic executes.
• The ambient machine: installed packages, a shim earlier in `$PATH` than the real binary, a systemd timer, a crontab, an `/etc/hosts` entry pointing your registry hostname somewhere interesting.
• The host's identity. A cloud VM's metadata service hands a role to anything that can make an HTTP request from that box, and the build script can make an HTTP request from that box. That's the whole attack; it requires a bug in nothing.
• The other jobs running right now — readable `/proc` entries and `/tmp` files, and, if a Docker socket is mounted (it usually is), a privileged container away from the host's disk.

A persistent CI runner isn't a machine that builds your code. It's a machine that accumulates trust, and hands all of it to whoever most recently opened a pull request.

Ephemeral, precisely: single-use, not "reset"

Two things get called ephemeral and only one of them is. Single-use means the runner accepts exactly one job and then the machine hosting it stops existing. Reset means a long-lived machine runs a teardown routine between jobs — much better than nothing, and much worse than it sounds, because its correctness depends on cleanup code running after the untrusted code, on a box the untrusted code just had root on. You can `git clean -xfd`; it will not remove the crontab. Check this distinction first in any vendor's docs, because 'a clean environment for every job' is true of both.

# ---------- The anti-pattern: one long-lived runner, registered in March ----------
# You did this because hosted runners were slow for your monorepo, and it worked,
# and then it kept working, which is the dangerous part.
./config.sh --url https://github.com/acme/api --token "$REG_TOKEN" --unattended
sudo ./svc.sh install && sudo ./svc.sh start

# Everything below survives every job, in both directions -- what the last job
# left behind, and what the next job gets to read:
#   ~/.npmrc  ~/.docker/config.json  ~/.aws/credentials    <- credentials at rest
#   ~/.npm  ~/.cache/pip  ~/.m2  ~/.gradle  ~/.cache/go-build  <- writable caches
#   _work/acme/api/         (the checkout, REUSED, with its .git/hooks)
#   $PATH, /usr/local/bin, crontabs, systemd units, /etc/hosts
#   169.254.169.254, which hands out a cloud role and never asks who is calling


# ---------- Single-use: one job, one machine, one short-lived token ----------
# The registration token is minted per job by the orchestrator and expires on its
# own. It is not a secret you store; it is a secret you generate and then lose.
REG_TOKEN="$(gh api -X POST /repos/acme/api/actions/runners/registration-token --jq .token)"

# --ephemeral is the load-bearing flag: the runner takes exactly ONE job and then
# deregisters. Without it, run.sh loops forever and you are back on the
# anti-pattern with extra steps.
./config.sh --url https://github.com/acme/api --token "$REG_TOKEN" \
  --ephemeral --unattended --disableupdate --labels "microvm,run-$RUN_ID"

./run.sh                      # blocks, runs one job, exits

# ...and then the ORCHESTRATOR destroys the machine. Not `git clean`. Not a
# teardown script running as the same user the build just had. The disk, the page
# cache, and the guest kernel stop existing -- the only teardown whose
# correctness does not depend on the build's cooperation.

# Necessary, not sufficient: --ephemeral scopes the RUNNER REGISTRATION to one
# job. It says nothing about the machine underneath. Ten ephemeral runner
# processes on one four-month-old VM still share a kernel, a filesystem, and an
# instance profile -- which is the single most common misconfiguration here.

The criteria that separate these platforms

Every option below will check out your repo and run your test suite; that baseline tells you nothing. Here is what actually differs. Work out which two or three are forcing your hand before you compare anything, because the answer flips completely depending on which you weight.

• Isolation boundary — shared kernel or own kernel. A container-per-job is a namespaced process: strong against accidents, and a kernel-level bug in the shared syscall surface is a host compromise. A microVM-per-job is hardware-virtualized with its own guest kernel, so the same bug gets you a throwaway machine. The full ladder is in /blog/code-isolation-hierarchy.
• Provisioning latency, and what it does to queue time — not a vanity metric. If a fresh environment takes ninety seconds you will pool, and pooling is how ephemeral quietly becomes reset. Fast provisioning is what makes the secure design the convenient one.
• Truly single-use versus reset — ask the direct question: does the machine hosting the runner outlive the job, yes or no?
• Cache strategy — the central tension of this whole topic. Ephemeral runners lose warm caches, and the obvious fix reintroduces exactly the cross-job write surface you paid to remove.
• Secrets handling and OIDC — can a job get a short-lived, narrowly-scoped credential minted per run instead of a long-lived secret in the environment? And what happens on a fork pull request, which is a different problem with different defaults?
• Egress control — can you default-deny outbound traffic and allowlist the registry, and is that policy enforced somewhere the job can't reach? A build script that can reach the internet can post your source tree to it; more in /blog/controlling-network-egress-untrusted-code.
• Cost model — the shape matters more than the rate. Bursty CI on always-on capacity means paying for the trough, and per-minute rounding is brutal across a matrix of forty two-minute jobs.
• Operational burden — somebody scales runners to demand, reaps orphans, rolls images, and debugs the pool at 2am. Every self-hosted option here is a bet on how much of that you want to own.

The field, option by option

GitHub-hosted runners

The default, and the one most teams should stay on far longer than they do. Hosted runners are ephemeral by construction — a fresh VM per job, destroyed afterward — so nothing in the inventory above applies, because there is nothing to accumulate into, and there's no control plane, no pool to scale, no orphan reaper to write. What pushes people off is some mix of cost at volume, machine size, and control: billed minutes add up across a wide matrix, the largest hosted shapes may not suit a heavy compile, and the job runs in GitHub's network rather than yours, so reaching a private database means opening a path inward and egress policy isn't really yours to set. If your reason is 'we need this inside our VPC' or 'we need a GPU', that's real. If it's 'builds feel slow', measure first — that's usually a five-minute caching fix, and leaving costs you a control plane forever. Verify current machine shapes and minutes pricing against GitHub's docs; it's the fastest-moving line item here.

• Isolation and lifecycle: fresh VM per job, destroyed after; genuinely single-use, operated by GitHub.
• Best fit: almost everyone, until a specific constraint — VPC access, machine size, egress policy, or volume economics — actually bites.

Self-hosted runners on VMs, done naively (the anti-pattern)

This gets a slot precisely because it isn't a product — it's what happens by default, and it's the most common self-hosted CI topology in the world. Someone needed a bigger machine or VPC access, spun up an instance, ran `config.sh`, installed it as a service, and moved on. It works immediately, it's fast because everything is already warm, and it costs one instance. Nobody makes a bad decision here; they make a reasonable one and never revisit it, and eighteen months later that box has a Docker socket, three sets of registry credentials, and a kubeconfig with more permissions than anyone remembers granting. The failure mode isn't sophisticated: any job there can read the credentials sitting on it, query the metadata service for the instance role, write a shim into `$PATH` for the next job, or plant a git hook in the reused checkout. Attach that runner to a repo that builds fork pull requests and you've published root on your build infrastructure as a feature — and the internal case is no better, because a compromised dependency's postinstall script does not care that the PR author was an employee.

• Isolation and lifecycle: none and none. Shared kernel, filesystem, and instance identity; state persists indefinitely across every job and every repo attached to it.
• Best fit: honestly nothing — but if you can't leave today, the mitigations that pay off fastest are `--ephemeral` plus a fresh VM per job from an image, never attaching the runner to fork PRs, blocking the metadata service at the firewall, and removing the Docker socket.

actions-runner-controller and Kubernetes-based ephemeral runners

If you already run Kubernetes, this is the natural landing spot and a genuinely large upgrade. actions-runner-controller — and the runner-scale-set model it moved to — provisions a pod per job, registers an ephemeral runner inside it, runs one job, and deletes the pod. You get autoscaling driven by queue depth, declarative runner images, and the observability you already built for pods, with a controller in a cluster you already operate rather than a new system to learn.

The caveat follows every container-isolation story. A pod per job resets the filesystem and the process namespace, but the kernel is shared with every other pod on that node, including the jobs running right now. For internal repos where contributors are employees and the concern is leftover state, that's a fine boundary and the reset is real. For fork pull requests, or model-generated build steps, you're betting the container boundary against arbitrary code. The usual hardening is worth doing regardless — no privileged pods, no Docker socket, a daemonless image builder like Buildah or Kaniko, seccomp and a restrictive Pod Security Standard, NetworkPolicy, and a dedicated node pool with no cluster-credential access. If you want the stronger boundary without leaving Kubernetes, a VM-backed runtime class (Kata over Firecracker or Cloud Hypervisor) is the bridge.

• Isolation and lifecycle: pod per job, deleted after — truly single-use at the pod level, shared kernel at the node level.
• Best fit: Kubernetes-native teams with a mostly-trusted contributor set who want queue-driven autoscaling and declarative runner images. Add a VM-backed runtime class if fork PRs are in scope.

Firecracker/microVM runners, and the DIY route

One rung up: a microVM per job. Each job gets its own guest kernel, isolated by KVM, behind a minimal virtio device model — the same primitive that underpins several large public serverless and container platforms, chosen there for exactly this reason. A kernel-level bug reached from a build script lands inside a guest that holds one checkout and dies four minutes later. No shared page cache with someone else's job, no shared `/tmp`, no shared kernel to find a bug in.

What makes this practical rather than academic is snapshot-restore. Cold-booting a VM per job puts tens of seconds in front of every build; a team facing that will pool the VMs, pooled VMs are reset rather than single-use, and you've bought moving parts without the property you wanted. Restoring a pre-booted, pre-warmed snapshot collapses provisioning to well under a second. The DIY route — raw Firecracker plus your own agent — gives maximum control and hands you a large project: the VMM boots a kernel and a rootfs and exposes an API socket, and tap devices, network namespaces, egress rules, a template pipeline, snapshot storage, cross-host scheduling, and an orphan reaper are all yours to build and then operate forever. The VMM is the easy tenth of it; the platform is the rest, and that trade is exactly why products in this category exist.

• Isolation and lifecycle: hardware-virtualized microVM with its own guest kernel per job, destroyed after — the strongest practical boundary for build steps you did not write.
• Best fit: fork PRs, multi-tenant CI (you build other people's repos), regulated workloads, and agent-driven builds. The wrong pick if your contributor set is trusted and pod-per-job already satisfies your threat model.

Managed ephemeral-runner services (as a category)

A category rather than a single product: several vendors now sell drop-in replacements for hosted runners — you change `runs-on`, they provision the machine. The pitch is usually faster spin-up, larger or cheaper machine shapes, ARM options, and, most interestingly here, managed caching, where the vendor keeps a warm layer close to the runner so ephemeral doesn't mean cold. The thing to check — and why I won't generalize — is that the isolation model varies enormously across this category and isn't always prominent in the marketing. 'Ephemeral runner' is used for a fresh VM per job, for a container per job on shared nodes, and occasionally for a pooled machine with a reset routine. All three are legitimate products; they are not the same product. Ask in writing whether the machine is single-use, what separates concurrent jobs, and who can write to the cache.

• Isolation and lifecycle: varies by vendor — VM-per-job, container-per-job, or pooled-with-reset. Establish which before anything else, and verify residency and pricing against current docs.
• Best fit: teams who want off hosted runners for cost or machine shape, want the cache problem solved for them, and don't need the runner inside their own VPC.

GitLab Runner with the docker and kubernetes executors

The non-GitHub case, and structurally the same decision with different names: GitLab Runner picks an executor, and the executor is the isolation story. The shell executor is the anti-pattern section above wearing a different logo — jobs run directly on the runner host as the runner user. The docker executor gives a fresh container per job from a declared image: a real reset boundary with a shared kernel. The kubernetes executor is the ARC-shaped answer, a pod per job. Autoscaling instance executors provision a fresh cloud VM per job, buying VM-level separation at the cost of instance provisioning in front of every build. The surrounding controls are good and worth using deliberately — protected variables exposed only on protected branches and tags, masked variables, and `id_tokens` for OIDC against a cloud provider or Vault instead of long-lived secrets. Merge requests from forks need the same explicit thought about which variables are exposed and which runners may pick the job up; verify the current rules against GitLab's docs.

• Isolation and lifecycle: executor-dependent — shell (none), docker (fresh container per job, shared kernel), kubernetes (pod per job, shared kernel), autoscaling instance executors (fresh VM per job).
• Best fit: GitLab shops. Choose the executor by threat model rather than by whichever tutorial you found first, and never leave shell attached to a project that builds forks.

PandaStack (per-job microVMs, driven from an SDK)

Plainly, so a roundup doesn't leave the wrong impression: PandaStack is not a drop-in GitHub Actions runner product. There is no `runs-on: pandastack`. It's a sandbox platform — an open-source Firecracker control plane plus a per-host agent — driven from a Python or TypeScript SDK, and making it your runner fleet means writing the orchestrator that mints a registration token, creates a sandbox per job, and reaps it. That's an integration, not a config change, and if your CI is fine on hosted runners it is straightforwardly the wrong tool.

Where it fits is when the runner is a component of something you're building rather than a box you're renting: build-your-own CI, a verification service that compiles and tests code your users or your agents submitted, a review bot that has to actually run the branch, an agent that needs to try a fix and see whether the suite goes green. The properties that matter there are the ones I can give numbers for. Every create is a snapshot restore rather than a cold boot — 179ms p50, about 203ms p99, restore step around 49ms; the only slow path is the first-ever spawn of a brand-new template, which cold-boots in roughly 3 seconds and bakes the snapshot for everything after. The rootfs is copy-on-write, so a snapshot holding a warm toolchain and populated dependency cache is cloned per job rather than reinstalled. Same-host forks run 400–750ms (1.2–3.5s cross-host), the primitive for fanning one warmed checkout across a test matrix. Each sandbox gets its own netns and TAP device, from 16,384 pre-allocated /30 subnets per agent, so egress policy is per-job and enforced outside the guest. TTL reaping means a wedged test suite is collected rather than billed until Friday.

from pandastack import Sandbox
import json

# A custom template baked by a TRUSTED job on a schedule: toolchain plus a
# populated dependency cache, frozen into a snapshot. Pinned by date, because
# "latest" is how a poisoned cache entry outlives the ephemerality you paid for.
TOOLCHAIN = "ci-node22-2026-07-21"


def run_ci_job(job: dict) -> dict:
    """One job, one microVM, destroyed at the end. No reset script, no reuse.

    The guest holds no publish token, no cloud role, and no path to the control
    plane. Whatever it produces leaves as an artifact the ORCHESTRATOR pulls
    out -- the job is never given a way to push.
    """
    sbx = Sandbox.create(
        template=TOOLCHAIN,
        ttl_seconds=1800,          # a wedged test suite gets reaped, not billed
        metadata={
            "repo": job["repo"],
            "sha": job["sha"],     # pin the commit: a fork can move a ref
                                   # between queue time and start time
            "run_id": job["run_id"],
            "trust": job["trust"], # "fork" | "internal" -- decides the token
        },
    )

    log: list[str] = []

    def on_out(chunk: str) -> None:
        log.append(chunk)
        publish_to_ui(job["run_id"], chunk)   # live logs: one stream, one job

    try:
        sbx.filesystem.write("/work/job.json", json.dumps(job["spec"], sort_keys=True))

        # Fences first, while the only code that has run is ours. If the egress
        # policy does not apply cleanly we do NOT run the build -- fail closed.
        fence = sbx.exec("bash /opt/ci/egress-fence.sh", timeout_seconds=30)
        if fence.exit_code != 0:
            raise FencesNotApplied(job["run_id"], fence.stderr[-2000:])

        # Only now does repo-controlled code get a CPU. A build script is
        # untrusted input that happens to compile.
        exit_code = sbx.exec_stream(
            "cd /work && bash /opt/ci/run-job.sh job.json",
            on_stdout=on_out,
            on_stderr=log.append,
            timeout_seconds=1500,
        )

        artifacts = sbx.filesystem.read("/work/dist.tar") if exit_code == 0 else b""
        return {"exit_code": exit_code, "logs": "".join(log), "artifacts": artifacts}
    finally:
        # The checkout, whatever the postinstall hooks wrote into node_modules,
        # the short-lived token, the page cache, and the guest kernel all stop
        # existing together -- the only teardown that does not depend on the
        # build's cooperation.
        sbx.kill()

• Isolation and lifecycle: Firecracker microVM per job with its own guest kernel, per-sandbox netns, CoW rootfs, destroyed on kill or TTL — single-use in the strict sense.
• Best fit: build-your-own CI and verification infrastructure, and agent-driven builds where the 'contributor' is a model. Not a zero-config replacement for hosted runners — if changing one line in a workflow file would solve your problem, do that instead.

The cache problem, which is the real objection

Here's where the argument actually happens. Ephemeral runners lose the warm cache: a fresh machine has no `node_modules`, no populated `~/.m2`, no Go build cache, no compiled Rust dependencies. A two-minute suite becomes a nine-minute one, forty times a day, across thirty engineers — the security improvement now has a price people feel on every push. So teams reach for the obvious fix: a shared network cache, restored at the start of each job and saved at the end. It works beautifully, and it quietly reintroduces exactly what you paid to remove. A cache any job can write to is a cross-job write surface with a stable name, and a poisoned entry in it is a supply-chain attack that survives your ephemerality completely — the machine was destroyed; the compromised tarball was not. Arguably it's worse than the persistent runner, because it's centralized: one bad entry reaches the whole fleet, and it looks like a cache hit in the logs. More on that in /blog/microvm-ci-artifact-build-cache-isolation. The good answers keep the warmth and remove the write surface, and they're all variations on one idea: caches that jobs read but cannot write.

• Bake the cache into the image or snapshot. Toolchain and resolved dependencies are installed by a trusted job — from the lockfile, on a schedule, with its own review — and frozen into a versioned artifact that jobs get read-only. Updating the cache becomes a reviewable event instead of a side effect of whichever PR ran last.
• Content-addressed remote caches with strict write authorization. Bazel-style caching, sccache, Turborepo, Nx: keys are hashes of the inputs, so a mismatched entry doesn't match. Make PR jobs read-only and let only trusted post-merge jobs write — content-addressing alone doesn't save you if an attacker chooses what sits under a key.
• Scope cache namespaces by trust, not just by branch. A fork PR's cache should never be readable by an internal job, nor writable into the shared one. Most CI systems offer some form of this and most teams have not configured it.
• Verify on restore. If an entry carries a digest recorded by the trusted job that produced it, checking that digest turns silent poisoning into a loud failure. Cheap; almost nobody does it.
• Accept a cold cache where the job is short. A lint step that takes eleven seconds cold does not need a cache architecture. It needs to be left alone.

This is the specific place where snapshot-restore plus copy-on-write earns its keep, because it makes 'warm but immutable' structurally cheap rather than a discipline you enforce. Bake a snapshot once — toolchain installed, dependencies resolved and downloaded, page cache already warm from a real boot. Every job restores that snapshot and gets a copy-on-write clone of the rootfs: reads come from the shared baked image, writes go to a private layer, and the private layer is discarded when the machine dies. The job sees a fully warm, writable environment, and what it writes is visible to nothing and nobody, ever, because there is no path from its CoW layer back to the baked image. Warm and immutable stop being in tension, which is the whole trick.

Secrets: short-lived tokens, and why forks are a separate problem

The default in most pipelines is still a long-lived secret in an environment variable: a publish token, a cloud access key, a deploy key. It exists before the job, survives the job, and is valid from anywhere — which undoes everything ephemerality bought you, because you destroyed the machine and the thing worth stealing was never tied to the machine. OIDC is the structural fix: the CI platform mints a signed token asserting who the job is (repo, ref, workflow, environment) and your cloud provider or secret manager exchanges it for a credential that expires in minutes and is scoped by those claims, so nothing long-lived is stored in the CI system at all. The part teams get wrong is the trust policy on the other end — a condition matching the repository but not the ref lets any branch in that repo assume the role, including one pushed five minutes ago. Pin the subject claim as tightly as the workflow allows.

Fork pull requests deserve their own paragraph, because this is where teams inherit a footgun they didn't build. The platforms already default to safe: a workflow triggered by `pull_request` from a fork gets a read-only token and no secrets. Then someone needs a comment posted or a preview deployed, discovers `pull_request_target`, and switches — and that trigger runs in the base repository's context, with secrets available. The intended use is running your trusted workflow code against the fork's metadata. What it becomes, roughly every time, is a workflow that checks out the fork's head commit and runs its build script with your secrets in the environment: arbitrary code execution with your credentials, added on purpose by a well-meaning engineer solving a real problem.

Split the pipeline instead: an untrusted stage that builds fork code with no secrets, no write token, and a locked-down egress policy, and a separate trusted stage that consumes only its declared artifacts. Which is the point of per-job isolation in the first place. You cannot always avoid a job holding a credential — deploy jobs exist, publish jobs exist. What per-job isolation buys is that when a job legitimately holds one, nothing else is in the room with it: no other job's process tree, no cached credential from last Tuesday, no shared kernel, no filesystem that will still be there when the next job starts. A short-lived token in a machine that dies with it is a survivable design. The same token on a shared runner is a matter of timing.

Egress: default-deny, enforced outside the guest

Isolation stops an escape. It does not stop an HTTP POST, and exfiltration needs no escape — a build script with network access and a copy of your source tree is already the whole incident. Default-deny plus an allowlist for the package registry is the control, and placement matters more than the rules: enforce them on the host side of the job's network namespace, not inside the job. Build steps run as root inside their own environment far more often than people admit, and rules a job can flush are a suggestion.

# Runs on the HOST, inside the per-sandbox network namespace -- OUTSIDE the
# guest. Rules configured inside the job are advisory: plenty of build steps run
# as root in there, and root can flush a table.

nft add table inet ci
nft add set  inet ci registry_v4 '{ type ipv4_addr; flags interval; }'

# Default DROP on forward. Everything the job reaches must be named below.
nft add chain inet ci fwd '{ type filter hook forward priority 0; policy drop; }'

# Return traffic, or nothing works and you will "temporarily" disable all of
# this at 2am -- and temporarily is a very long time.
nft add rule inet ci fwd ct state established,related accept

# DNS to OUR resolver only, which answers only for allowlisted names. Leave port
# 53 open to the world and the exfiltration channel is TXT lookups: slow,
# entirely adequate for a private key, and invisible in an HTTP log.
nft add rule inet ci fwd iif "tap0" ip daddr 10.200.0.1 udp dport 53 accept

# The registry, as an IP SET refreshed by the trusted orchestrator. Do not
# resolve the hostname in here to build the rule -- the answer would come from a
# resolver the job can influence, which is a rule that allowlists itself.
nft add rule inet ci fwd iif "tap0" ip daddr @registry_v4 tcp dport 443 accept

# The metadata service, explicitly. It hands a cloud role to anything that asks
# and it never asks who is calling.
nft add rule inet ci fwd ip daddr 169.254.169.254 counter drop

# Everything else: dropped, counted, logged. The counter is the useful part --
# "this build tried to reach 46 hosts you have never heard of" is a finding.
nft add rule inet ci fwd counter log prefix "ci-egress-drop " drop

Four shapes, side by side

• Isolation boundary — GitHub-hosted: fresh VM per job, a strong boundary you inherit but don't set. Self-hosted VM (naive): none — shared kernel, filesystem, and instance identity. K8s pod-per-job: fresh namespaces per job, kernel shared with every other pod on the node. MicroVM-per-job: own guest kernel under KVM with a minimal device model, so a kernel-level bug in the build path costs a machine that was about to be deleted anyway.
• State between jobs — GitHub-hosted: none. Self-hosted VM (naive): everything — caches, credentials, `$PATH`, crontabs, the reused checkout and its git hooks. K8s pod-per-job: none in the pod, but anything mounted from a shared volume or cache service persists and is writable. MicroVM-per-job: none — the CoW layer holding every write is discarded with the guest.
• Provisioning latency — GitHub-hosted: queue plus VM start, generally fine and outside your control. Self-hosted VM (naive): effectively zero, which is exactly why it's so sticky. K8s pod-per-job: pod scheduling plus image pull, tail dominated by image size. MicroVM-per-job: snapshot restore rather than cold boot — on PandaStack 179ms p50, ~203ms p99, restore step around 49ms; the ~3s cold boot happens once, at bake time.
• Cache strategy — GitHub-hosted: platform cache actions, convenient and shared, so write-scoping is on you. Self-hosted VM (naive): the warmest caches available and the worst ownership model — everyone writes, nobody reviews. K8s pod-per-job: image layers plus a shared cache volume, which is the cross-job write surface. MicroVM-per-job: bake the warm cache into the snapshot and CoW-clone it — warm on read, private on write, gone at the end.
• Secrets exposure — GitHub-hosted: OIDC available, fork PRs safe by default until someone reaches for `pull_request_target`. Self-hosted VM (naive): credentials at rest plus an instance profile any job can query. K8s pod-per-job: projected service-account tokens and OIDC, undone instantly by a mounted Docker socket. MicroVM-per-job: a short-lived token in a machine that dies with it, nothing else sharing the room.
• Egress control — GitHub-hosted: limited; the job runs in GitHub's network and the policy is theirs. Self-hosted VM (naive): technically yours, practically unset, enforceable only host-wide. K8s pod-per-job: NetworkPolicy is real and effective at pod granularity, if your CNI enforces it. MicroVM-per-job: per-sandbox netns and TAP, so default-deny is enforced outside the guest where the job can't flush it.
• Cost and operational burden — GitHub-hosted: billed minutes, no idle cost, essentially no ops, which is worth real money. Self-hosted VM (naive): one instance-hour bill whether you're building or asleep, and near-zero ops on day one that is unbounded by month eighteen, all of it paid as risk. K8s pod-per-job: node capacity you provision and autoscale, run by a controller you already operate — the best marginal-effort story if you're already there. MicroVM-per-job: near-zero idle thanks to TTL reaping, but a real platform (agents, snapshots, networking, scheduling) you either build or buy.

The row that quietly settles the argument is provisioning latency, because it decides whether the secure design stays secure. Every ephemeral CI architecture I've watched decay decayed the same way: provisioning was slow, so someone added a pool to keep queue times down, the pool's machines started outliving jobs, and within a quarter 'ephemeral' meant 'we run a cleanup script'. When a fresh machine costs a fifth of a second in front of a job that takes four minutes, nobody proposes the pool, because there's nothing to optimize.

How to choose

1. Establish whether you build untrusted code at all. Fork pull requests on a public repo, user-submitted repos, or model-generated build steps mean yes. Ten employees on a private monorepo means the threat is a compromised dependency rather than a hostile contributor — still real, but it changes which boundary you need.
2. Stay on hosted runners unless a specific, nameable constraint forces you off: VPC access to a private database, a machine shape that doesn't exist, a compliance requirement about where code executes, or volume economics you have actually modelled. 'Builds feel slow' is usually a caching problem you can fix without owning infrastructure.
3. If you're already Kubernetes-native and your contributor set is trusted, use ARC or a runner scale set with pod-per-job — the best effort-to-improvement ratio on this list. Dedicated node pool with no cluster-credential access, no Docker socket, daemonless image builder, NetworkPolicy. Then stop; you're done.
4. If you build genuinely untrusted code — forks, customers' repos, agent-authored branches — get a per-job kernel: a VM-backed runtime class inside Kubernetes, a managed service whose docs say VM-per-job in writing, or a microVM platform. Verify that claim against documentation, not a marketing page.
5. If the runner is a product component rather than a rented box — you're building CI, a verification service, or an agent that has to run the branch to know whether it fixed the bug — drive per-job microVMs from an SDK and own the orchestration. Budget for that orchestrator honestly; it's a build, not a config change.
6. Whichever you pick, fix the cache and the credentials on the same day: read-only baked or content-addressed caches with writes limited to trusted post-merge jobs, OIDC with tightly-pinned subject claims, and default-deny egress enforced outside the job. An ephemeral runner in front of a world-writable cache and a static publish token has moved the problem, not solved it.

Where this leaves you

The ephemeral argument is settled, and it was never really about exotic attacks. It's that a persistent runner accumulates trust — caches, credentials, a checkout, a `$PATH`, a cloud role — and hands the accumulated pile to whoever most recently opened a pull request. No cleanup script fixes that, because the cleanup script runs after the untrusted code, on a machine the untrusted code just had. Destroying the machine is the only teardown whose correctness doesn't depend on the build's cooperation.

What's left to decide is the boundary and the cost. Hosted runners give you ephemerality for free and should be the default until something specific forces you off. Pod-per-job on Kubernetes is the best marginal-effort upgrade if you're already there, with the honest caveat that the kernel is shared. A microVM per job is the right boundary when the code is genuinely untrusted, and snapshot-restore is what makes it practical rather than a thing you talk about. And the two objections that sink most implementations aren't about isolation at all: solve the cache with warm-but-immutable images or snapshots plus content-addressed remote caches only trusted jobs may write, and solve secrets with short-lived OIDC credentials and a hard split between the stage that runs fork code and the stage that holds anything worth stealing.

PandaStack's position, once more: we're not a drop-in Actions runner, and if changing one line in a workflow file would solve your problem, do that instead. Per-job microVMs driven from an SDK make sense when the runner is infrastructure you're building — 179ms p50 creates via snapshot-restore, CoW clones of a warm toolchain, same-host forks at 400–750ms for matrix fan-out, per-sandbox netns for egress, TTL reaping so nothing outlives its job. For the adjacent detail: the per-job runner loop end to end is in /blog/ephemeral-github-actions-runners-firecracker, and protecting build credentials from a malicious postinstall is in /blog/secure-ci-secrets-microvm.