Self-Hosted GitHub Actions Runners in Firecracker MicroVMs

A fork pull request is a stranger handing you a script and politely asking you to run it as root, on a machine that also holds your deploy keys. That's not hyperbole — it's the literal job of a self-hosted GitHub Actions runner: check out code you didn't write and execute it with whatever privileges the runner has. GitHub's own documentation is blunt about this: they recommend self-hosted runners only for private repositories, because a public repo means anyone on the internet can open a PR and your runner will faithfully run their code. This post is about making that safe by giving each job a fresh Firecracker microVM, registering it as an ephemeral runner, letting it do exactly one job, and then deleting the entire machine — and making that loop cheap enough that per-job VMs are the obvious default rather than an expensive indulgence.

The threat model: untrusted code, your host, your secrets

Start with who's actually running code on your infrastructure, because it's not just your team. The moment you attach a self-hosted runner to a repo that accepts external contributions, the set of people who can execute code on that runner expands to 'everyone with a GitHub account.' A pull request from a fork is arbitrary code, authored by someone you can't vet, that your CI will clone and run. Most contributors are honest. You cannot tell the honest ones from the person who opened the PR specifically to get code execution on your build fleet, and you have to design for the second one.

So enumerate what that PR's code can try to do on a persistent, shared runner:

• Steal secrets and tokens: the runner has a registration token, often cached credentials, and — depending on your workflow — access to secrets. Malicious workflow code, a poisoned build script, or a compromised third-party action can read anything the runner process can reach. This is the 'poisoned pipeline execution' pattern: the attacker's leverage is that your CI runs their build steps.
• Persist to attack the next job: on a runner that survives between jobs, PR code can drop a binary in a cached tool directory, poison the dependency cache, plant a git hook, or leave a background process running — so the next job (maybe a trusted one, on the main branch, with real secrets) inherits the compromise. State that outlives a job is a weapon.
• Escape to the host: a runner is usually a container or a bare process on a shared-kernel VM. A kernel exploit or container escape turns 'this job misbehaved' into 'the attacker owns the runner host and every job scheduled on it.' Container escapes are a well-documented, recurring class of bug, not a thought experiment.
• Exfiltrate outbound: PR code can POST your source, your secrets, or your internal service responses to an attacker's server, or pivot to your internal network — the cloud metadata endpoint at 169.254.169.254 is the classic prize, since it often hands out IAM credentials to anyone on the box who asks.

Why GitHub's `--ephemeral` flag alone doesn't close the gap

GitHub gives you a real, useful tool here: register the runner with the `--ephemeral` flag and it accepts exactly one job, then automatically deregisters itself. That solves half the problem — it kills the persistent-runner state-leakage vector, because the runner agent won't pick up a second job. If you pair it with a fresh container per registration, you also get a clean filesystem per job. Use it. It's the right foundation.

But `--ephemeral` is a statement about the runner agent's job lifecycle, not about the isolation boundary underneath it. If your ephemeral runner is a container, the job still runs behind a shared host kernel — the same large, privileged attack surface every container on that machine shares. A malicious PR that finds a kernel bug escapes the container and reaches the host regardless of how many times the runner deregistered itself. And if you run the runner agent as a long-lived process that spawns containers per job, the host process itself — with its registration token — is still sitting there for a compromised job to reach for.

`--ephemeral` guarantees one job per runner registration. It does not guarantee one kernel per job. For untrusted code, you want both.

The fix is to make the isolation boundary a whole machine: a Firecracker microVM with its own guest kernel, its own memory, and its own network namespace, enforced by the CPU's virtualization extensions. Register the ephemeral runner inside that VM, run the one job, and then delete the VM — kernel, filesystem, registration token, and any implant the job tried to leave behind, all gone together. This is the same reason AWS runs Lambda on Firecracker instead of trusting containers between tenants: when the code is a stranger's, you want a hardware wall, not a namespace.

The loop: register one ephemeral runner, run one job, destroy the VM

The control loop is small and it's the whole design. An orchestrator (a tiny queue worker, a webhook handler, or a controller like the autoscaling-runner pattern) does five things per queued job:

1. Provision a fresh microVM from a runner template — ideally a snapshot you baked with the GitHub Actions runner binary and your toolchain already installed, so there's no download or install on the hot path.
2. Mint a short-lived registration token from the GitHub API (the registration-token endpoint for your repo or org) and inject it into the VM. The token is scoped and expires; it never touches disk on a persistent host.
3. Register the runner with `--ephemeral` inside the guest and start it. It connects to GitHub, claims exactly one queued job, and runs it — the untrusted checkout and every workflow step execute inside this disposable VM.
4. Wait for the one job to finish. Because it's `--ephemeral`, the runner deregisters itself automatically after the single job; you also capture logs and any artifacts you deliberately want to keep.
5. Delete the VM. Nothing survives — not the runner agent, not the workspace, not the registration token, not anything the job stashed. That total erasure is the isolation.

Because the VM is disposable, there is no cleanup script to get wrong and no 'reset the runner between jobs' logic to forget. If a job mines a block, plants a hook, or fork-bombs itself, it does so inside a machine that's about to cease to exist. The blast radius is one job, and the recovery is `delete`.

A per-job ephemeral runner in Python

Here's the whole loop with the PandaStack Python SDK: create a runner-template microVM, inject a fresh registration token, register and run an ephemeral runner for one job, then tear the VM down no matter what. Set PANDASTACK_API_KEY in the environment first. You mint the `RUNNER_TOKEN` from GitHub's registration-token API just before calling this (it's short-lived by design); the `metadata` tag attributes the VM for auditing, and `ttl_seconds` is a backstop so a hung job kills its own VM even if the orchestrator crashes.

import os
from pandastack import PandaStack

ps = PandaStack()  # reads PANDASTACK_API_KEY, base url https://api.pandastack.ai

REPO_URL = "https://github.com/acme/widget"

def run_one_job(runner_token: str, labels: str = "self-hosted,fc-microvm") -> bool:
    # 1. Fresh, hardware-isolated microVM. Every create restores a baked
    #    snapshot on demand (~179ms p50), so a VM-per-job is cheap. Bake the
    #    Actions runner binary + toolchain into the template to skip the download.
    sb = ps.sandboxes.create(
        template="gha-runner",                 # snapshot with actions/runner baked in
        ttl_seconds=1800,                      # hard cap: VM self-destructs after 30 min
        metadata={"repo": REPO_URL, "kind": "gha-ephemeral-runner"},
    )
    try:
        # 2. Deliver the short-lived registration token as a file, not a flag,
        #    so it never lands in the process table / shell history.
        sb.filesystem.write("/opt/runner/.reg-token", runner_token)

        # 3. Register + run ONE job. `--ephemeral` makes the runner accept a
        #    single job and self-deregister. `run.sh` blocks until that job ends;
        #    the untrusted checkout + every step happen INSIDE this VM.
        result = sb.exec(
            "cd /opt/runner && "
            f"./config.sh --url {REPO_URL} "
            "--token \"$(cat .reg-token)\" "
            f"--labels {labels} --ephemeral --unattended --replace && "
            "rm -f .reg-token && ./run.sh",
            timeout_seconds=1500,
        )

        # 4. The runner exits when its single job completes. Capture the tail of
        #    the log for observability; the job's real result lives on GitHub.
        print(result.stdout[-2000:])
        return result.exit_code == 0

    finally:
        # 5. Destroy. The kernel, workspace, token file, and anything the job
        #    tried to leave behind all die with the VM. That's the whole point.
        sb.delete()

Note the two safety rails, because they earn their place. The `timeout_seconds` on `exec` is a circuit breaker for a job that hangs or deliberately stalls; the `ttl_seconds` on create is a backstop so a VM your orchestrator forgot to reap reaps itself. And notice the token is written as a file and deleted right after `config.sh` consumes it — a registration token is a credential, and you don't want it living in the argv of a process an untrusted job could inspect. If you'd rather stream a long-running job's console live instead of blocking, the SDK exposes a streaming exec that emits stdout, stderr, and exit events.

What the registration actually looks like inside the guest

The `exec` above runs GitHub's own runner tooling — the same `config.sh` / `run.sh` scripts from the `actions/runner` release. Unrolled, the guest is doing roughly this:

#!/usr/bin/env bash
set -euo pipefail
cd /opt/runner

# Register this VM as a runner for the repo. --ephemeral is the load-bearing
# flag: the runner takes exactly ONE job, then removes its own registration.
./config.sh \
  --url "https://github.com/acme/widget" \
  --token "$(cat .reg-token)" \
  --labels "self-hosted,fc-microvm" \
  --name "fc-$(hostname)" \
  --ephemeral \
  --unattended \
  --replace

# The token is single-use for registration; scrub it immediately.
rm -f .reg-token

# Block here until the one job finishes, then the runner deregisters and exits.
# We never loop: this guest exists to run a single job and then be deleted.
exec ./run.sh

There is deliberately no `while true` and no service supervisor restarting the runner. The guest's entire reason to exist is one job. When `run.sh` returns, the VM is inert and the orchestrator deletes it. If you want the VM to reap itself even if the orchestrator vanishes, wrap `run.sh` in a `timeout` and let the create-time `ttl_seconds` be the final backstop.

Egress control: the supply-chain seatbelt

Isolation stops a job from reading its neighbors. Egress control stops it from phoning home. You need both, and for CI the egress side is really a supply-chain control: the danger isn't only the PR's own code, it's the transitive dependencies and third-party actions that code pulls in. A single compromised npm postinstall script or a typosquatted action can turn a routine build into an exfiltration event. The microVM helps here structurally because every sandbox gets its own network namespace with its own TAP device and NAT rules — PandaStack pre-allocates 16,384 /30 subnets per agent, one per sandbox, so a job's network is genuinely its own segment rather than a shared bridge. That per-VM network is your enforcement point:

• Block the internal ranges and metadata endpoint first: deny RFC1918 (10/8, 172.16/12, 192.168/16) and the link-local 169.254.0.0/16 that covers the cloud metadata IP. A build step should never be able to reach your database, your control plane, or your instance's IAM credentials.
• Allowlist package registries, not the whole internet: a job legitimately needs npm, PyPI, your artifact store, and github.com. It does not need to open an arbitrary socket to anywhere. Allowlists fail closed; blocklists lose the arms race.
• Rate-limit and cap outbound: bound how much a single job can send so a compromised one can't become an exfiltration firehose on your bandwidth bill.
• Deny egress entirely for offline steps: a test suite that runs against baked, vendored dependencies needs no network at all. The safest build step is one that can't open a socket.

Caching without leaking secrets between jobs

The obvious objection to a throwaway VM per job is that you throw away your warm caches too, and re-downloading node_modules every job is slow. The microVM model gives you a cleaner answer than a shared writable cache — which is exactly the thing you're trying to avoid, because a shared writable cache is how one job poisons the next.

Bake dependencies into the snapshot. Do the install once, snapshot the VM, and restore from that snapshot for every subsequent job. Restore is memory copy-on-write (MAP_PRIVATE) plus a reflink rootfs clone, so each job shares the baked pages until it writes — the Nth job doesn't re-copy the cache, it inherits a read-mostly view of it. Crucially, the cache is baked and immutable at job start: an untrusted PR can dirty its own copy-on-write view, but it cannot write back into the shared baked layer that the next job restores from. You get cache reuse and clean isolation in the same move. Refresh the snapshot when your lockfile changes; treat it like a cache key. If you must share a fetched-once cache across jobs (a big Docker layer store, say), mount it read-only into the untrusted job and let a separate trusted job be the only writer.

Making a VM-per-job cheap: snapshot-restore, not cold boot

A full VM per CI job sounds absurd if you picture cold-booting a conventional VM — tens of seconds and gigabytes of RAM per job would make this a non-starter. But a microVM snapshot restore is a different operation entirely. A cold boot does real work: the kernel initializes, userspace comes up, the runtime loads. A restore skips all of it — Firecracker maps a frozen, already-booted VM's memory and device state back in and resumes it mid-instruction. It's closer to unpausing than to starting a computer.

On PandaStack that restore-based create runs at a p50 of about 179ms and a p99 around 203ms (the restore-and-resume core is roughly 49ms), versus a genuine cold boot of about 3 seconds that happens exactly once per template and is then baked into a snapshot and amortized away. Because memory is copy-on-write and the rootfs is a reflink clone, standing up the Nth identical runner VM doesn't copy gigabytes. If you'd rather branch a live warm runner to fan out several jobs from one prepared state, a same-host fork is 400–750ms (cross-host 1.2–3.5s). At those numbers, a fresh VM per job stops being a cost problem: the create overhead disappears under your actual build-and-test time, which is where the seconds belong.

Ephemeral microVM per job vs. reused runner vs. GitHub-hosted

There are three common shapes for where a workflow job runs. It's worth laying them side by side rather than picking one dogmatically — the right answer depends on your trust posture, your control needs, and your volume. Treat the GitHub-hosted characterizations qualitatively and verify current specifics against GitHub's own documentation, since managed offerings change.

• Isolation strength — Ephemeral microVM per job: strongest; a clean guest kernel per job behind a hardware boundary, nothing survives. Reused self-hosted runner: weakest; a shared host kernel plus persistent state, wrong for untrusted PRs. GitHub-hosted: strong; GitHub runs each job on a fresh managed VM (verify the current isolation model in their docs).
• State between jobs — Ephemeral microVM: none by construction; every job is a clean slate. Reused runner: everything persists — caches, installed tools, planted implants — which is the whole danger. GitHub-hosted: fresh each job, no cross-job persistence.
• Fork-PR safety — Ephemeral microVM: safe by design; untrusted code is trapped in a disposable VM with your egress policy. Reused runner: dangerous; GitHub explicitly recommends against self-hosted runners on public repos for this reason. GitHub-hosted: designed for it — public-repo fork PRs run on ephemeral managed infrastructure.
• Control over image + egress — Ephemeral microVM: total; you bake the rootfs, pin the kernel, and own the network namespace and egress rules byte-for-byte. Reused runner: total, but you also own all the cleanup you'll inevitably get wrong. GitHub-hosted: limited to what the managed runner and its network exposes.
• Cost shape — Ephemeral microVM: you run the fleet; near-zero idle cost (no warm pool), pay for the build time plus a sub-200ms create. Reused runner: cheap to run, expensive in blast radius and flaky-state debugging. GitHub-hosted: pay-per-minute managed pricing, zero infra to operate — great until the bill or the minutes-cap bites at volume.
• Ops burden — Ephemeral microVM: you own scheduling, autoscaling, snapshot hygiene, and token minting. Reused runner: minimal to stand up, maximal to keep safe. GitHub-hosted: essentially none; that's the product.

In practice a lot of teams blend these: GitHub-hosted runners for low-risk trusted workflows where zero ops wins, and ephemeral microVM runners for the jobs that actually justify the fleet — untrusted fork PRs, compliance boundaries, or high volume where per-minute managed pricing hurts. The reused persistent runner is the one shape that's hard to defend for anything that touches untrusted code.

When you don't need to run your own microVM runners

This is real infrastructure, not a free lunch. You own the orchestrator, snapshot hygiene, egress policy, token minting, and autoscaling the VM fleet against your queue depth. Be honest about whether your threat model calls for it.

• All your repos are private and every contributor is trusted. If there are no fork PRs and no untrusted code, the supply-chain-adversary argument mostly evaporates, and GitHub-hosted or a hardened container runner may be plenty.
• Your job volume is low and bursty. If you run a handful of jobs a day, the operational overhead of running a microVM fleet outweighs the per-job savings — let GitHub run the runners.
• You need zero infrastructure. GitHub-hosted runners hand you the fleet, the scaling, and the isolation as a managed product. That's a genuine feature, not a compromise, when your workflows fit inside it.

Reach for ephemeral microVM runners when you run untrusted fork PRs, when compliance or contracts demand a hardware isolation boundary, when you want byte-for-byte control over the runner image and its egress plane, or when your job volume makes managed per-minute pricing painful. In those cases snapshot-restore gives you what a container runner can't — a real kernel boundary per job, and total erasure after it — without the cold-boot tax that used to make per-job VMs look impossible. A fork PR is a stranger's script; register it a fresh machine, let it run once, and throw the machine away.