Quarantine Flaky Tests with Ephemeral MicroVMs

A flaky test is one that passes and fails on the same code. The industry-standard response is a retry: mark it flaky, re-run it, and if the second attempt goes green, ship. But "it passed on retry" is not a fix — it's a confession that your test outcome depends on something other than your code. Most of the time that something is the runner itself: a leftover process from the previous job, a port that's still bound, temp files in /tmp, a mutated ~/.config, a background daemon, or a database row a neighbor test left behind. The retry "works" because the second attempt happened to land on a slightly less dirty machine, not because anything got better.

The durable fix is to make dirty state impossible: give each test suite — or each individual flaky test under quarantine — a fresh, byte-for-byte identical environment for every single run. Firecracker microVMs make this cheap enough to actually do. Every run restores a baked snapshot into its own microVM in about 49ms (p50 create 179ms end to end), so "clean, identical machine per run" stops being an aspiration and becomes the default. I'm Ajay, I built PandaStack — here's how the ephemeral-runner model kills neighbor-pollution flakiness, and how forking lets you measure a test's true flake rate instead of guessing.

Why "works on retry" tests are usually about dirty state

Some flakiness is genuinely in the test — a race between two threads, an assertion on an unordered collection, a sleep that's a hair too short. But a large share of what teams call flaky is really the environment leaking between runs on a reused runner. The test didn't change; the machine it landed on did. The usual culprits are boring and repeatable once you know to look for them.

• Leftover ports: the previous job's server didn't shut down cleanly, so :8080 is still bound. Your test's "address already in use" is really the last test's ghost.
• Temp-file and cache cruft: a fixture in /tmp, a half-written cache, a global npm or pip install from job N that job N+1 silently picks up (or trips over).
• Background processes: a daemon, a watcher, or a stray worker from an earlier suite still eating CPU and racing your timing-sensitive assertions.
• Shared database state: a row, a sequence value, or a migration a neighbor test left behind, so your test sees data it never created.
• Wall-clock and timing coupling: a test that only fails when the run crosses a minute boundary, or under load from a co-scheduled job — the notorious 'only fails on a full moon' test is almost always this plus one of the above.

A fresh microVM per run removes the shared state entirely

You cannot leak state from a previous run into a machine that didn't exist until this run started. That's the whole idea. Instead of scrubbing a long-lived runner between jobs — deleting caches, killing processes, resetting the filesystem, and hoping your cleanup script didn't miss the one thing that matters — you throw the machine away and restore a new one from a snapshot. Every run sees an identical filesystem, an empty process table, no bound ports, and a database at whatever state the snapshot baked in. There is no 'previous run' to inherit.

The reason this is practical and not just correct-in-theory is snapshot-restore. On PandaStack there's no warm pool of idle VMs to pay for and no multi-second cold boot per run. Each create restores a baked Firecracker snapshot on demand: the memory is copy-on-write (MAP_PRIVATE, paged in lazily) and the rootfs is a reflink clone, so spinning up the 500th identical runner doesn't copy gigabytes. The first-ever cold boot of a template is around 3 seconds; every create after that restores in ~49ms, p50 179ms end to end and p99 ~203ms. A fresh, identical machine per run costs you milliseconds, not minutes.

• Environment provenance — Reused runner: accumulated state from every prior job; 'clean' is a best-effort script. Fresh microVM: identical baked snapshot every run; clean by construction.
• Leftover ports/processes — Reused runner: possible, and the source of half your flakes. Fresh microVM: impossible — empty process table, nothing bound.
• Cleanup cost — Reused runner: a scrub script you maintain and that silently rots. Fresh microVM: none; you delete the VM.
• Isolation boundary — Reused runner: shared kernel across jobs. Fresh microVM: own guest kernel, hardware-virtualized.
• Cost of a clean slate — Reused runner: slow to rebuild, so people skip it. Fresh microVM: ~49ms restore, so it's the default.
• Reproducing a flake — Reused runner: 'can't repro, it was the machine.' Fresh microVM: fork the exact snapshot and run it N times in parallel.

The quarantine loop: one microVM per suite

The pattern is the same one you'd use for ephemeral CI runners, applied at the granularity of a suite (or a single quarantined test). Bake a snapshot with your toolchain and dependencies already installed, then for each run: create a sandbox from it, get the code in, run the suite, capture the result, and destroy the VM. Because the machine is disposable, you never write defensive cleanup — if the suite corrupts its environment, it corrupts a VM you're about to delete.

from pandastack import Sandbox

# One quarantined suite, one fresh microVM. No neighbor to pollute it.
suite = "tests/flaky/test_scheduler.py"

with Sandbox.create(template="code-interpreter", ttl_seconds=600) as sbx:
    # Get the code in. Clone runs INSIDE the microVM, on a machine born this run.
    sbx.exec(
        "git clone --depth 1 https://github.com/acme/widget.git /work",
        timeout_seconds=120,
    )

    # Run only the quarantined test, on a guaranteed-clean box.
    # No leftover :8080, no /tmp cruft, no stray daemon from a prior job.
    result = sbx.exec(
        "cd /work && python -m pytest -q " + suite,
        timeout_seconds=300,
    )

    print("exit:", result.exit_code)      # 0 = pass
    print(result.stdout[-2000:])          # tail for the PR annotation
    if result.exit_code != 0:
        print("FAIL stderr:\n", result.stderr)
# microVM is destroyed here — nothing survives to taint the next run

`exec` returns a result with `stdout`, `stderr`, and `exit_code`; the exit code is your pass/fail. Always pass a `timeout_seconds` — a test that hangs waiting on a port that will never come free is exactly the failure mode you're trying to surface, and the timeout is your circuit breaker. Set a `ttl_seconds` on create too so a forgotten VM reaps itself. If this suite still fails here, on a provably clean machine, you've learned something real: the flake is in the test, not the runner, and now it's reproducible.

Fork to measure a test's true flake rate

Once a test is quarantined on a clean machine, the next question is quantitative: is this a 1-in-1000 race or a 1-in-3 disaster? Retrying serially is slow and still pollutes across attempts if you reuse the box. The microVM answer is to fork. Get the suite to a ready-to-run state once, snapshot it, then fork that snapshot N ways — each fork is an independent, identical microVM sharing the baked memory copy-on-write until it writes. A same-host fork is 400–750ms (cross-host 1.2–3.5s), so you can launch 50 identical runs in parallel and let statistics do the work.

from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox

SUITE = "tests/flaky/test_scheduler.py::test_race"
N = 50  # number of identical parallel runs

# 1. Prepare one machine to a known-good, ready-to-run state, then snapshot it.
base = Sandbox.create(template="code-interpreter", ttl_seconds=900)
base.exec("git clone --depth 1 https://github.com/acme/widget.git /work",
          timeout_seconds=120)
base.exec("cd /work && pip install -e .", timeout_seconds=300)
snap = base.snapshot()   # capture filesystem + memory
base.kill()

# 2. Fork the snapshot N ways. Every fork is an identical, isolated microVM.
def run_once(i: int) -> int:
    fork = snap.fork(ttl_seconds=300)
    try:
        r = fork.exec("cd /work && python -m pytest -q " + SUITE,
                      timeout_seconds=180)
        return r.exit_code
    finally:
        fork.kill()

with ThreadPoolExecutor(max_workers=N) as pool:
    codes = list(pool.map(run_once, range(N)))

# 3. Each run started from byte-identical state, so any variance is the TEST.
failures = sum(1 for c in codes if c != 0)
print(f"flake rate: {failures}/{N} = {failures / N:.1%}")

Because every fork started from the same baked memory and disk, the environment is controlled for — the only variable left is the test itself. A 4/50 result is a genuine 8% intrinsic flake rate you can now attack (a real race, a timing assumption, an unseeded random). A 0/50 result on forks when the test 'flakes' in your normal pipeline is the loudest possible signal that the problem was never the test — it was the shared runner. Either way you've replaced a shrug and a retry with a number.

Bake dependencies once so per-run stays cheap

Installing dependencies on every run would drown the millisecond restore under minutes of npm ci or pip install. Do it once: prepare a sandbox with your toolchain, lockfile, and any database seed installed, snapshot it, and restore that snapshot for every run and every fork. The install is amortized to near zero, and — crucially for flake hunting — every run is guaranteed to start from the exact same dependency tree, so a version skew can't masquerade as flakiness. Refresh the snapshot when your lockfile changes and treat it like a cache key.

# Inside a prepared microVM: install deps once, then it gets snapshotted.
# Every later run/fork restores this exact state in ~49ms — no reinstall,
# no version drift, no 'works on my machine' between runs.
set -euo pipefail

git clone --depth 1 https://github.com/acme/widget.git /work
cd /work

# Node + Python toolchains, pinned by lockfiles baked into the snapshot.
npm ci
pip install -e '.[test]'

# Seed a throwaway Postgres so DB-touching tests start from identical rows.
pg_ctl -D /var/lib/pg -l /tmp/pg.log start
psql -f tests/fixtures/seed.sql

echo 'ready to snapshot: deps installed, db seeded, ports free'

For DB-touching suites you have two clean options: bake a seeded database into the snapshot so every run starts from identical rows (fork gives each run its own copy-on-write disk, so writes never collide), or spin up a managed PostgreSQL microVM per run — that create is 30–90s because it blocks on Postgres bootstrap, so bake-and-fork is usually the faster path for a tight quarantine loop. Either way, the point stands: identical DB state per run means a leftover row can never make a green test go red.

When a fresh microVM per run is overkill

Per-run microVMs are not free of operational cost — you own snapshot hygiene, scheduling, and artifact plumbing — so match the tool to the problem. If your test genuinely races two threads, or asserts on an unordered set, or depends on an unseeded random, a clean machine won't save you: the flake is in the code and belongs in a debugger, not a fresh VM. Likewise, if your suite already runs in well-isolated containers and you've never seen a neighbor-pollution flake, you may not need to change anything.

But the moment you catch yourself adding a retry to make CI green, stop and ask which kind of flaky you have. If 'run it again' fixes it, that's the dirty-runner signature, and a fresh microVM per run removes the entire class of failure by construction — plus gives you a fork-based way to measure what's left. Snapshot-restore is what makes that affordable: a clean, identical, hardware-isolated machine per test run for the price of a ~49ms restore, so you can stop shipping on the strength of a coin flip that landed heads the second time.