Run Browser E2E Tests in Isolated MicroVMs

End-to-end tests are the flakiest thing most teams own, and it isn't because Playwright or Cypress is bad — it's because a browser test is a small, stateful, process-spawning program that assumes it owns the machine. It launches a real Chromium, writes to a shared profile directory, downloads files into ~/Downloads, binds a dev server to a port, and occasionally hangs on a `waitForSelector` that will never resolve. Run a hundred of those on the same CI runner, in parallel, and they contaminate each other in ways that surface as 'this test passes locally but fails on CI' — the single most demoralizing sentence in software. The fix is boring and total: give each shard its own disposable Firecracker microVM with a clean kernel, a fresh browser, its own port space, and a hard wall-clock limit. Run the shard, read back the traces, kill the VM. Nothing survives to poison the next run.

Why E2E tests contaminate each other

A browser test does far more to the host than a unit test. It's not asserting on a pure function — it's driving a real user agent with real disk, real network, and real child processes. When several of those share a machine, everything that isn't explicitly partitioned becomes a shared variable, and shared variables in a test suite are just races with better PR.

• Shared browser profile: Playwright and Selenium keep a user-data dir with cookies, localStorage, service workers, and cached auth. Test A logs in, test B inherits the session and 'passes' as the wrong user; or A corrupts the profile and B launches into a chrome://crash screen. Incognito contexts help until something writes outside the context.
• Leaked processes: a browser is a process tree. A test that times out mid-navigation can leave a headless Chromium and a renderer child alive, holding a port and a chunk of RAM. The next shard inherits a machine with fewer resources and a mystery listener on :3000.
• Downloads and the filesystem: E2E tests that exercise a 'download CSV' button drop files into a shared directory. Two tests download report.csv and now assertion order decides which one you're checking. Screenshots, videos, and traces pile up in the same folder too.
• Ports and dev servers: the app-under-test binds :3000, the mock API binds :8080, the WebSocket server binds something random-but-not-random-enough. Parallel shards fight for them, get EADDRINUSE, or — worse — connect to a sibling shard's server and quietly test the wrong process.
• Kernel and clock state: containers share the host kernel. A test that runs the clock forward to check a token-expiry flow, exhausts file descriptors launching browsers, or fills the conntrack table with WebDriver connections leaks that into every neighbor. You don't get a stack trace — you get 'the network call sometimes times out.'

Why a container per shard isn't enough

The standard answer is 'run each shard in its own container,' and it genuinely helps — a fresh container gives you a clean filesystem and process namespace, which kills the profile-directory and leftover-process problems for that shard. If that's all you had before, do it. But a container is a process group sharing the host kernel, and browser tests are unusually good at poking the parts a container still shares.

• One kernel, global tunables: sysctls, the conntrack table, fd limits, and the clock are host-wide. A test that steps the clock to expire a session, or a browser storm that exhausts file descriptors, is visible to every other container on the box because there's exactly one kernel underneath them all.
• Host-port arbitration: containers on a shared host network still race for the same host ports, so you end up writing port-allocation logic for your dev server and mock backends — which is now a thing you have to test, in a test suite you were trying to make less flaky.
• Timing under load: page cache, disk, and noisy-neighbor CPU are shared. Browser tests are exquisitely timing-sensitive — a shard that's slow because a neighbor is thrashing the disk fails a 5-second `expect(...).toBeVisible()` that would've passed idle. That's not state leaking, it's timing leaking, and timing is most of what E2E flake actually is.
• A hung test can't always be killed cleanly: reap a container whose browser spawned detached children and you can leave zombies or a wedged cgroup. The isolation boundary you most want — 'make all of this stop, now' — is exactly the one a shared kernel makes fuzzy.

A microVM closes those gaps by not sharing the thing that's leaking. Each PandaStack sandbox is a full Firecracker microVM with its own guest kernel, its own network namespace and full port space (every VM has its own :3000 and :8080, no host-port arbitration), and its own filesystem. A clock skew, an fd exhaustion, a conntrack flood, a corrupted browser profile, a hung renderer — all of it is scoped to one VM you're about to throw away. And 'throw away' is literal: killing the VM takes the whole process tree, ports, and downloads with it, because there's nothing outside the VM to leave behind.

The pattern: bake browsers once, fork per shard

The obvious objection is cost. Browsers and their system dependencies are heavy — a cold `npx playwright install --with-deps` pulls hundreds of megabytes and takes real time, and doing that per shard would be absurd. The microVM model removes that cost the same way it removes boot cost: pay for the browser install exactly once, snapshot the result, then stamp out forks that start from that identical, browsers-already-installed state.

1. Build a 'browsers + deps' base once: create a sandbox, install your app's dependencies, install the browser binaries and their system libraries, and pull down your app or check out the repo — then snapshot it. This is your known-good starting state with Chromium, Firefox, and WebKit already on disk.
2. Fork the base per shard: each shard gets its own microVM that starts from byte-for-byte identical state. A same-host fork is ~400–750ms and uses copy-on-write memory and rootfs, so the Nth shard doesn't recopy the browser binaries — it shares pages with the base until it writes.
3. Run the shard inside its fork: it launches its own browser, binds its own ports, downloads into its own filesystem, and mutates its own profile. Nothing it does is visible to any sibling shard.
4. Tear down by killing the VM, on a hard deadline: set a TTL on the VM and a timeout on the test command. A shard that hangs on a doomed selector gets killed on schedule instead of wedging a runner — and every leftover process, port binding, and half-written download dies with the VM.

Fanning out Playwright shards in Python

Here's the whole shape: build the browsers-installed base once and snapshot it, then fork it per shard, run one Playwright shard in each fork with a hard timeout, pull the trace and any failure screenshots back through the filesystem API, and kill the fork. Set PANDASTACK_API_KEY in your environment first. Each fork is an independent microVM, so this fan-out is safe to run as wide as your hosts have memory for.

from pandastack import Sandbox
from concurrent.futures import ThreadPoolExecutor

# ---- One-time setup: bake a 'browsers + deps installed' base, then snapshot. ----
# Do this once per suite run (cache it across runs keyed on your lockfile +
# the Playwright version). Every shard below forks from this state, so no shard
# ever re-runs `playwright install`.
base = Sandbox.create(template="base", ttl_seconds=1800)
base.exec("git clone --depth 1 https://github.com/acme/webapp.git /app")
base.exec("cd /app && npm ci")                              # app deps, once
base.exec("cd /app && npx playwright install --with-deps")  # browsers, once
seed = base.snapshot()   # Chromium/Firefox/WebKit + deps, frozen
base.kill()

SHARDS = 6  # fan out the suite into 6 slices

def run_shard(index: int) -> dict:
    # Fork = a fresh microVM with the browsers already installed (~400-750ms,
    # copy-on-write). Its browser profile, ports, and /downloads are its own.
    sbx = seed.fork()
    try:
        # A hard timeout is your kill switch for a shard that hangs on a
        # selector that will never resolve. The VM dies on the TTL regardless.
        r = sbx.exec(
            f"cd /app && npx playwright test "
            f"--shard={index}/{SHARDS} "
            f"--trace=on --output=/app/artifacts",
            timeout_seconds=600,
        )
        passed = r.exit_code == 0

        # Pull the trace back out through the filesystem API so you can open it
        # in `npx playwright show-trace` later, even though the VM is gone.
        artifacts = {}
        if not passed:
            trace = sbx.filesystem.read("/app/artifacts/trace.zip")
            with open(f"trace-shard-{index}.zip", "wb") as f:
                f.write(trace)
            artifacts["trace"] = f"trace-shard-{index}.zip"
        return {"shard": index, "passed": passed, "log": r.stdout, **artifacts}
    finally:
        sbx.kill()   # browser tree, ports, downloads — all gone with the VM

with ThreadPoolExecutor(max_workers=SHARDS) as pool:
    results = list(pool.map(run_shard, range(1, SHARDS + 1)))

for res in results:
    print(("PASS" if res["passed"] else "FAIL"), f"shard {res['shard']}/{SHARDS}")

The important properties are in the `finally` and the timeout. Teardown is `sbx.kill()`, full stop — no 'close all browser contexts,' no 'clear the profile dir,' no 'reap orphaned chromedriver' script that's only as correct as the last person who edited it. And the `timeout_seconds` (backed by the VM's TTL) means a shard that hangs forever doesn't hang your pipeline forever: it gets killed and the remaining shards keep going. A hung E2E test is normally a landmine; here it's a VM that gets deleted on schedule.

Reading back traces, screenshots, and videos

The one real cost of running tests in a disposable VM is that the debug artifacts you love — Playwright traces, failure screenshots, Cypress videos — live inside a VM you're about to destroy. The fix is the same read-back pattern the code snippet above uses: the shard writes artifacts to a known directory, and your host pulls them out with `filesystem.read`, which returns raw bytes, before the VM dies. A trace you saved locally opens in `npx playwright show-trace` exactly as if the test had run on your laptop.

# After a shard fails and you've pulled trace-shard-3.zip back to the host,
# open the full time-travel trace locally — DOM snapshots, network, console,
# and the exact action that failed — without ever having touched the VM's shell.
npx playwright show-trace trace-shard-3.zip

# Cypress is the same idea: videos land in cypress/videos and screenshots in
# cypress/screenshots inside the guest. Read them out the same way before kill:
#   data = sbx.filesystem.read("/app/cypress/videos/login.cy.ts.mp4")
#   open("login-shard-3.mp4", "wb").write(data)

For a failing shard you want to inspect interactively rather than just download from, don't kill the VM — keep it alive and shell in. The browser profile, the app logs, the half-finished download, and the frozen dev server are all sitting there exactly as the failure left them, uncontaminated by any sibling shard, because there are no siblings in this VM. The same isolation that made the flake go away is what makes the surviving failure debuggable: nothing else has stomped on the scene.

Shared CI runner vs. a microVM per shard

• Browser profile: shared runner reuses one user-data dir across tests, so sessions and cached auth bleed between them; microVM-per-shard gives each shard a fresh browser and a private profile that's deleted on teardown.
• Leftover processes: shared runner accumulates orphaned browsers and drivers from timed-out tests; each microVM's process tree dies entirely on kill, so no shard inherits a neighbor's zombies.
• Ports: shared runner forces port-allocation logic for the app and mock servers and risks EADDRINUSE or hitting the wrong process; each microVM has its own full port space, so every shard binds :3000 and :8080 freely.
• Downloads and files: shared runner drops downloads and screenshots into shared directories that collide across tests; each microVM has a private filesystem, so a 'download CSV' test can't clobber another's file.
• Kernel and clock: shared runner (and containers) share one kernel, so a clock-skew test or fd exhaustion leaks everywhere; each microVM has its own guest kernel, scoping those changes to one disposable VM.
• Hung tests: shared runner lets a wedged test hold a runner hostage; a microVM has a TTL and a per-command timeout, so a hang is killed on a hard deadline.
• Teardown: shared runner needs a cleanup script (close contexts, clear profile, reap drivers) that silently rots; microVM teardown is 'kill the VM' — every side effect goes with it.
• Cost of a clean start: shared runner pays a slow re-install or scrub for a truly fresh browser; a same-host fork is ~400–750ms with copy-on-write, so a clean, browsers-installed start per shard is cheap.

Parallelism stops being scary

Teams cap E2E parallelism at two or three workers and pray, and it isn't fear of CPUs — it's fear of the contamination wider parallelism exposes. When there's no shared mutable state, that fear evaporates. Fork N and fork N+1 have nothing in common except read-only copy-on-write pages they'll never both write, so you can fan out as wide as your hosts have memory for. The binding constraint becomes memory and CPU, not isolation — the per-agent network design alone pre-allocates 16,384 /30 subnets per agent, so you run out of RAM long before you run out of network namespaces. Add hosts and the suite's wall-clock time drops roughly linearly, because the shards genuinely don't interact. The first shard on a host pays a one-time cold cost to materialize the base (~3s for a first cold boot if the snapshot isn't warm yet); every fork after that is the fast copy-on-write path.

There's a second dividend beyond speed: your E2E suite stops being a source of learned helplessness. The reason engineers hit 're-run failed jobs' on E2E without reading the logs is that they've been trained that the failure is probably environmental noise, not a real bug. Once every shard forks the same snapshot and can't contaminate another, a red shard means something is actually broken — so people start reading the trace instead of gambling on a retry. Removing flake doesn't just save compute; it repairs the suite's credibility.

When a VM per shard is overkill

This is a sledgehammer, and not every test is a walnut. Be honest about where it earns its keep.

• Component and unit tests don't need it. A React component test in jsdom or a pure function assertion is already isolated and runs in milliseconds in-process — don't fork a microVM to check that a button renders.
• One VM per shard, not per test. A single Playwright spec is small; forking a VM per individual test spends more time booting browsers than testing. Slice the suite into shards, fork one VM per shard, and run many specs inside it. You still kill the VM at the shard's end, so no state crosses shards.
• Tests against a shared external system you don't own — a third-party staging API, a shared Stripe test account, a live SSO provider — aren't isolated by your VM. The shared thing is on the other side of the network; a microVM isolates your side, not theirs. Mock it or accept the coupling.
• Suites that already pass deterministically at high parallelism don't need fixing. If your E2E tests are green at -j8 today, you've solved isolation some other way — leave it alone and spend the effort elsewhere.

The sweet spot is the suite everyone reruns without reading: browser E2E tests that launch a real Chromium, log in, download files, bind ports, and fail nondeterministically the moment you parallelize them. For those, a forked-from-snapshot microVM per shard trades a few hundred milliseconds of fork time for the elimination of an entire genre of bug. Fresh browser and clean kernel by construction, artifacts you read back before teardown, a hard timeout for the hangs, and cleanup that's just `kill()` — that's how 'works on my machine' stops being a thing your team says out loud.