A microVM Harness for SWE-bench & terminal-bench

A terminal-agent benchmark is a specific and unforgiving thing. You hand a coding agent a repository, a broken or incomplete state, and a task description, and then you let it drive a real shell: it runs `git checkout`, `pip install`, `make`, `pytest`, edits files, reruns the tests, and iterates until it either turns the hidden test suite green or burns its step budget. SWE-bench and terminal-bench are the well-known public shapes; most teams also carry an internal suite of tasks that look exactly like their real work. The score that falls out — resolve rate, pass@1 — is only worth publishing if every task ran in a box the agent could trash without touching any other task. This post is about that box: one microVM per task, a snapshot for a clean reset, the full command trace captured, the exit code graded.

I'm Ajay, I built PandaStack, so treat the SDK examples as one concrete way to do this — the isolation argument holds whether you run plain Firecracker, a disciplined Docker setup, or us. What I want to be precise about is the part that's specific to terminal benchmarks and not to evals in general: the agent is a shell operator, the graded artifact is a command's exit code, and the thing you most need to keep is the trace of what it actually typed.

The agent is a shell operator, and that's the hard part

The reason a terminal benchmark needs harder isolation than a text eval is that the graded unit of work is a shell command, not a token. The agent doesn't describe a fix — it runs one. That means it mutates a real filesystem, installs real packages, spawns real processes, and touches real global state, all in ways you cannot review before they happen because reviewing them is the thing you're measuring.

Watch what a flailing agent does to a shared box over a few hundred tasks:

• It pip-installs a conflicting version of a dependency globally, and the next task inherits the wrong pin.
• It runs a build that leaves a compiled artifact or a warmed cache on disk, so the next task passes a step it should have had to earn.
• It backgrounds a dev server or a watcher that outlives the task and holds a port the next task wants.
• It writes to ~/.gitconfig, ~/.npmrc, or /etc, and every subsequent task now runs against a subtly different environment.
• It runs a genuinely destructive command — rm -rf on the wrong path, a disk-filling loop, a fork bomb — because that is exactly the kind of mistake a model under pressure makes.

The fix is unglamorous and absolute: one disposable environment per task, from an identical starting state, destroyed after. A microVM fits because its boundary is real — its own guest kernel, its own filesystem, its own network namespace — so the agent's `rm -rf /`, its runaway process, and its 40GB allocation are all contained to a VM you were going to throw away anyway. Sharing a kernel (a plain container) means a kernel-level side effect or a resource spike can cross the line; sharing a whole machine (a CI runner) means everything crosses the line. For the substrate reasoning in general, see /blog/why-docker-is-not-a-sandbox.

Snapshot the prepared task, fork it for a clean reset

Every terminal-benchmark task has an expensive, flaky setup phase before the agent ever gets the keyboard: clone the repo at a pinned commit, install the exact dependencies, apply the task's setup patch. That phase reaches out to a network that drifts underneath you — a mirror goes down, a version gets yanked, a base image gets re-patched — so re-running it per task is both slow and quietly non-reproducible. The move is to do it once, freeze the result, and start every attempt from the frozen point.

On PandaStack that frozen point is a snapshot, and the clean reset is a fork of it. You bake the task environment into one sandbox, snapshot it, and then every attempt is a copy-on-write clone that boots into the identical post-setup state — same packages, same files, same git HEAD — without rerunning any setup. A same-host fork is 400–750ms and shares the parent's memory and disk copy-on-write, so a clean reset between attempts costs a sub-second fork rather than a multi-minute reprovision. The snapshot ID is your reproducibility contract: pin it, and the starting state is byte-identical across runs, machines, and the version of you that reruns this next quarter. /blog/snapshot-and-fork-explained covers the mechanics.

from pandastack import Sandbox

# ---- Bake each task's prepared shell environment ONCE ----
# Do the slow, networked setup here so no scored attempt ever pays for it.
def bake_task(task) -> str:
    sbx = Sandbox.create(template="agent", ttl_seconds=1800)
    try:
        # Clone the exact commit and install deps: git, pip, the flaky networked part.
        sbx.exec(
            f"git clone --depth 1 {task['repo_url']} /workspace/repo",
            timeout_seconds=120,
        )
        sbx.exec(
            f"cd /workspace/repo && git checkout {task['base_commit']}",
            timeout_seconds=30,
        )
        sbx.exec(
            "cd /workspace/repo && pip install -e '.[test]' && make deps",
            timeout_seconds=600,
        )
        # Write the task prompt into the guest so the agent reads it from disk.
        sbx.filesystem.write("/workspace/TASK.md", task["prompt"])

        # Freeze the prepared state. Every attempt forks from here for a clean reset.
        snap = sbx.snapshot()
        return snap.id
    finally:
        sbx.kill()  # the bake VM is disposable; the snapshot is the keepsake

Capture the full command trace, then grade the exit code

This is the part specific to terminal benchmarks. The agent's whole run is a sequence of shell commands, and the two things you need out of each attempt are (1) the graded signal — almost always the exit code of the test command, since pytest, go test, and cargo test all exit non-zero on failure — and (2) the trace of every command the agent ran and what it printed back. The exit code tells you whether it passed; the trace is the only thing that tells you why it passed or regressed when you're staring at a diff of last night's run against tonight's.

The harness drives one attempt like this: fork the baked snapshot, let the agent run its commands through `sbx.exec` (logging each one), pull the produced patch out with the filesystem API, apply the hidden test patch, and grade. Every `exec` call is a natural trace point — you already have the command string, the stdout, the stderr, and the exit code in hand.

import concurrent.futures as cf
from pandastack import Sandbox

def traced_exec(sbx, trace: list, cmd: str, timeout_seconds: int = 120):
    """Run a shell command in the task's VM and record it in the trace."""
    r = sbx.exec(cmd, timeout_seconds=timeout_seconds)
    trace.append({"cmd": cmd, "exit": r.exit_code,
                  "out": r.stdout[-2000:], "err": r.stderr[-2000:]})
    return r

def run_attempt(snapshot_id: str, task, agent, i: int) -> dict:
    sbx = Sandbox.fork(snapshot_id, ttl_seconds=900)  # ~400-750ms clean reset
    trace: list = []
    try:
        # The agent drives the shell. It reads /workspace/TASK.md and issues
        # commands; route each one through traced_exec so we keep the record.
        agent.solve(task, run=lambda c: traced_exec(sbx, trace, c))

        # Snapshot what the agent changed BEFORE we apply the hidden tests.
        traced_exec(sbx, trace, "cd /workspace/repo && git diff > /workspace/agent.patch", 30)
        patch = sbx.filesystem.read("/workspace/agent.patch")

        # Apply the graded test patch and run the hidden suite. Exit code = pass/fail.
        sbx.filesystem.write("/workspace/repo/test.diff", task["test_patch"])
        traced_exec(sbx, trace, "cd /workspace/repo && git apply test.diff", 30)
        graded = traced_exec(sbx, trace,
                             f"cd /workspace/repo && {task['test_cmd']}", 600)
        return {"task": task["id"], "attempt": i,
                "passed": graded.exit_code == 0, "trace": trace,
                "patch": patch.decode("utf-8", "replace")}
    except Exception as e:  # a crashed attempt is a fail, not a lost run
        return {"task": task["id"], "attempt": i, "passed": False,
                "error": repr(e), "trace": trace}
    finally:
        sbx.kill()  # every fork must die: on pass, fail, and exception

def run_task(task, agent, k: int = 5) -> list[dict]:
    snap = bake_task(task)  # cache this per task; bake once, fork k times
    with cf.ThreadPoolExecutor(max_workers=k) as pool:
        futs = [pool.submit(run_attempt, snap, task, agent, i) for i in range(k)]
        return [f.result() for f in cf.as_completed(futs)]

A few load-bearing details. Every command the agent runs goes through `traced_exec`, so `result["trace"]` is a replayable record of the whole attempt — when a task flips from pass to fail across runs, diffing the two command sequences is the fastest way to see what changed. The patch is read out with `filesystem.read` before the hidden tests are applied, so it's exactly what the agent produced, uncontaminated by the grading step — that's your leaderboard artifact and your regression-triage input. And every fork ends in `sbx.kill()` inside a `finally`: a crashed agent or a timed-out test still tears down its VM. A leaked VM isn't a correctness bug, but at 300 tasks times 5 attempts it's an expensive one.

Running the suite in parallel

Two axes of parallelism show up in a terminal benchmark. Across tasks: 300 SWE-bench instances should run concurrently, not sequentially over six hours. Within a task: pass@k reruns the same task k times with different samples to see how reliably the agent lands it. Both reduce to the same primitive — fork the task's snapshot, hand the fork to one attempt, grade it, kill it — and because forks restore a snapshot on demand with no warm pool of idle VMs, you pay for VMs only while attempts run and effectively nothing between benchmark runs. That matters when a suite runs on a cron a few times a day rather than continuously.

The SDK client is I/O-bound — it blocks on `exec` against the API — so a `ThreadPoolExecutor` gives real concurrency without asyncio (and it works fine under `asyncio.to_thread` if the rest of your harness is async). To fan out across tasks, wrap `run_task` in an outer bounded pool: the fork is cheap, but your host memory budget is the real ceiling, so cap concurrent VMs rather than trying to hold a thousand at once. The hard structural ceiling on a single agent is 16,384 concurrent microVMs (the /30 subnet space); the practical ceiling you'll hit long before that is RAM.

Shared runner vs container-per-task vs microVM-per-task

These are the three substrates teams actually reach for. All three produce a number; they differ on how much you can trust it and how fast you get it.

• Isolation — Shared CI runner: worst; tasks share a long-lived machine and all its accumulated state. Container-per-task: containers share the host kernel, so a kernel-level side effect or resource spike can cross the boundary. microVM-per-task: hardware-isolated guest kernel, filesystem, and network per task — the real boundary the agent's shell can't cross.
• Reproducibility — Shared CI runner: poor; task order and leftover state make runs non-deterministic. Container-per-task: good if you pin image digests and disable the network, but reprovisioning per task still hits flaky mirrors. microVM-per-task: strongest; every attempt forks a byte-identical snapshot, so the start state is pinned across runs and weeks.
• State leakage between tasks — Shared CI runner: high; reuse of a warm machine is the whole design, and exactly wrong for a benchmark. Container-per-task: low with fresh-container-per-task, but shared volumes and caches quietly reintroduce it. microVM-per-task: none; a killed fork takes all its state with it.
• Command-trace fidelity — Shared CI runner: muddy; the trace mixes with other tasks' side effects on the shared box. Container-per-task: clean per container if you log each exec. microVM-per-task: clean and attributable; each attempt's trace belongs to exactly one VM.
• Parallelism & density — Shared CI runner: bounded by your runner fleet and usually serialized per runner. Container-per-task: bounded by one host's cores and RAM. microVM-per-task: forks are cheap and schedule across a fleet, so hundreds of concurrent tasks is a config number, not a rebuild.
• Best fit — Shared CI runner: never, for a scored benchmark; fine for a smoke test. Container-per-task: small suites on a trusted first-party agent where you control the code. microVM-per-task: large, parallel, reproducible benchmark suites where the number has to survive scrutiny.

The honest summary: a container-per-task setup with fresh containers and pinned image digests is a defensible way to run SWE-bench, and plenty of teams do — verify the isolation and reproducibility specifics against the container runtime's own docs for your config, since how disciplined your volume and cache setup is decides whether leakage sneaks back in. The shared CI runner is the one to avoid for scored benchmarks; its entire design goal (reuse a warm machine) is precisely the failure mode you're eliminating. The microVM-per-task model wins when the suite is large, the parallelism is real, and you need the starting state pinned hard enough that the number reproduces.

The task setup and grade, as a shell script

It helps to see the shape of a single task's setup-and-grade as plain shell, independent of the harness plumbing — this is what runs inside the guest. The setup half is what you bake into the snapshot; the grade half is what runs after the agent has had its turn.

#!/usr/bin/env bash
set -euo pipefail

REPO=/workspace/repo

# ---- SETUP (runs once, baked into the snapshot) ----
git clone --depth 1 "$REPO_URL" "$REPO"
cd "$REPO"
git checkout "$BASE_COMMIT"
pip install -e '.[test]'
make deps

# ---- GRADE (runs after the agent's attempt, inside the fork) ----
cd "$REPO"
# Capture exactly what the agent changed, before touching the tests.
git diff > /workspace/agent.patch

# Apply the HIDDEN test patch the agent never saw, then run the suite.
git apply /workspace/tests/test_patch.diff
if pytest -q --timeout=120 tests/; then
  echo "RESULT: pass"
  exit 0
else
  echo "RESULT: fail"
  exit 1
fi

The exit code of that script is the graded signal; the harness reads it back through `exec`. Keeping the hidden test patch out of the snapshot and applying it only at grade time is what stops an agent from reading the tests and gaming them — the agent's fork contains the repo and the task, never the answer key.

Putting it together

The whole harness is small and it composes: bake each task's prepared shell environment once into a snapshot, fork it per attempt for a clean reset, let the agent drive the shell while you trace every command, read the produced patch out before grading, run the hidden tests for the exit code, and kill every fork in a `finally` with a TTL backstop. The per-VM isolation is what makes the resolve rate mean something; the snapshot is what makes it reproducible next week; the trace is what makes a regression debuggable; and the cheap fork is what makes running hundreds of tasks concurrently a config number instead of a warm-pool budget line. For the closely related eval-harness framing, see /blog/ai-agent-eval-harness-sandbox; for fanning forks out across agents exploring solution branches, /blog/agent-swarm-parallel-sandboxes. The rule that keeps the benchmark honest is the boring one: a fresh box per task, every time — because the moment two tasks share a shell, your score is measuring the shell.