Running AI Agent Eval Harnesses in Isolated microVMs

If you're building a coding agent, sooner or later you run an eval harness against it: SWE-bench, terminal-bench, a custom suite of internal tasks, or all three. Each task hands the agent a repo, a broken state, and a hidden test suite, then lets it run arbitrary shell — install deps, apply patches, run pytest — until it either passes the tests or gives up. The number you get out the other side (pass@1, resolve rate) is only meaningful if every task ran in a clean, isolated, reproducible box. An eval where task 7's leftover /tmp state inflates task 8's score is not an eval — it's a rumor. This post is about the substrate that makes the number trustworthy: a fresh microVM per task, forked from a baked snapshot, run hundreds at a time.

I'm Ajay — I built PandaStack, so I have a bias, but the isolation argument here is substrate-agnostic and applies whether you use us, plain Firecracker, or Docker with a lot of discipline. I'll show the eval-runner shape with the PandaStack SDK, be specific about where per-task isolation actually matters, and give you the honest comparison against a local Docker matrix and a shared CI runner.

Why per-task isolation is non-negotiable

An agent under eval runs commands you did not write and cannot review — that's the whole point of measuring it. It will `pip install` conflicting versions, write to global config, leave daemons running, mutate the system clock, fill the disk, or fork a process that outlives the task. None of that is malicious; it's just what real agents do when they're flailing toward a solution. The problem is what happens next: if task N+1 runs in the same environment, its result is now a function of task N's mess. Your resolve rate becomes a measurement of leftover state, not agent capability.

The failure modes are subtle and directional, which is the dangerous part:

• State leakage inflates scores: task N leaves a cached wheel, a built artifact, or a warmed dependency that makes task N+1 pass when a cold environment would fail it.
• State leakage deflates scores: task N corrupts a global (pins a bad numpy, breaks the git config, leaves a lock file) and task N+1 fails through no fault of the agent.
• Non-determinism kills reproducibility: run the same suite twice in a shared box and the ordering of tasks changes the results, so nobody can reproduce your headline number — including you, next week.
• Resource contention corrupts timing and OOM behavior: two tasks sharing a host fight over CPU and RAM, so an agent that would pass under its budget gets OOM-killed because a neighbor spiked.

The fix is boring and absolute: one disposable environment per task attempt, created from an identical starting state, destroyed after. A microVM is a clean fit because the isolation boundary is a real one — its own guest kernel, its own filesystem, its own network namespace — so a task that does `rm -rf /`, forks a runaway process, or eats all the RAM is contained to a VM you were going to throw away anyway. This is the same argument as running any untrusted code in a sandbox; evals just make the untrusted-code volume enormous and the correctness stakes explicit. See /blog/parallel-test-isolation-microvms for the same reasoning applied to a test matrix.

Reproducibility: bake the task environment once, fork per attempt

Cold-booting and re-provisioning a full environment for every one of hundreds of tasks is slow and — worse — not actually reproducible, because `apt-get`, `pip install`, and `git clone` all reach out to a network that changes underneath you. A mirror goes down, a package yanks a version, a base image gets a new patch, and your "identical" environment silently drifts. The reproducible move is to do the expensive, network-dependent setup exactly once, freeze it, and start every attempt from that frozen point.

On PandaStack that's a snapshot. You provision one sandbox per task, install the repo's dependencies and check out the exact commit, then snapshot it. Every attempt of that task is a fork of the snapshot: a copy-on-write clone that boots into the identical post-setup state — same installed packages, same files, same everything — without re-running any of the setup. A same-host fork is 400–750ms and shares the parent's memory and disk copy-on-write, so N attempts cost N cheap forks, not N cold provisions. The snapshot is your reproducibility contract: as long as you pin the snapshot ID, the starting state is byte-identical across runs, machines, and weeks. /blog/snapshot-and-fork-explained goes deeper on the mechanics.

from pandastack import Sandbox

# ---- Phase 1: bake the task environment ONCE ----
# Provision, do all the slow network-dependent setup, then snapshot.
def bake_task_env(task) -> str:
    sbx = Sandbox.create(template="agent", ttl_seconds=1800)
    try:
        # Clone the exact commit and install deps: the slow, flaky, networked part.
        sbx.exec(
            f"git clone --depth 1 {task['repo_url']} /workspace/repo",
            timeout_seconds=120,
        )
        sbx.exec(
            f"cd /workspace/repo && git checkout {task['base_commit']}",
            timeout_seconds=30,
        )
        sbx.exec(
            "cd /workspace/repo && pip install -e '.[test]'",
            timeout_seconds=600,
        )
        # Freeze this post-setup state. Every attempt forks from here.
        snap = sbx.snapshot()
        return snap.id
    finally:
        sbx.kill()  # the bake VM is disposable; the snapshot is what we keep

`snapshot()` returns a handle whose `id` you fork from later. The bake sandbox itself is torn down immediately — you only wanted the frozen state, not the VM. Do this once per task (not once per attempt), cache the snapshot IDs keyed by task, and you've turned a flaky multi-minute provision into a sub-second fork you can repeat deterministically.

Running N attempts of a task in parallel forks

Two kinds of parallelism matter in an eval harness. Across tasks: you have 300 SWE-bench instances and want them running concurrently, not sequentially over six hours. Within a task: for pass@k or best-of-N you run the same task k times with different samples and see how often the agent lands it. Both reduce to the same primitive — fork the task's snapshot, hand the fork to one attempt, score it, kill it — and forks are cheap enough that you can fan out wide without a warm pool of idle VMs sitting around costing money.

Here's a runner that forks a task snapshot k times and runs the attempts concurrently. Each attempt gets its own VM, so nothing leaks between the k samples of the same task either — pass@k is only honest if attempt 2 can't see attempt 1's files.

import concurrent.futures as cf
from pandastack import Sandbox

def run_one_attempt(snapshot_id: str, task, agent, attempt_idx: int) -> dict:
    """Fork the baked snapshot, let the agent try, run the hidden tests, score."""
    sbx = Sandbox.fork(snapshot_id, ttl_seconds=900)  # ~400-750ms same-host
    try:
        # Let the agent drive shell commands inside its private VM.
        # agent.solve() calls sbx.exec(...) / sbx.filesystem.write(...) internally.
        agent.solve(sbx, task)

        # Apply the hidden test patch and run the graded suite.
        sbx.filesystem.write("/workspace/repo/test_patch.diff", task["test_patch"])
        sbx.exec(
            "cd /workspace/repo && git apply test_patch.diff",
            timeout_seconds=30,
        )
        result = sbx.exec(
            f"cd /workspace/repo && {task['test_cmd']}",
            timeout_seconds=600,
        )
        return {
            "task_id": task["id"],
            "attempt": attempt_idx,
            "passed": result.exit_code == 0,
            "exit_code": result.exit_code,
            "stdout": result.stdout[-8000:],  # tail is where the pytest summary lives
            "stderr": result.stderr[-4000:],
        }
    except Exception as e:  # a crashed attempt is a fail, not a lost run
        return {"task_id": task["id"], "attempt": attempt_idx,
                "passed": False, "error": repr(e)}
    finally:
        sbx.kill()  # teardown is not optional; leaked VMs cost real memory

def run_task(task, agent, k: int = 5) -> list[dict]:
    snapshot_id = bake_task_env(task)  # from Phase 1, cache this per task
    with cf.ThreadPoolExecutor(max_workers=k) as pool:
        futures = [
            pool.submit(run_one_attempt, snapshot_id, task, agent, i)
            for i in range(k)
        ]
        return [f.result() for f in cf.as_completed(futures)]

A few things worth calling out. The SDK client is I/O-bound (it's talking to the API and blocking on `exec`), so a thread pool gets you real concurrency without asyncio — but the SDK works fine under `asyncio.to_thread` too if the rest of your harness is async. Every attempt is a fresh `Sandbox.fork`, and every attempt ends in `sbx.kill()` inside a `finally`, so a crashed agent or a timed-out test still tears down its VM. A leaked VM isn't a correctness bug, but at 300 tasks times 5 attempts it's a very expensive one. To scale the across-task fan-out, wrap `run_task` in an outer pool with a bounded worker count so you don't try to hold a thousand VMs at once — the fork is fast, but your memory budget is the real ceiling.

Capturing pass/fail, logs, and teardown

The graded signal is almost always the exit code of the test command — pytest, cargo test, go test all exit non-zero on failure — so `result.exit_code == 0` is your pass/fail. But the exit code alone is useless when you're debugging why the agent regressed on 12 tasks overnight, so capture more:

• Test output: keep the tail of stdout/stderr — that's where the pytest summary line and the tracebacks live. Store it per attempt, not per task, so pass@k failures are individually inspectable.
• The agent's transcript: log every command the agent ran and its output. When a task flips from pass to fail across runs, the diff in the agent's command sequence is the first thing you'll want.
• The produced patch: `git diff` inside the fork before you apply the hidden tests captures exactly what the agent changed — the artifact you'll re-apply, review, or ship to a leaderboard.
• Timing and resource ceilings: record wall-clock per attempt and whether the test command hit its timeout, so you can tell a genuine fail from a too-tight budget.

Pull the agent's patch out with the filesystem API before teardown, the same way you'd read any artifact out of a sandbox — write it to a known path, read the bytes back, done:

# Inside run_one_attempt, before applying the hidden tests:
sbx.exec(
    "cd /workspace/repo && git diff > /workspace/agent.patch",
    timeout_seconds=30,
)
patch_bytes = sbx.filesystem.read("/workspace/agent.patch")
# Persist it on the host for review / leaderboard submission / regression triage.
with open(f"artifacts/{task['id']}_attempt{attempt_idx}.patch", "wb") as f:
    f.write(patch_bytes)

Teardown is the step people skip and regret. Every fork must be killed — on success, on failure, and on the exception path — which is why `sbx.kill()` belongs in a `finally`. Set a `ttl_seconds` on the fork too, as a backstop: if your harness process crashes mid-run, the TTL reaps the orphaned VMs so a failed eval run doesn't leave a fleet of zombie sandboxes billing you until you notice. Defense in depth: explicit kill for the normal path, TTL for the day your harness segfaults.

Local Docker matrix vs shared CI runner vs microVM-per-task

These are the three shapes teams actually reach for. All three can produce a number; they differ on how much you can trust it and how fast you get it.

• Isolation — Docker matrix: containers share the host kernel, so a kernel-level side effect, a runaway process, or a resource spike can cross the boundary. CI runner: worst — tasks share a long-lived machine and its accumulated state. microVM-per-task: hardware-isolated guest kernel per task, the real boundary.
• Reproducibility — Docker matrix: good if you pin image digests and disable the network, but re-provisioning per task still hits flaky mirrors. CI runner: poor — task order and leftover state make runs non-deterministic. microVM-per-task: strongest — every attempt forks a byte-identical snapshot, so the start state is pinned across runs and weeks.
• State leakage risk — Docker matrix: low if every task gets a fresh container, but shared volumes and caches quietly reintroduce it. CI runner: high — the whole point of a CI runner is to be reused, which is exactly wrong for evals. microVM-per-task: none — a killed fork takes all its state with it.
• Parallelism / density — Docker matrix: bounded by one host's cores and RAM; the container-per-task model doesn't fan out across machines on its own. CI runner: bounded by your runner fleet and usually serialized per runner. microVM-per-task: fork is cheap and schedules across a fleet, so hundreds of concurrent tasks is a config number, not a rebuild.
• Setup cost per task — Docker matrix: re-run install/build per container, or share a cache and reintroduce leakage. CI runner: often a warm checkout, which is fast precisely because it's contaminated. microVM-per-task: bake once, fork in 400-750ms, so clean and fast stop being a trade-off.
• Best fit — Docker matrix: small suites on a trusted first-party agent where you control the code. CI runner: never, for scored evals — fine for a smoke test, not for a resolve rate. microVM-per-task: large, parallel, reproducible eval suites where the number has to survive scrutiny.

The honest summary: a Docker matrix with fresh-container-per-task and pinned digests is a defensible eval substrate and plenty of teams run SWE-bench that way — verify the isolation and reproducibility claims against Docker's own docs for your setup, since a lot depends on how disciplined your volume and cache config is. A shared CI runner is the one to avoid for scored evals; its entire design goal (reuse a warm machine) is the failure mode you're trying to eliminate. The microVM-per-task model wins when the suite is large, the parallelism is real, and you need the starting state pinned hard enough that the number reproduces.

Putting it together

The pattern is small and it composes: bake each task's environment once into a snapshot, fork the snapshot per attempt, run attempts in parallel across a thread pool, capture exit code plus logs plus the produced patch, and kill every fork in a `finally` with a TTL backstop. The isolation is what makes the resolve rate mean something; the snapshot is what makes it reproducible next week; the fork is what makes running hundreds of tasks concurrently cheap instead of a warm-pool budget line. If you want the same fan-out pattern applied to agents exploring solution branches rather than eval tasks, /blog/agent-swarm-parallel-sandboxes covers that shape. Either way, the rule that keeps your evals honest is the unglamorous one: a fresh box per task, every time, no exceptions — because the moment two tasks share a filesystem, your score is measuring the filesystem.