Running a SWE-bench Evaluation Harness on microVMs

SWE-bench looks like a benchmark. You read the paper, you look at the leaderboard, you form opinions about which model is best at fixing real GitHub issues. Then you try to run it yourself and discover that the interesting part was never the scoring function. It is the environments. Each instance wants a specific repository at a specific commit with a specific, frequently ancient dependency set that resolved cleanly in 2021 and does not resolve cleanly now. You have to construct that environment faithfully, hand it to a model, take back a patch, apply it, and run a test suite that the model has every incentive to defeat by means other than fixing the bug. Do all of that a few hundred times, in parallel, and produce one number you are willing to put your name on.

I am Ajay, and I build PandaStack, so treat the product bits accordingly. The infrastructure argument here is substrate-agnostic and mostly applies whether you run this on Firecracker, on Docker with a lot of discipline, or on a pile of EC2 instances and hope. Do not take my word for the benchmark specifics either — instance counts, the exact composition of SWE-bench Verified, and current resolve rates all move, so check the dataset and the live leaderboard rather than a blog post. What follows is about the harness underneath the number.

An infrastructure problem in a benchmark costume

Strip out the machine learning and a SWE-bench run is a very demanding CI system with three unusual properties. First, every job needs a different environment, and those environments are historical: the dependency set that the repo's maintainers used at that commit, not the one pip would give you today. Second, the code under test is written by an adversary — not a malicious one, but an optimiser, which is close enough. Third, the output is a scalar that people will compare against other people's scalars, which means any environmental sloppiness on your side becomes someone else's wrong conclusion.

Those three properties push in the same direction: you want one disposable, faithfully reconstructed, network-controlled environment per instance, created from an identical starting state every time, and destroyed the moment grading finishes. Everything below is a consequence of that sentence.

Per-instance environment reproducibility is the whole ballgame

The naive harness builds a container image per instance: clone the repo, check out the base commit, install the pinned dependencies, run the test suite once to confirm the fail-to-pass tests actually fail. That is the correct logic and it is agonisingly slow, because you are re-running the flaky, network-dependent part of the pipeline for every instance, every time you touch the harness. Worse, it is not actually reproducible. A mirror goes down, a wheel gets yanked, a transitive dependency publishes a new patch release, a base image gets rebuilt — and the environment you called identical last month has quietly drifted.

The move is to separate construction from execution. Build each instance's environment exactly once, freeze the whole machine at that moment, and start every subsequent run from the frozen point. On PandaStack that freeze is a Firecracker snapshot: memory and disk captured together, restored copy-on-write. A create from a baked snapshot lands around 179ms at p50 and 203ms at p99 — the restore step itself is roughly 49ms — versus roughly 3 seconds for a genuine cold boot. Forking an existing snapshot on the same host is 400 to 750ms, or 1.2 to 3.5 seconds if the fork has to cross hosts.

The practical effect is that "rebuild the environment" stops being a thing you avoid doing and becomes a thing you do several hundred times a minute without thinking about it.

from pandastack import Sandbox


def bake_instance(instance: dict) -> tuple[str, str]:
    """Construct one SWE-bench instance environment ONCE and freeze it.

    Everything slow, networked and flaky happens here. Grading runs never
    repeat it -- they fork the snapshot this returns.
    """
    sbx = Sandbox.create(template="code-interpreter", ttl_seconds=2400)
    try:
        repo = instance["repo"]           # e.g. "django/django"
        commit = instance["base_commit"]  # the exact pre-fix commit

        sbx.exec(
            f"git clone https://github.com/{repo}.git /workspace/repo",
            timeout_seconds=600,
        )
        sbx.exec(
            f"cd /workspace/repo && git checkout -q {commit}",
            timeout_seconds=60,
        )

        # The historical dependency set. This is the part that rots.
        for cmd in instance["install_commands"]:
            sbx.exec(f"cd /workspace/repo && {cmd}", timeout_seconds=1800)

        # Sanity gate: the fail-to-pass tests must actually FAIL here.
        # If they pass before any patch, the instance is mis-specified and
        # every model will "resolve" it. Catch that now, not on the leaderboard.
        pre = sbx.exec(
            f"cd /workspace/repo && {instance['test_cmd']}",
            timeout_seconds=1800,
        )
        if pre.exit_code == 0:
            raise RuntimeError(f"{instance['id']}: baseline unexpectedly green")

        # Record the pre-patch tree hash so grading can prove what changed.
        tree = sbx.exec("cd /workspace/repo && git rev-parse HEAD^{tree}")
        snap = sbx.snapshot()
        return snap.id, tree.stdout.strip()
    finally:
        sbx.kill()  # the construction VM is scaffolding; the snapshot is the artifact

The baseline check in the middle is not optional. An instance whose fail-to-pass tests already pass at the base commit is free score for every model you evaluate, and it will not announce itself — it just makes your harness look generous. Catching it during the bake, once, is much cheaper than discovering it after someone quotes your number.

The model's patch is untrusted code, and it knows what it is being graded on

This is the part harness authors underrate. A SWE-bench patch is a diff produced by a system that has been optimised to make a test command exit zero. It has not been optimised to fix the bug. Those two objectives correlate — that is why the benchmark works at all — but they are not the same objective, and the gap is where your grading pipeline lives or dies.

The failure modes are not hypothetical, and every team that runs this at scale has seen some of them:

• The patch edits the tests. Deleting the failing assertion is a spectacularly effective way to make a test suite pass, and it is the first thing you should mechanically forbid rather than hope against.
• The patch edits conftest.py, a pytest plugin, tox.ini, or setup.cfg. Now the collection rules, the fixtures, or the markers have changed, and the suite you are running is not the suite you meant to run.
• The patch monkeypatches the runner. A sitecustomize.py, a conftest hook, or a module-level import side effect that rewrites assertions or force-passes a test is well within what a capable model will emit when the direct fix is hard.
• The patch reaches the network. Fetching the upstream fix from GitHub is, from the model's perspective, a completely reasonable strategy, and it is also the exact thing your benchmark is supposed to be measuring.
• The patch does something expensive and stupid. Infinite loops, forkbombs, filling the disk with a log file, spawning a process that outlives the run. Usually incompetence rather than malice, but the blast radius is identical.
• The patch does something genuinely hostile. Rare, but the moment you run patches from a third-party model you did not train, on shared infrastructure, you are executing code of unknown provenance at scale. That is a supply-chain posture, not a benchmark detail.

Two defences, and you want both. The first is mechanical: refuse to apply patches that touch grading-relevant paths, and check it rather than prompting for it. The second is architectural: run each instance behind a boundary strong enough that "did it cheat" is a question you can actually answer from outside the box. On a shared kernel, distinguishing a well-behaved patch from a badly-behaved one means trusting instrumentation that the patch could, in principle, have influenced. With a microVM per instance you have a separate guest kernel, filesystem, and network namespace, so the host's view of what happened is not something the guest gets a vote on.

Grading untrusted patches on a shared kernel and calling the resulting number science is a choice. It might even be the right choice for your threat model. It should be a choice you made on purpose.

Here is the mechanical half, as a guard script you run inside the instance sandbox before the graded suite. It is deliberately boring:

#!/usr/bin/env bash
# guard.sh -- run INSIDE the instance sandbox, after applying the model patch
# and BEFORE applying the hidden test patch. Exit non-zero = instance voided.
set -euo pipefail

cd /workspace/repo

# 1. Nothing the model wrote may touch grading-relevant files.
FORBIDDEN='^(tests?/|test/|.*/tests?/|conftest\.py|tox\.ini|setup\.cfg|pytest\.ini|pyproject\.toml|sitecustomize\.py|.*_test\.py|test_.*\.py)'
if git diff --name-only HEAD | grep -Eq "$FORBIDDEN"; then
  echo "VOID: model patch modifies test or runner configuration" >&2
  git diff --name-only HEAD | grep -E "$FORBIDDEN" >&2
  exit 65
fi

# 2. No new files outside the source tree. A stray sitecustomize.py anywhere
#    on sys.path is a passing grade with extra steps.
if git status --porcelain --untracked-files=all | grep -Eq '^\?\? .*(sitecustomize|conftest)'; then
  echo "VOID: model patch introduces an interpreter/collection hook" >&2
  exit 65
fi

# 3. Prove egress is actually off. Do not assume the policy applied.
if curl -sS --max-time 3 -o /dev/null https://pypi.org/simple/ 2>/dev/null; then
  echo "VOID: network reachable during grading" >&2
  exit 66
fi

echo "guard: ok"

Point three matters more than it looks. Every harness I have seen asserts that the network is off; a minority verify it from inside the box on every single run. Verify it. A misapplied policy is silent, and its symptom is a suspiciously good score.

Grading one instance, end to end

With the environment frozen and the guard written, one instance's grading run is short. Fork the snapshot, write the model's patch, apply it, run the guard, apply the hidden test patch, run the suite, collect artifacts, kill the VM. Every step that can fail produces a distinct outcome, because "failed" and "voided" and "errored" are three different things and collapsing them into one number is how harnesses start lying to you.

import json
from pandastack import Sandbox

GUARD = open("guard.sh").read()


def grade(instance: dict, snapshot_id: str, model_patch: str) -> dict:
    """Fork the frozen instance env, apply an untrusted patch, score it."""
    sbx = Sandbox.fork(snapshot_id, ttl_seconds=1800)  # 400-750ms same-host
    out = {"instance_id": instance["id"], "snapshot_id": snapshot_id}
    try:
        # 1. The model's patch. Untrusted input -- write it, never eval it.
        sbx.filesystem.write("/workspace/model.patch", model_patch)
        applied = sbx.exec(
            "cd /workspace/repo && git apply -v /workspace/model.patch",
            timeout_seconds=60,
        )
        if applied.exit_code != 0:
            out["status"] = "unapplied"   # not a fail: the patch never ran
            out["stderr"] = applied.stderr[-4000:]
            return out

        # 2. Mechanical anti-gaming guard, before the graded suite exists.
        sbx.filesystem.write("/workspace/guard.sh", GUARD)
        guard = sbx.exec("bash /workspace/guard.sh", timeout_seconds=60)
        if guard.exit_code != 0:
            out["status"] = "void"        # cheated, or the sandbox misbehaved
            out["guard"] = guard.stderr[-4000:]
            return out

        # 3. Only NOW does the hidden test patch land, so the model never
        #    saw the tests it is being graded against.
        sbx.filesystem.write("/workspace/tests.patch", instance["test_patch"])
        sbx.exec(
            "cd /workspace/repo && git apply /workspace/tests.patch",
            timeout_seconds=60,
        )

        run = sbx.exec(
            f"cd /workspace/repo && {instance['test_cmd']}",
            timeout_seconds=1800,
        )
        out["status"] = "resolved" if run.exit_code == 0 else "failed"
        out["exit_code"] = run.exit_code
        out["stdout_tail"] = run.stdout[-16000:]  # the summary lives at the end
        out["stderr_tail"] = run.stderr[-8000:]

        # 4. Artifacts, before the box evaporates.
        sbx.exec("cd /workspace/repo && git diff HEAD > /workspace/final.diff")
        out["final_diff"] = sbx.filesystem.read("/workspace/final.diff").decode()
        return out
    except Exception as exc:
        out["status"] = "error"           # harness fault, NOT a model failure
        out["error"] = repr(exc)
        return out
    finally:
        sbx.kill()  # teardown in finally, always; TTL is only the backstop

Note the ordering: the hidden test patch is applied after the guard, which means the model's patch was written against a tree that did not contain the graded tests. If you apply the tests first because it is more convenient, you have handed the model the answer key and your resolve rate is measuring reading comprehension.

Note also that "unapplied", "void", "failed", and "error" are separate statuses. A patch that does not apply is a model failure of a specific kind. A voided instance is a cheating signal you should count and report. An error is your bug. Reporting all four as "not resolved" makes your harness less useful to you than to whoever reads the headline.

Fan-out: hundreds of instances, each wanting a clean machine

A full SWE-bench run means hundreds of instances, and if you are doing pass@k or comparing checkpoints, several runs each. Sequentially that is an overnight job with a coffee-based feedback loop. The whole point of one-microVM-per-instance is that the fan-out is a config number rather than a capacity project.

Two things make it cheap. There is no warm pool of idle VMs waiting to be used — every create restores a snapshot on demand, so between runs your idle cost is effectively zero and you are not paying a standing fleet to be ready for a benchmark you run twice a week. And networking is pre-allocated: each agent host keeps 16,384 pre-built /30 subnets with their namespaces and TAP devices already assembled, so attaching a new VM to the network is a MAC patch rather than a hundred milliseconds of iptables work per instance.

import concurrent.futures as cf

def run_suite(instances, snapshots, patches, max_parallel: int = 64):
    """Grade a whole split. Bounded pool: memory is the ceiling, not slots."""
    results = []
    with cf.ThreadPoolExecutor(max_workers=max_parallel) as pool:
        futures = {
            pool.submit(grade, inst, snapshots[inst["id"]], patches[inst["id"]]): inst
            for inst in instances
            if inst["id"] in patches
        }
        for fut in cf.as_completed(futures):
            results.append(fut.result())
    return results

The SDK blocks on I/O, so a thread pool gives you genuine concurrency without rewriting the harness in asyncio. Keep the pool bounded: the constraint is aggregate memory across your hosts, not a slot count, and a suite that tries to hold a thousand VMs at once will discover that in the least helpful way. Start conservative, watch memory, raise it.

Network policy: deny egress, then prove it

During construction the sandbox needs the network — you cannot clone a repo or install ancient dependencies from a vacuum. During grading it needs nothing. That asymmetry is the single most valuable policy boundary in the harness, and it is why splitting bake from grade pays for itself twice: once in speed, once in security.

With the network denied at grading time, a patch cannot fetch the upstream fix, cannot phone home with the contents of the test file it just read, cannot pull a package that changes behaviour, and cannot make your run depend on whether GitHub was healthy that afternoon. If some instance genuinely needs a service to be reachable, run that service inside the same sandbox rather than opening a hole — a locally bound database on the guest's loopback is available to the tests and to nothing else. Because each VM sits in its own network namespace, there is no shared bridge for one instance to discover another on.

Determinism: time, seeds, and the tests that flake anyway

Once you have removed state leakage and network variance, the remaining flakiness is inside the test suites themselves, and old repositories are full of it. Tests that assume the current year. Tests that depend on dict ordering, or on a hash seed, or on the number of CPUs visible. Tests that race. None of that is your fault, but all of it lands in your number.

• Pin the clock's meaning, not the clock. Do not run a 2021 test suite against a certificate-expiry check with today's date and call the resulting failure a model failure. Where a suite is date-sensitive, record it and treat those instances separately.
• Pin the seeds. Export PYTHONHASHSEED, set any framework-level random seed the suite honours, and keep the value in the harness config so a re-run reproduces it.
• Pin the shape of the machine. vCPU count and memory come from the baked snapshot rather than the request, which is a feature here: every fork of an instance sees the same machine, so a test that is sensitive to parallelism sees the same parallelism.
• Disable test-order randomisation, or fix its seed. A suite using random ordering will give you a different answer per run, and you will spend a day blaming your patch pipeline.
• Measure flakiness explicitly. Run the baseline suite on the unpatched snapshot a few times during the bake and record which tests are unstable. An instance whose graded tests flake is not a valid instance.

The point is not to eliminate flakiness — you cannot, these are other people's test suites — but to know exactly how much of it you have, so that when a model's score moves by a point you can say whether that is signal.

Artifacts: what you keep after the box is gone

A microVM per instance means the evidence disappears when the instance is killed, so decide up front what leaves the box. The exit code alone is nearly useless three weeks later when a checkpoint regresses on eleven instances and you need to know why.

• The applied diff, read back with git diff HEAD before teardown. This is what actually ran, which is not always what the model emitted — a partially-applied patch is a real and confusing state.
• The full test output, or at least a generous tail. The pytest summary and the tracebacks are at the end, so tail rather than head, and keep more than you think you need.
• The guard verdict. If an instance was voided, keep the file list that voided it. Aggregated across a run, that is a genuinely interesting measurement of a model's willingness to game the grader.
• The environment identity: snapshot ID, base commit, install commands, harness version. This is what makes a result comparable to another result rather than merely adjacent to it.
• Timings and whether anything hit its timeout. A test command killed at the boundary is not a failed patch, and conflating them makes slow-but-correct fixes look wrong.
• The agent transcript, if you are grading an agent rather than a single-shot patch. When behaviour changes between runs, the diff in the command sequence is the first thing you will want.

Write all of it keyed by instance and attempt, not by instance alone, or pass@k results become impossible to inspect individually.

Container per instance vs microVM per instance

A container per instance with pinned image digests is a legitimate way to run SWE-bench and plenty of serious teams do exactly that. Here is the honest comparison rather than the marketing one.

• Isolation boundary — Container per instance: shared host kernel, so a kernel-level side effect, a resource spike, or a container escape crosses between instances. Adequate when you trust the patch source. microVM per instance: its own guest kernel, filesystem, and network namespace, so an untrusted patch is contained by hardware virtualisation rather than by namespace configuration.
• Environment fidelity — Container per instance: excellent, provided you pin digests and never rebuild; the image is the environment. microVM per instance: the snapshot captures memory and disk together, so you restore a machine that was already set up rather than replaying setup steps.
• Cold-start per instance — Container per instance: fast to start, but the environment build is where the time goes, and rebuilding is minutes. microVM per instance: restore from a baked snapshot is around 179ms p50 and 203ms p99; a genuine cold boot is around 3 seconds and you only pay it once per instance.
• Cheat detection — Container per instance: possible, but the instrumentation shares a kernel with the code it is watching, so a sufficiently motivated patch has a larger surface to work with. microVM per instance: the host's view is outside the guest, so the guest does not get a vote on what the host observed.
• Egress control — Container per instance: network namespaces plus firewall rules, which works and which you must verify per run because a misconfigured policy is silent. microVM per instance: per-VM network namespace with 16,384 pre-allocated /30 subnets per host, so deny-by-default is the shape of the thing rather than a rule you remembered to add.
• Fan-out — Container per instance: bounded by one host's cores and memory unless you build scheduling yourself. microVM per instance: forks schedule across a fleet and there is no warm pool, so parallelism is a config number and idle cost between runs is effectively zero.
• Resource blast radius — Container per instance: a forkbomb or disk-filling patch pressures the shared host and can degrade its neighbours' timings. microVM per instance: capped at the VM's own allocation, which is also what makes timing-sensitive tests comparable across instances.
• Best fit — Container per instance: first-party models you trained, smaller splits, a team that already has strong image discipline. microVM per instance: third-party or open-weight models whose patches you did not write, large parallel runs, and any number that will be quoted publicly.

Verify the container-side claims against Docker's own documentation for your configuration, because how much of that is true depends heavily on your volume, cache, and seccomp setup. The summary I would defend: containers are fine when you trust the patch and the split is small; microVMs earn their keep when the patch is untrusted, the fan-out is wide, or the result is going to be quoted.

What this actually buys you

Bake each instance's environment once and freeze it. Fork the snapshot per grading run so the starting state is identical and cheap. Treat the model's patch as untrusted code — guard the paths it may touch, deny egress, verify the denial from inside the guest. Apply the hidden tests only after the guard. Record four outcomes, not one. Keep the diff, the output, the guard verdict, and the environment identity. Kill every VM in a finally block with a TTL as backstop.

None of that makes your model better at fixing bugs. What it buys is the ability to say, when someone asks how you got your number, that the environment was identical across runs, that no patch could reach the network, that patches which edited the tests were counted as void rather than resolved, and that you can rebuild the whole thing and get the same answer. That is a smaller claim than a leaderboard position and a much more durable one — and it is the claim that will still be true after everyone has moved on to the next benchmark.