Running academic artifact evaluation on microVMs

Artifact evaluation is the volunteer job nobody signs up for twice. A committee of grad students and industry engineers is handed a set of tarballs and a deadline, and has to answer a deceptively small question about each one: if we run this, do the paper's claims come out the other side? The answer becomes a badge on the paper — Artifacts Available, Artifacts Functional, Results Reproduced, or the ACM variants of the same idea — and the badge is worth something, so the process has to be more than a vibe check.

The mechanics of that question are almost entirely an infrastructure problem, and it is the same infrastructure problem every time. Someone has to take code they did not write, from a person they cannot ask follow-up questions of quickly, run it with elevated privileges on a real machine, and then decide whether the numbers that fell out are close enough to the ones in the PDF. Then a second reviewer has to do it again, from scratch, because the first reviewer's machine is not a thing that can be shared.

I'm Ajay, I built PandaStack. This post is about the shape of AE as an infrastructure problem: what reviewers actually receive, why Docker fixes half of it and quietly hides the other half, and why the useful unit of work is a disposable microVM that gets snapshotted after the environment is built — so the second reviewer starts from the machine the first reviewer produced instead of re-running a forty-minute setup script and hoping the mirror is still up.

What an AE reviewer actually receives

Strip away the good intentions and a typical artifact is a compressed directory, a README written by someone who has never watched a stranger use it, and a shell script that assumes it owns the machine. The failure modes are boringly consistent across fields:

• install.sh with sudo sprinkled through it. Somewhere in there is a line like 'sudo rm -rf /usr/local/lib/python3.8' — not malice, just an author who fought their own laptop for a week and committed the winning move. Running it on a reviewer's machine is a genuinely bad idea, and reviewers know it, which is why so many artifacts get a Functional badge from someone who never actually ran the setup.
• Dependency rot with a timestamp on it. The paper was submitted eighteen months ago. The pinned CUDA archive moved, the PPA is gone, a transitive dependency yanked a version, and 'pip install -r requirements.txt' now resolves to a package set the authors never tested. None of this is the authors' fault and all of it lands on the reviewer.
• Code that assumes it owns the machine. Hardcoded /data paths, a fixed port, a global config file, a systemd unit, an assumption that it can take all the cores or all the RAM. Perfectly reasonable on the cluster node where it was developed; hostile on a shared reviewer box.
• Setup that takes forty minutes and then fails at minute thirty-eight. Every retry pays the full cost, and the second reviewer pays it again from zero.
• Numbers that are close but not equal. The paper says one thing, the run says something adjacent, and now a human has to decide whether the gap is a different CPU, a different thread count, a warm cache, or an actual problem with the claim.

Why "just ship a Docker image" solves half of it

Containers were the right answer to the previous generation of this problem, and most AE guidelines now recommend them. They genuinely fix dependency rot at the userspace layer: an image freezes a resolved package set, so the reviewer stops re-running a resolver against a moving internet. That is a real win and nothing below is an argument for going back to tarballs.

But a container image is userspace only, and the parts it leaves out are exactly the parts a systems paper tends to depend on. Three specific gaps:

First, the kernel is the reviewer's. An artifact that measures io_uring throughput, a BPF program, a scheduler change, a filesystem behaviour, or anything touching /proc tunables is running against whatever kernel the reviewer's laptop or the department server happens to have. It usually works, which is worse than failing, because a silently-different kernel produces a plausible number rather than an error. Second, the reviewer's kernel is also the boundary — the artifact has root in a namespace that shares a kernel with the machine holding the reviewer's SSH keys and their own unpublished work. Third, the artifact can still wander the network. Container egress is open by default, so a reproduction can quietly download a 'latest' wheel or call an API and produce a result that depends on a server, not on the code you were handed.

There's also an archival wrinkle. If what gets deposited is a Dockerfile rather than an image, a rebuild in 2030 re-resolves everything and you are back to tarball-era rot with extra steps. If it's the image, you're in much better shape — but you've archived half a machine and borrowed the other half from a future reviewer.

• Tarball plus install.sh — Longevity: decays the moment a mirror, a PPA, or a pinned vendor archive moves; the script encodes the package names of the year it was written. Kernel assumptions: inherited from whatever machine runs it, and almost never documented. Reviewer setup cost: the full setup, per reviewer, per retry. Isolation: none — it has sudo and it is on the reviewer's laptop.
• Docker image — Longevity: good if the built image is what gets archived, poor if the Dockerfile is, because a rebuild re-resolves against a different internet. Kernel assumptions: the real hole — userspace is frozen, the kernel is borrowed from the reviewer, so kernel-sensitive claims are measured against the wrong kernel and still return a number. Reviewer setup cost: one pull, which is the genuine and substantial win. Isolation: shared kernel with the reviewer's machine, open egress unless someone configured otherwise.
• Full VM image (OVA/qcow2) — Longevity: excellent; kernel and userspace travel together, which is why long-horizon archives like them. Kernel assumptions: none, it brings its own. Reviewer setup cost: a multi-gigabyte download, a hypervisor, and a boot before anything runs. Isolation: strong. Downside: heavy to move, and everyone's experiments mutate the one image unless the reviewer remembers to snapshot first.
• MicroVM snapshot — Longevity: same structural property as a VM image — own guest kernel, own userspace, frozen after the environment is built rather than before. Kernel assumptions: none. Reviewer setup cost: a restore instead of a boot (on PandaStack a create is p50 179ms, p99 around 203ms) and no re-run of install.sh at all. Isolation: hardware-virtualized guest per artifact with its own network namespace, so default-deny egress is enforced below the guest. Downside: less established as an archival format than a qcow2 file, and it needs a host that can restore it.

One disposable guest per artifact, snapshotted after setup

The model that fits AE is small. Give each artifact its own microVM. Let the setup script do whatever it wants in there — sudo, global installs, deleting a system Python, all of it — because the machine is disposable and the artifact genuinely does own it. Then, crucially, snapshot the guest after the environment is built, not before.

That ordering is the whole trick. The conventional artifact freezes the recipe and makes every consumer cook; freezing the built machine means the expensive, fragile, network-dependent part happens exactly once, by whoever had the patience to get it working. Every reviewer after that starts from an identical machine. Every experiment forks from the same frozen state, so run three doesn't inherit whatever run two left in /tmp — a same-host fork lands in 400-750ms and shares memory copy-on-write until it's written, which is what makes per-experiment forking affordable rather than theoretical.

from pandastack import Sandbox

PAPER = "conf26-paper-147"


class SetupFailed(Exception):
    """Not a crash -- a finding. Record it verbatim for the author response."""


def build_artifact_env(tarball_url: str):
    """Run the artifact's setup ONCE, then freeze the machine it produced."""
    sbx = Sandbox.create(
        template="base",
        ttl_seconds=5400,   # 90 min: install.sh may be slow, it may not be eternal
        metadata={"paper": PAPER, "kind": "ae-build"},
    )

    sbx.exec(
        f"mkdir -p /ae && curl -fsSL {tarball_url} | tar -xz -C /ae --strip-components=1",
        timeout_seconds=600,
    )

    # The setup script has sudo in it and assumes it owns the machine.
    # Both are fine here: it does own this machine, and this machine is disposable.
    setup = sbx.exec("cd /ae && bash install.sh", timeout_seconds=3600)
    if setup.exit_code != 0:
        sbx.kill()
        raise SetupFailed(setup.stderr[-8000:])

    # Record what the environment ACTUALLY became, not what the README claimed.
    sbx.exec(
        "{ uname -a; cat /etc/os-release; "
        "lscpu; pip3 freeze; dpkg -l; } > /ae/ENVIRONMENT.txt 2>&1"
    )
    sbx.filesystem.write("/ae/harness.sh", HARNESS)   # the hermetic runner, below
    sbx.exec("chmod +x /ae/harness.sh")

    snap = sbx.snapshot()   # the built machine itself -- not a recipe for it
    sbx.kill()
    return snap


def run_experiment(snap, name: str, cmd: str) -> dict:
    """Each experiment forks the SAME frozen machine. No run inherits another."""
    vm = snap.fork()
    try:
        r = vm.exec(f"/ae/harness.sh {name} -- {cmd}", timeout_seconds=7200)
        results = None
        if r.exit_code == 0:
            results = vm.filesystem.read(f"/ae/out/{name}.json")
        return {
            "experiment": name,
            "exit_code": r.exit_code,
            "log": r.stdout[-32000:],
            "stderr": r.stderr[-8000:],
            "results": results,
        }
    finally:
        vm.kill()   # background daemons, scratch files and half-written state included


if __name__ == "__main__":
    snapshot = build_artifact_env("https://example.org/artifacts/conf26-147.tar.gz")
    for claim in ("figure3", "figure7", "table2"):
        print(run_experiment(snapshot, claim, f"python3 -m bench.{claim}"))

Two details worth stealing. The setup failure is raised with the artifact's own stderr attached, because during author-response the exact error text is the entire conversation — 'it didn't build' helps nobody. And ENVIRONMENT.txt is generated by interrogating the machine rather than trusting the README, which is how you later explain a numeric gap to a co-reviewer running on different hardware.

Determinism traps that are not the artifact's fault

Here is where an AE process gets unfair without anyone noticing. Once you're forking a snapshot per experiment, you inherit a specific set of failure modes that look exactly like the artifact being broken. Every one of these has produced a false negative somewhere:

• The clock is frozen at snapshot time. A restored guest resumes believing it is whenever the snapshot was taken. Anything that timestamps output, measures elapsed wall time across a restore, or validates a TLS certificate is now living in the past — and expired-certificate errors from a machine whose clock never moved look like a broken artifact. PandaStack re-syncs the guest clock on restore, resume and wake for exactly this reason; if you build your own snapshotting, this is the first thing that bites you: /blog/firecracker-guest-clock-and-time-drift-explained.
• Every fork gets the same entropy. Clone one snapshot ten times and all ten children start from identical kernel RNG state. Two 'independent' runs then draw the same 'random' seeds, and a paper's variance across trials collapses into an artificially tidy result — the most dangerous kind of wrong number, because it looks better than the truth. The mechanics are in /blog/snapshot-clone-randomness-problem-explained.
• CPU features differ across hosts. Restore the same snapshot on a machine with different vector extensions and floating-point reduction order can change, which moves the last few digits of anything numerical. Best case you get a tolerance-sized gap; worst case an illegal instruction from a binary compiled with -march=native on the author's workstation.
• Page-cache warmth and thread counts. The second run of an experiment reads a file that is already in the guest's page cache, so it's faster than the first, and the artifact gets credit or blame for your harness's ordering. Likewise a BLAS library that picks its thread count from the visible core count will produce different reduction orders on a different guest shape.
• Filesystem ordering, hash seeds, and locale. Directory read order, Python's per-process hash randomization, and LC_ALL all leak into output that gets diffed. A results file that differs only by row order fails a naive comparison and passes a correct one.

The defence is a harness that runs between the reviewer and the artifact, pinning the ambient environment before the artifact's own code gets a vote. It belongs inside the guest, it belongs in the snapshot, and it should be identical for every experiment.

#!/usr/bin/env bash
# /ae/harness.sh -- run ONE experiment hermetically inside the artifact guest.
# Everything here pins ambient state the artifact never thought to control.
set -euo pipefail

EXP="${1:?experiment name}"; shift
[[ "${1:-}" == "--" ]] && shift
SEED="${AE_SEED:-20260820}"
mkdir -p /ae/out

# 1. The guest woke up from a snapshot with a stale clock. Fix it before any
#    result file gets a timestamp, or a TLS handshake decides it's 2024.
sudo chronyc makestep >/dev/null 2>&1 || sudo hwclock -s || true

# 2. Locale, timezone and build epoch: cheap, and they silently change sort
#    order, date formatting, and bytes baked into artifacts.
export TZ=UTC LC_ALL=C.UTF-8 LANG=C.UTF-8
export SOURCE_DATE_EPOCH="$(date -u +%s)"

# 3. Seed everything with a seed. PYTHONHASHSEED must be set BEFORE the
#    interpreter starts -- setting it inside main() does nothing at all.
export PYTHONHASHSEED="$SEED" AE_SEED="$SEED"
export PYTHONDONTWRITEBYTECODE=1

# 4. Pin thread counts. Parallel float reductions are not associative, so a
#    different core count is a different answer in the low-order digits.
export OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1
export CUBLAS_WORKSPACE_CONFIG=:4096:8   # cuBLAS determinism, if a GPU is present

# 5. Network off by default. A "reproduction" that downloads a latest wheel or
#    calls an API is measuring someone else's server, not this artifact.
if [[ "${AE_ALLOW_NET:-0}" != "1" ]]; then
  sudo ip link set eth0 down 2>/dev/null || true
fi

# 6. ASLR off ONLY for artifacts whose claims depend on memory layout
#    (allocator studies, pointer-chasing microbenchmarks). Otherwise leave it
#    alone -- disabling it changes the thing some papers are measuring.
RUNNER=(env)
[[ "${AE_NO_ASLR:-0}" == "1" ]] && RUNNER=(setarch "$(uname -m)" -R env)

# 7. Drop caches so run N+1 doesn't get credit for run N's warm page cache.
sync && sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' 2>/dev/null || true

# 8. Record the machine alongside the result. When two reviewers disagree,
#    this file is the argument.
{ date -u; uname -a; nproc; grep -m1 'model name' /proc/cpuinfo; } \
  > "/ae/out/$EXP.env"

set +e
"${RUNNER[@]}" "$@" > "/ae/out/$EXP.log" 2>&1
rc=$?
set -e

sha256sum "/ae/out/$EXP".* > "/ae/out/$EXP.sha256" || true
exit "$rc"

Note step 6. Turning ASLR off is a tool, not a policy — for an allocator or memory-layout paper it removes a real source of run-to-run noise, and for everything else it is you quietly modifying the system under test. The general rule for the whole harness: pin the things the paper is not about, and never pin the thing the paper is about.

Deciding whether it actually reproduced

The last mile is comparing what came out against what the paper claims, and it deserves to be code rather than a reviewer squinting at two tables at 1am. Exact equality is the wrong bar for anything with a timer in it; the right bar is a per-claim tolerance the authors declare and the committee can argue with.

import json


def check(claimed: dict, measured: dict, tolerances: dict) -> list[dict]:
    """Compare a run against the paper's claims with per-claim tolerances.

    claimed/measured: {"figure3.speedup": 3.4, ...}
    tolerances:       {"figure3.speedup": 0.10}   # relative, declared by authors
    """
    findings = []
    for key, want in claimed.items():
        if key not in measured:
            findings.append({"claim": key, "verdict": "missing"})
            continue
        got = measured[key]
        tol = tolerances.get(key, 0.05)
        rel = abs(got - want) / abs(want) if want else abs(got)
        findings.append({
            "claim": key,
            "claimed": want,
            "measured": got,
            "relative_delta": round(rel, 4),
            # "within" is not "correct" -- it means the number reproduced.
            "verdict": "within" if rel <= tol else "outside",
        })
    return findings


if __name__ == "__main__":
    claims = json.loads(open("paper_claims.json").read())
    run = json.loads(open("out/figure3.json").read())
    for f in check(claims["values"], run, claims["tolerances"]):
        if f["verdict"] != "within":
            print("REVIEW:", f)

Two things follow from writing it this way. Authors have to state a tolerance up front, which is a useful conversation to force — a claim with no acceptable error bar is not a reproducible claim. And a reviewer's report becomes a diff with the environment file attached, so 'outside tolerance on a different CPU' and 'outside tolerance on identical hardware' stop being the same sentence.

Publishing the snapshot as the artifact

Follow the model one step further and something nicer than a badge appears. If the built machine is the deliverable, then what gets deposited alongside the paper is a snapshot, and a reader in 2030 gets the environment rather than instructions for recreating it. No mirror to be gone, no CUDA archive to have moved, no resolver to run against an internet that has changed underneath the requirements file. They restore the machine and fork it — and because a fork is copy-on-write, ten readers poking at the same published environment cost roughly one environment plus their own writes. The mechanics of what a snapshot captures are in /blog/snapshot-and-fork-explained.

This is what the Available badge is reaching for and mostly missing. Depositing a tarball makes the source available; depositing a working machine makes the result available. The difference shows up the first time somebody wants to extend the work rather than merely verify it — building on an artifact currently means reconstructing its environment from scratch, which is why so much follow-up work quietly reimplements the baseline instead.

A README is a claim about a machine. A snapshot is the machine. Only one of them still works in five years, and it isn't the one that gets deposited today.

The practical caveats are real. Snapshots are large, so someone has to pay for storage on a horizon measured in years, and an archive that goes away takes the artifact with it. A snapshot is opaque compared to a Dockerfile — you can read a recipe, you can only run a machine — so deposit both, with the source tree and the build script inside the guest where they can be inspected. And a published guest full of someone's code should be restored under a default-deny egress policy, because nobody is watching it.

Honest limits

None of this makes artifact evaluation easy, and several categories of artifact are simply outside what a microVM can help with.

• GPU and accelerator artifacts. A snapshot freezes CPU state and memory; it does not freeze a GPU, a driver stack, or the specific accelerator a paper measured. Passthrough, driver-version matching, and device availability are all real work that this model does not remove.
• Hardware-specific claims. If the contribution is about a NIC, a persistent-memory module, a particular NUMA topology, or an FPGA, then reproducing it requires that hardware. A guest with its own kernel removes the software variables and leaves the hardware ones exactly where they were.
• Licensed and restricted datasets. Plenty of artifacts cannot ship their input data — medical, proprietary, or scale-restricted. The environment reproduces; the experiment waits on an access agreement, and the badge criteria should say so out loud.
• Scale. An artifact whose claims are about a 500-node cluster does not fit in a guest, and a scaled-down reproduction is a different claim than the one in the paper.
• Non-determinism the authors do control. Unseeded randomness, races, and timing-dependent output stay broken no matter how hermetic the harness. The harness's job is to stop adding new sources, not to fix the artifact's.

And the largest limit is the one the badges themselves warn about: reproduced is not correct. A reproduction says the code, run on this environment, produces the numbers in the paper. It says nothing about whether the experiment measures what the authors claim, whether the baseline was tuned as carefully as the proposal, or whether the metric answers the question. A perfectly reproducible artifact can support a wrong conclusion, and an artifact-evaluation pipeline that reports 'within tolerance' on every claim has verified an implementation, not a finding. That judgment is still human work, which is precisely why it's worth automating everything underneath it.

What the microVM model buys is narrower and worth having: nobody has to run install.sh on their laptop, the setup cost is paid once instead of once per reviewer, every experiment starts from the same machine instead of the residue of the last one, and the environment that produced the numbers can be handed to the next person as an object rather than a description. The reviewers get to spend their remaining attention on whether the paper is right. The determinism traps above are the price of admission — and unlike dependency rot, they stay fixed once you fix them.