Building an autograder that runs student code safely in microVMs

An autograder is a server whose entire job is to run code that its own authors did not write, cannot review in advance, and often actively resent writing. A university course or coding bootcamp posts an assignment, a few hundred students submit solutions, and a hidden test suite decides who passed. Most of those submissions are honest and merely broken. A few are broken in ways that will take your grader down with them, and a few are deliberately trying to cheat. If your autograder runs all of that in a shared process, a shared container, or the same box as your gradebook, one bad Tuesday night before the deadline will teach you why that was a mistake. This post is about building an autograder where each submission runs inside its own ephemeral Firecracker microVM — with the test suite injected, hard resource limits, no network, and the whole machine deleted the moment grading finishes.

I'm Ajay, I built PandaStack. The autograder loop is short and the same for every language: take a submission, boot a fresh sandbox, inject the student's source plus your hidden test harness, run the tests under strict CPU/memory/wall-clock limits, collect a pass/fail report, and throw the whole VM away. I'll show the code, the failure modes you're actually defending against, and why the boring-sounding property — reproducibility — is the one that saves you the most grief.

What students actually submit

The threat model for an autograder is unusual because it's a mix of accidental chaos and motivated cheating, and you can't tell which is which at submission time. Over a term you will grade all of the following, and you should assume every one of them arrives during the final hour before the deadline when your queue is deepest:

• The fork bomb. Someone writes `while True: os.fork()` or the bash `:(){ :|:& };:`. It is almost never malice — it is a rite of passage, a recursion bug, or a misread of how threads work. Your grader must survive it without dropping the other 200 submissions in the queue.
• The resource hog. An accidental `O(2^n)` solution, a runaway allocation, a `range(10**12)` loop. Correct code and a timeout-bomb look identical until you run them, so the grader has to run them safely.
• The answer-key reader. A student who realizes the tests are on disk and tries `open('/grader/expected.txt')`, or greps the filesystem for the test harness, or reads another student's submission out of a shared temp dir.
• The grader patcher. Cleverer cheating: monkey-patch the test framework so every assertion passes, overwrite the reference solution, or `sys.exit(0)` before the failing test runs so the harness records a clean pass.
• The persister. Code that leaves a process, a file, or a modified environment behind so it taints the next submission graded on the same box — accidentally (a daemon that didn't die) or on purpose (poisoning a classmate's run).
• The exfiltrator. On a networked grader, code that POSTs the hidden tests to a pastebin so the whole class has them by morning.

Why a microVM per submission, not a shared runner

The tempting cheap option is a shared Python process or a pool of long-lived containers that you reset between runs. Both share one kernel across every submission, and that shared kernel is exactly the thing a motivated student — or a buggy one — reaches through. A container isolates a submission with namespaces and cgroups on top of the host's kernel; every submission on that host is one kernel bug, one cgroup misconfiguration, or one `/proc` leak away from every other submission and from your gradebook. Resetting a container between runs is also never quite complete: a leftover process, a poisoned pip cache, a file in `/tmp` the reset script forgot. The autograder's whole promise is that student N+1's grade doesn't depend on what student N did, and shared-kernel reuse quietly breaks that promise.

A Firecracker microVM boots its own guest kernel under KVM. The submission can only reach the outside world through a handful of emulated virtio devices — there is no host syscall interface to attack, only the hypervisor's tiny, heavily-audited surface (the same VMM AWS Lambda and Fargate rely on for cross-tenant isolation). Resource limits stop being best-effort cgroup accounting and become hard properties of the machine: the guest is booted with a fixed vCPU count and a fixed memory size and physically cannot exceed them. A fork bomb hits a wall that is a real, capped computer; a memory hog gets OOM-killed by its own guest kernel; and because you destroy the VM after one submission, persistence and cross-student leakage aren't risks you police — they're states that can't exist.

The historical objection to per-submission VMs was startup cost — nobody grades a class if each submission needs a 30-second boot. That's what snapshot-restore solves. Instead of cold-booting, every create restores a pre-baked snapshot of an already-booted machine. On PandaStack a create is p50 179ms (p99 ~203ms), with the fast restore path around 49ms; a genuine cold boot happens only on the very first spawn before the snapshot exists (~3s). Per-submission VM isolation for roughly the price of a container start is the entire trick that makes this practical for a course.

The grading loop: inject source, run hidden tests, emit a report

Here's the core of an autograder for a Python assignment: one fresh sandbox per submission, write the student's file plus your hidden `pytest` suite into the guest, run the tests with a wall-clock timeout, parse the machine-readable report, and destroy the VM. The tests are injected at grade time and never live in the student's repo — they come from your control plane, not from the submission. Set `PANDASTACK_API_KEY` in your environment and the SDK picks it up.

import json
from dataclasses import dataclass, field
from pandastack import Sandbox

@dataclass
class GradeReport:
    passed: int = 0
    total: int = 0
    score: float = 0.0
    status: str = "graded"      # graded | timeout | error | crashed
    detail: str = ""
    failures: list[str] = field(default_factory=list)

def grade(student_code: str, hidden_tests: str) -> GradeReport:
    # One disposable microVM per submission. ttl_seconds is a backstop in
    # case we crash before kill(); the finally block is the real cleanup.
    sbx = Sandbox.create(template="code-interpreter", ttl_seconds=120)
    try:
        # The student's file and OUR hidden test suite, side by side in
        # the guest. The student never had the tests in their repo.
        sbx.filesystem.write("/grader/solution.py", student_code)
        sbx.filesystem.write("/grader/test_hidden.py", hidden_tests)

        # Run pytest, emit a JSON report the host can trust. The wall-clock
        # timeout is the circuit breaker for infinite loops.
        run = sbx.exec(
            "cd /grader && python3 -m pytest test_hidden.py "
            "--json-report --json-report-file=/grader/report.json -q",
            timeout_seconds=30,
        )

        if run.exit_code == 124:                 # wall-clock timeout tripped
            return GradeReport(status="timeout",
                               detail="submission exceeded 30s wall clock")

        # Read the report the tests wrote; the student's exit code is not
        # trusted (a cheater can sys.exit(0) early) -- the JSON is.
        try:
            raw = sbx.filesystem.read("/grader/report.json")
            data = json.loads(raw)
        except Exception:
            return GradeReport(status="crashed",
                               detail=run.stderr[:1000] or "no report written")

        summary = data.get("summary", {})
        passed = summary.get("passed", 0)
        total = summary.get("total", 0)
        fails = [t["nodeid"] for t in data.get("tests", [])
                 if t.get("outcome") != "passed"]
        return GradeReport(
            passed=passed, total=total,
            score=round(passed / total, 4) if total else 0.0,
            failures=fails[:20],
        )
    finally:
        sbx.kill()   # the VM, its processes, and anything it wrote are gone

The one detail that catches people: don't trust the submission's exit code as the grade. A student can `sys.exit(0)` before a failing test runs, catch `SystemExit`, or monkey-patch the assertion. The trustworthy signal is the structured report the test framework itself writes to a file, read back by your host — and even that lives inside the VM only long enough to be read out. Everything the submission spawned, wrote, or corrupted dies with `sbx.kill()`, so the next student boots from a pristine snapshot with zero inheritance from this one.

Hard limits: making the fork bomb boring

Two layers enforce limits, and a good autograder uses both. The host-side wall-clock timeout on `exec` is the belt — when it trips you read exit code 124 and record a timeout. Inside the guest, wrap the run with the kernel's own `ulimit` accounting for hard per-process ceilings — the suspenders. The reason both matter is that the belt bounds total wall time while the suspenders make a fork bomb or an allocation storm hit a real limit instantly, before it can even matter, because the whole thing is already happening inside a VM capped at its baked vCPU and RAM.

# The command the grader runs INSIDE the guest. ulimits turn a fork bomb
# and a memory hog into deterministic, instant failures instead of a
# thundering-herd incident on your grading host.
cd /grader && \
  ulimit -t 10 && \
  ulimit -v 524288 && \
  ulimit -u 128 && \
  ulimit -f 51200 && \
  timeout --signal=KILL 20 \
    python3 -m pytest test_hidden.py \
      --json-report --json-report-file=/grader/report.json -q

# ulimit -t 10      -> 10 CPU-seconds, then SIGKILL (not wall time)
# ulimit -v 524288  -> ~512 MB address space (KB); allocation storms die
# ulimit -u 128     -> max 128 processes -> the fork bomb hits a wall
# ulimit -f 51200   -> max ~50 MB of written files -> no filling the disk
# timeout 20        -> wall-clock backstop, slightly above the CPU limit

Because the guest is a real VM with a fixed memory size baked into the snapshot, a submission that ignores the `ulimit` and keeps allocating simply gets OOM-killed by the guest kernel — it cannot reach past its own RAM to touch the host or the submission grading next door. That is the deterministic-accounting property, and it's what makes an autograder's timing verdicts fair. On a shared-kernel runner, a noisy neighbor's fork bomb can steal CPU from a correct solution and turn it into a spurious timeout — a bug that is miserable to reproduce and worse to explain to a student who was marked down for someone else's code.

Hiding the hidden tests (and the answer key)

A hidden test suite that a student can read isn't hidden — they'll hardcode the expected outputs and hand in a lookup table. With a microVM this is structural rather than something you constantly police, because the only tests that ever touch the guest are the ones you chose to inject, and they vanish with the VM:

• Inject tests at grade time, never in the repo. The student's submission is just their source. Your host writes the hidden suite into the guest immediately before running it. There is no committed test file for a student to read before the deadline.
• Keep sensitive reference solutions on the host. If your grading needs a reference implementation or a fuzzing oracle, run it in your control plane against the guest's output — don't ship the reference into the sandbox where the code being graded can read it.
• No secrets in the guest env. A sandbox isolates execution, not the credentials you hand it. Never inject the gradebook DB URL, an LMS API token, or a roster into the VM the submission runs in.
• One VM per submission closes cross-student leakage entirely. Even a submission that reads its whole filesystem, snoops all of its own /proc, and dumps its own memory finds only its own code and the tests it was handed — the next student boots from a clean snapshot that has never seen it.

The grader-patching class of cheat — monkey-patching the test framework, overwriting the reference, exiting early — is defended the same way it was in the loop above: the grade comes from the structured report the framework writes and your host reads back, not from anything the student's process claims. And because the report is parsed on the host from a file the tests produced, a submission that never lets the tests run at all simply produces no report, which you score as a crash rather than a pass.

Grading a whole class in parallel

Coursework arrives in one enormous burst against the deadline, and autograding is embarrassingly parallel — submissions don't depend on each other. The reason snapshot-restore matters here isn't only latency, it's throughput: there's no warm pool of idle VMs to maintain and every create is an independent snapshot restore, so you can fan out grading wide, run many submissions at once each in its own VM, then reap them. This scales close to linearly with the hosts you throw at it, which is what lets a grader clear a 300-person class in the minutes after a deadline instead of overnight.

from concurrent.futures import ThreadPoolExecutor

# Grade a deadline burst concurrently: each submission gets its own
# microVM, graded independently, then destroyed. No shared state, no
# cross-talk, so the same submission earns the same grade every time.
def grade_class(submissions: dict[str, str], hidden_tests: str,
                workers: int = 32) -> dict[str, GradeReport]:
    results: dict[str, GradeReport] = {}

    def _one(item):
        student_id, source = item
        return student_id, grade(source, hidden_tests)   # fresh VM inside

    with ThreadPoolExecutor(max_workers=workers) as pool:
        for student_id, report in pool.map(_one, submissions.items()):
            results[student_id] = report
            print(f"{student_id}: {report.status} "
                  f"{report.passed}/{report.total} ({report.score:.0%})")
    return results

If every submission for an assignment runs against the same heavy setup — a large dataset, a compiled fixture, a pre-installed package set the base template doesn't carry — you can go one step further: bake a snapshot of a sandbox that already has that assignment's scaffolding in place, then `fork` it per submission instead of creating from the base template. A same-host fork is 400–750ms and shares memory copy-on-write; a cross-host fork is 1.2–3.5s. For most assignments the plain per-submission create is simplest and already fast enough — reach for fork-per-submission when the per-assignment setup is expensive enough to be worth caching in a snapshot.

Why reproducibility is the real feature

Security gets the headlines, but the property that saves an instructor the most time is reproducibility. A grade has to be defensible: when a student appeals, you need to re-run their exact submission and get the exact same result, and you need every student in the class to have been measured against an identical environment. A shared runner can't promise that — the environment drifts as submissions install packages, leave files, and load the box unevenly, so the same code can pass at 2pm and time out at midnight. A snapshot-restored microVM makes the environment a frozen, versioned artifact: every submission starts from byte-identical state, with the same libraries, the same CPU and memory budget, and no residue from the run before it. Re-grading an appeal is just running the same submission against the same snapshot again.

Shared process vs container pool vs microVM autograder

• Isolation boundary — Shared process: a Python subprocess and rlimits on your host, same kernel as the grader. Container pool: namespaces + cgroups, still the shared host kernel, reset between runs. microVM: a separate guest kernel under KVM, hypervisor boundary, destroyed per submission.
• Fork bomb / resource hog — Shared process: needs a perfect pids/mem rlimit or it takes the grading host down mid-queue. Container pool: needs correct cgroup caps; a mistake degrades every concurrent submission. microVM: contained to the VM's own capped vCPU/RAM by construction; a fork bomb is boring.
• Answer-key / test leakage — Shared process & container pool: tests and other students' files sit on a shared filesystem you must carefully police. microVM: tests injected per run into a throwaway VM; nothing leaks across students.
• Reproducibility — Shared process: environment drifts as submissions run; same code can grade differently over the queue. Container pool: better, but reset is never quite complete. microVM: byte-identical snapshot per submission; re-grade an appeal and get the same result.
• Cheating resistance — Shared process & container pool: grader patching and early-exit are harder to fully rule out on a shared kernel. microVM: grade from the host-read report, in a VM that's deleted after one submission.
• Startup cost — Shared process: ~none. Container pool: ms to seconds, plus reset overhead. microVM: ~179ms p50 via snapshot-restore (~49ms fast path), ~3s only on the very first cold boot.
• Maintenance — Shared process: you own an ever-growing rlimit/seccomp policy. Container pool: you own cgroup/namespace hardening, reset scripts, and kernel patching. microVM: the boundary is the hypervisor; you maintain far less policy.

The pattern to notice: a shared process and a container pool both put you in the business of correctly enumerating everything a hostile-or-buggy submission must NOT be allowed to do, against a kernel and a filesystem that everyone shares. The microVM autograder inverts it — each submission is handed a whole throwaway computer with the tests inside it and simply cannot reach anything else. For deeper reading on the general problem, see our guides on how to sandbox untrusted code and multi-tenant code execution.

Honest limits

A microVM autograder is not magic. Network egress from the guest is real by default — an autograder almost always wants zero egress, so a submission can't POST the hidden tests to a pastebin or fetch a solution mid-run; disable outbound networking at the network layer rather than trusting the code. The hypervisor surface is small but non-zero; keep Firecracker patched. Plagiarism detection between students is a separate problem the sandbox doesn't solve — it isolates execution, not similarity. And the VM boundary protects your grader, not your architecture: if you hand the guest a token or a reachable internal service, you've undone it. Within those bounds, a per-submission microVM lets your autograder treat the semester's inevitable `while True: fork()` as exactly what it is — someone else's problem, contained in a VM you're about to delete.