Building an Online Judge on microVMs

An online judge — the engine behind LeetCode, HackerRank, Codeforces, a classroom autograder, or a CTF scoreboard — has one job that makes it different from almost every other kind of server: it runs code written specifically to defeat it. Not accidentally hostile, deliberately hostile. In a graded contest with a leaderboard on the line, a contestant WILL submit a fork bomb; the only question is whether it's your host or their throwaway VM that finds out. Someone will try to read the expected outputs off disk. Someone will spawn a thread that keeps mining after their solution 'finishes.' Someone will snoop /proc to fingerprint the box. This post is about building a judge where all of that is contained by a real kernel boundary — one Firecracker microVM per submission — with clean, deterministic resource accounting so the scores are actually fair.

I'm Ajay, I built PandaStack. The judge loop is short: take a submission, boot a fresh sandbox, inject the source and the hidden test harness, compile and run against each test case under strict CPU/memory/wall-clock limits, collect a verdict (AC/WA/TLE/RE/CE), and throw the whole VM away. I'll show the code, the threat model, and why a shared-kernel container judge is a liability you'll eventually pay for.

The threat model: adversarial by design

Most sandboxing advice assumes untrusted code that's untrusted by accident — an AI wrote it, a plugin came from a marketplace. A judge is a harder problem, because the input is untrusted on purpose and the author is motivated. Your threat model has to include, at minimum:

• Resource exhaustion — fork bombs, infinite loops, 40GB allocations, filling the disk. The judge must survive these without degrading other submissions.
• Information disclosure — reading the test harness, the expected outputs, other contestants' submissions, or grader secrets that happen to be on the machine.
• Escape — breaking out of the jail to reach the host, the scoring database, or the network. This is the class of bug that has historically killed ptrace/seccomp judges.
• Timing and side channels — using shared CPU state or /proc to infer test data, or to game the resource accounting so a slow solution reads as fast.
• Persistence — leaving a process, a cron job, or a file behind that affects the next submission graded on the same environment.

Classic judges (think the original ICPC-style setups, or early self-hosted graders) ran submissions as a low-privilege user inside a chroot, wrapped in a ptrace or seccomp syscall filter that killed the process on a disallowed syscall. It works until it doesn't. Syscall filters are allow/deny lists against the entire Linux ABI, and the ABI is enormous and keeps growing; a single missed syscall, an ioctl corner case, or a kernel LPE turns 'contained' into 'root on the judge.' The CVE history of container escapes and seccomp bypasses is the reason platforms that grade code at scale moved to a stronger boundary.

Why a microVM per submission, not a container

A container isolates a submission with namespaces and cgroups on top of the host's shared kernel. Every submission on that host is one kernel bug away from every other submission and from the grader itself. For a code interpreter running an AI's helpful-but-clumsy code that's a manageable risk; for a contest where thousands of people are actively probing for the exact syscall that breaks out, the shared kernel is a standing liability.

A Firecracker microVM boots its own guest kernel under KVM. The submission can only touch the outside world through a handful of emulated virtio devices — there is no host syscall interface to attack, only the hypervisor's tiny, heavily-audited surface (the same VMM AWS Lambda and Fargate lean on for cross-tenant isolation). Resource limits stop being best-effort cgroup accounting on a shared kernel and become hard properties of a VM: the guest is booted with N vCPUs and M megabytes and physically cannot exceed them. That last part matters more than it sounds — deterministic resource accounting is what makes a judge fair. If a fork bomb in one submission can steal CPU from the submission grading next door, your timing verdicts are noise.

The historical objection to VMs was startup cost — nobody's spinning up a 30-second VM per submission. That's the problem snapshot-restore solves: instead of cold-booting, every create restores a pre-baked snapshot of an already-booted machine. On PandaStack a create is p50 179ms (p99 ~203ms), with the fast restore path around 49ms; a genuine cold boot (first spawn, before the snapshot exists) is ~3s. Per-submission VM isolation for the price of a container's startup is the whole trick.

The grading loop: submit, run test cases, emit a verdict

Here's the core of a judge: one fresh sandbox per submission, compile once, then run the compiled binary against each hidden test case with a wall-clock timeout and captured output. Set PANDASTACK_API_KEY in your environment and the SDK picks it up. Note that the test cases — inputs and expected outputs — live on the host and are fed in one at a time; the guest never gets the full answer key on disk.

from dataclasses import dataclass
from pandastack import Sandbox

@dataclass
class TestCase:
    stdin: str
    expected: str
    time_limit_s: float = 2.0

@dataclass
class Verdict:
    status: str          # AC | WA | TLE | RE | CE
    case: int = 0        # 1-based index of the failing case, 0 if none
    time_ms: int = 0     # max wall time across cases
    detail: str = ""

def grade(source: str, tests: list[TestCase]) -> Verdict:
    # One disposable microVM per submission. ttl_seconds is a backstop in
    # case we crash before kill(); the finally block is the real cleanup.
    sbx = Sandbox.create(template="code-interpreter", ttl_seconds=120)
    try:
        sbx.filesystem.write("/judge/main.cpp", source)

        # Compile once. A compile error is its own verdict (CE).
        build = sbx.exec(
            "g++ -O2 -std=c++20 -o /judge/sol /judge/main.cpp",
            timeout_seconds=15,
        )
        if build.exit_code != 0:
            return Verdict("CE", detail=build.stderr[:2000])

        max_ms = 0
        for i, tc in enumerate(tests, start=1):
            sbx.filesystem.write("/judge/in.txt", tc.stdin)
            run = sbx.exec(
                "/judge/sol < /judge/in.txt",
                timeout_seconds=tc.time_limit_s,
            )
            max_ms = max(max_ms, run.duration_ms)

            if run.exit_code == 124:            # timed out
                return Verdict("TLE", case=i, time_ms=max_ms)
            if run.exit_code != 0:              # crashed, killed, OOM
                return Verdict("RE", case=i, time_ms=max_ms,
                               detail=run.stderr[:500])
            if run.stdout.strip() != tc.expected.strip():
                return Verdict("WA", case=i, time_ms=max_ms)

        return Verdict("AC", time_ms=max_ms)
    finally:
        sbx.kill()   # the VM (and anything the submission left behind) is gone

`exec` returns an ExecResult with `stdout`, `stderr`, `exit_code`, and `duration_ms`. The `timeout_seconds` is your wall-clock circuit breaker — when it trips, the process is killed and you read exit code 124, which maps cleanly to TLE. A non-zero, non-124 exit is a runtime error (RE): a segfault, an uncaught exception, or the guest kernel OOM-killing a memory hog. Mismatched output is WA. Everything the submission spawned, wrote, or corrupted dies with `sbx.kill()`, so the next submission starts from a pristine snapshot with zero inheritance from this one.

Enforcing time and memory limits

There are two layers of limit enforcement, and a good judge uses both. Wall-clock time is enforced at the exec layer with `timeout_seconds` — the belt. Inside the guest, wrap the run with the kernel's own accounting for a hard per-process ceiling — the suspenders. The point of doing it inside the guest is that even a fork bomb hits a real limit before it can matter, because the whole thing is happening inside a VM that's already capped at its baked vCPU/RAM.

# A stricter runner: cap CPU seconds, address space, open files, and
# processes with ulimits *inside* the guest, then still bound wall time
# from the host. `prlimit`/ulimit make TLE and MLE deterministic per run.
run = sbx.exec(
    "bash -lc '"
    "ulimit -t 2; "          # 2s of CPU time (not wall) -> SIGKILL
    "ulimit -v 262144; "     # ~256MB address space (KB)
    "ulimit -u 64; "         # max 64 processes -> fork bomb hits a wall
    "ulimit -f 8192; "       # max 8MB written files
    "/judge/sol < /judge/in.txt'",
    timeout_seconds=3,       # wall-clock backstop, slightly above CPU limit
)

# Distinguish the failure modes for an accurate verdict:
#   exit 124            -> wall timeout           -> TLE
#   exit 137 (128+9)    -> SIGKILL (CPU/mem cap)  -> TLE or MLE
#   other non-zero      -> crash / runtime error  -> RE
if run.exit_code in (124, 137):
    verdict = "TLE"
elif run.exit_code != 0:
    verdict = "RE"
else:
    verdict = "OK"

Because the guest is a real VM with a fixed memory size baked into the snapshot, a submission that ignores the ulimit and keeps allocating simply gets OOM-killed by the guest kernel — it can't reach past its own RAM to touch the host or a neighbor. This is the deterministic-accounting property: the same submission graded twice gets the same time and memory numbers within noise, because it's never sharing a kernel scheduler with someone else's fork bomb. On a shared-kernel container judge, a noisy neighbor can inflate your wall time and turn a correct solution into a spurious TLE — a fairness bug that's very hard to reproduce and very easy to lose a contest over.

Hiding the test harness and expected outputs

A contestant who can read the expected outputs can hardcode them; a contestant who can read the checker can reverse it. So the answer key must never be reachable from the code being graded. With a microVM this is structural rather than something you have to police:

• Feed one test at a time. Write only the current case's stdin into the guest, run, read stdout back, and compare the expected output on the host. The guest never holds the full set of expected outputs, so there's nothing on disk to grep.
• Keep the checker on the host. For problems with special judges (multiple valid answers, floating-point tolerance), run the checker in your control plane against the guest's stdout — don't ship the checker binary into the sandbox.
• No secrets in the guest env. A sandbox isolates execution, not the credentials you hand it. Never inject the scoring DB URL, an API key, or grader tokens into the VM the submission runs in.
• Wipe between cases if you must reuse. If you compile once and loop cases in one VM (a reasonable optimization within a single submission), delete the previous case's input before writing the next — but still one VM per submission, never across submissions.

The one-VM-per-submission rule is what closes the cross-submission leakage vector entirely. Even if a submission manages to scribble across its whole guest filesystem, snoop every byte of its own /proc, and dump its own memory, there is nothing there but its own code and the single test input it was handed — the next submission boots from a clean snapshot that has never seen it.

Batch grading throughput

Contests submit in bursts — the last five minutes of a Codeforces round is a thundering herd. The reason snapshot-restore matters here isn't just latency, it's throughput: because there's no warm pool of idle VMs to maintain and every create is an independent snapshot restore, you can fan out grading wide and grade many submissions in parallel, each in its own VM, then reap them. A judge is embarrassingly parallel — submissions don't depend on each other — so this scales almost linearly with hosts.

from concurrent.futures import ThreadPoolExecutor

# Grade a burst of submissions concurrently: each gets its own microVM,
# graded independently, then destroyed. No shared state, no cross-talk.
def grade_queue(submissions: list[tuple[str, str]], tests, workers: int = 32):
    results: dict[str, Verdict] = {}

    def _one(item):
        sub_id, source = item
        return sub_id, grade(source, tests)   # fresh VM inside grade()

    with ThreadPoolExecutor(max_workers=workers) as pool:
        for sub_id, verdict in pool.map(_one, submissions):
            results[sub_id] = verdict
            print(f"{sub_id}: {verdict.status} "
                  f"(case {verdict.case}, {verdict.time_ms}ms)")
    return results

If every submission for a given problem runs against the same compiled harness or the same warmed environment, you can go further: bake a snapshot of a sandbox that already has the problem's judge scaffolding in place, then `fork` it per submission instead of creating from the base template. A same-host fork is 400–750ms and shares memory copy-on-write; a cross-host fork is 1.2–3.5s. For most judges the plain per-submission create is simplest and already fast enough — reach for fork-per-submission when the per-problem setup is expensive enough to be worth caching in a snapshot.

chroot+seccomp vs container judge vs microVM judge

• Isolation boundary — A (chroot+seccomp): a syscall allow/deny filter over the full host kernel ABI. B (container): namespaces + cgroups, still the shared host kernel. C (microVM): a separate guest kernel under KVM, hypervisor boundary.
• Escape blast radius — A: a missed syscall or kernel LPE = root on the judge host. B: a container escape or kernel bug = the host and every neighboring submission. C: an escape gets a throwaway guest that's about to be deleted, with nothing else in it.
• Resource accounting — A: rlimits + wall time, honest but shares the host scheduler. B: cgroups, best-effort, noisy neighbors can skew timings. C: hard vCPU/RAM baked into the VM; deterministic, fair across submissions.
• Fork bomb / OOM — A: needs a correct pids/mem rlimit or it can take the box. B: needs correct cgroup caps; a mistake degrades the whole host. C: contained to the VM's own capped resources by construction.
• Answer-key leakage — A/B: same host filesystem/kernel; you must carefully police what's reachable. C: one VM per submission from a clean snapshot; nothing to leak across submissions.
• Startup cost — A: ~none (just a process). B: ms to seconds. C: ~179ms p50 via snapshot-restore (~49ms fast path), ~3s only on the first cold boot.
• Maintenance — A: you own an ever-growing syscall policy against a moving ABI. B: you own cgroup/namespace hardening + kernel patching. C: the boundary is the hypervisor; you maintain far less policy.

The pattern to notice: chroot+seccomp and containers both put you in the business of correctly enumerating everything a hostile submission must NOT be allowed to do, against a kernel interface that grows every release. The microVM judge inverts it — the submission is handed a whole (throwaway) computer and simply cannot reach anything that isn't inside it. For deeper reading on the general problem, see our guides on how to sandbox untrusted code and multi-tenant code execution.

Honest limits

A microVM judge is not magic. Network egress from the guest is real by default — if your threat model includes exfiltration or a submission phoning home, disable outbound networking at the network layer rather than trusting the code (a judge almost always wants zero egress). The hypervisor surface is small but non-zero; keep Firecracker patched. And the VM boundary isolates execution, not your architecture: if you hand the guest a token or a reachable internal service, you've undone it. Within those bounds, a per-submission microVM gives you a judge that treats an adversarial submission as exactly what it is — someone else's problem, contained in a VM you're about to throw away.