Running live technical interviews on microVMs

Every live-coding interview product has the same three moving parts. A browser editor. A Run button. Something on your side that executes whatever the candidate typed and streams stdout back into a panel while an interviewer watches. Then a hidden test suite decides whether the person gets a callback.

Sit with the middle part. For the next forty-five minutes a stranger has an editor, a shell, and a strong incentive to look better than they are, running arbitrary code on machines you pay for. A candidate is untrusted code with a resume attached — and unlike most untrusted-code problems, you can't decline to run it. Running it is the product.

I'm Ajay, I built PandaStack — this post is about the specific shape of that problem for live interviews: per-candidate isolation, a Run button that has to feel instant, hidden tests that stay hidden, time limits fair enough to reject someone with, and a session you can replay when the hiring panel disagrees.

The trust model: a stranger with a shell and a deadline

Interview code is easy to underrate as a threat, because the person writing it is polite and on a video call with your engineer. The threat model doesn't care. The combination of properties is genuinely unusual:

• The code is written live, adapting to your platform. A batch grader faces a fixed submission; a session faces someone who probes, reads your error message, and tries again forty times in an hour.
• The session is long. Forty-five minutes live, or hours of take-home, versus the few seconds a judge gives a submission — long enough that "a little CPU" becomes worth stealing.
• Valuable things are in reach: your hidden tests, your question bank, the other candidates on the same host. Good problems are expensive to write, and one leak to an answers forum retires them.
• The payoff for cheating is a job — a stronger incentive than most abuse you'll ever design against, aimed at a system that hands out a shell voluntarily.
• You owe them fairness. A false negative here isn't a failed job, it's a person who didn't get hired because your runner was busy.

So the requirement isn't only "contain the blast radius." It's that, plus feel like a local machine, keep the answer key behind a wall, and give every candidate identical hardware — simultaneously.

Why "a container with a 5s timeout" fails

Nearly every platform starts here, because it works on day one and the failure modes only appear when someone goes looking. The trouble is that a container is a polite suggestion to the kernel, and candidates include people who read kernel documentation for fun.

• Shared kernel. Every container on the box is one kernel bug from every other container — on a system that deliberately runs attacker-supplied code with a compiler available. It's exactly why AWS runs Lambda on microVMs instead.
• The timeout kills the wrong thing. A wall-clock timeout kills the process you started; a fork bomb, a detached child, or anything that reparents to PID 1 outlives it. Unless you're killing a whole cgroup, you have a timeout on the polite parts.
• /proc is a window into the neighbours. Shared hosts leak process names, command lines, and timing signals from whatever else is running — including another candidate's grading pass, arguments attached.
• Egress cuts both ways. Outbound network lets candidate code POST your hidden tests somewhere, pull an answer from a model API, or fetch a payload you never reviewed — and most runners have open egress because setup needed a package index and nobody closed it afterwards.
• It's a mining rig with a friendly UI. Forty-five minutes of real CPU, requested anonymously through a signup form, is the exact shape of abuse people automate — and take-homes are worse: longer budget, nobody watching, one throwaway email per identity.

The realistic alternatives are a microVM per candidate and the "just run it in the browser" approach — WASM, Pyodide, an in-page interpreter.

• Isolation boundary — Shared container runner: namespaces and cgroups on one host kernel, so a kernel bug is a cross-candidate compromise. Per-candidate microVM: hardware-virtualized guest with its own kernel; blast radius is one disposable VM. Browser/WASM: it runs on their device, so there's nothing of yours to compromise and nothing of yours in control.
• Fork bombs and exhaustion — Container: needs correct PID and memory limits, and the host OOM killer doesn't always pick the offender. MicroVM: fixed vCPU and RAM ceilings baked into the guest, so a fork bomb saturates its own machine and dies with it. Browser: it hangs their tab, indistinguishable from a slow laptop.
• What the candidate can run — Container: anything Linux. MicroVM: anything Linux, including installing packages and serving a port. Browser/WASM: whatever compiled to wasm, so no native extensions and no real subprocesses.
• Hidden test secrecy — Container: tests sit in a filesystem the candidate may be root in; permissions are theatre. MicroVM: a second guest is cheap, so tests never enter the candidate's VM at all. Browser: tests ship to the client, which makes them public — disqualifying on its own.
• Egress control — Container: per-container policy, if someone remembered. MicroVM: its own network namespace and routing, so default-deny is enforced below the guest. Browser: it's their network; you control nothing.
• Fairness of time limits — Container: runtime depends on who else hit Run that second. MicroVM: fixed CPU and memory shape per guest, so a limit means the same thing for everyone. Browser: it becomes a test of their hardware.

The browser option deserves its due — for a pure algorithms round in JavaScript it's cheap, instant, and there's no infrastructure to attack. It falls over the moment the interview involves a real dependency, a build step, or hidden tests. The full trade-off is in /blog/wasm-vs-firecracker-untrusted-code.

The Run button has to feel local

This constraint quietly decides your architecture. A candidate hits Run and expects output when a terminal would give it. If it takes a couple of seconds they don't think "the platform is provisioning a machine" — they think something is broken, hit Run again, and start debugging your product instead of their algorithm, on a clock being used to evaluate them. Latency here is a fairness metric, not a UX one.

It's also why platforms end up with shared runners in the first place. If creating a VM means booting one, per-run isolation is off the table — a ~3s cold boot on every Run is unusable, so you keep a warm pool, and a warm pool means reused machines. Snapshot-restore breaks that loop. On PandaStack every create restores a pre-baked Firecracker snapshot instead of booting: the restore step is around 49ms, end-to-end create is p50 179ms and p99 ~203ms. The ~3s cold boot happens only the first time a template is baked.

A fifth of a second still reads as "it ran" rather than "it queued," which makes a fresh hardware-isolated VM per Run a real option. Concurrency is the other half: each agent pre-allocates 16,384 /30 subnets, so a hiring event with hundreds of simultaneous candidates is bounded by host memory and CPU, not network plumbing. The mechanics are in /blog/snapshot-restore-boot-path.

Session-persistent sandbox vs per-run fresh sandbox

Once per-run VMs are affordable you have a real choice. A session-persistent sandbox lives for the whole interview: files stay put, a pip install happens once, a dev server keeps running, and it feels like a machine they're sitting at. The cost is accumulated state — a run at minute 40 can be affected by something they did at minute 5, like a background process eating CPU or a module they monkeypatched to pass an earlier test. "Did their code work?" acquires an asterisk.

A per-run fresh sandbox restores a clean guest every time. It's deterministic, the fork bomb becomes a non-event, and the run that decides someone's career isn't contaminated by their own earlier experiments. The cost is that nothing persists: re-writing the file set each run is cheap, reinstalling dependencies is not. That's what snapshots are for — prepare the environment once, snapshot it, fork per run. A same-host fork lands in 400–750ms (cross-host 1.2–3.5s), sharing memory copy-on-write until it writes.

1. Live pairing round where the candidate builds and iterates: session-persistent sandbox, ttl_seconds sized to the session plus slack.
2. Algorithms round with a Run button and no dependencies: per-run fresh sandbox — determinism beats persistence here.
3. Take-home with a repo and a build: session-persistent, graded later in a guest the candidate never touched.
4. Anything that produces a score: fresh guest, always. Never grade in the machine the candidate had a shell on.

The hybrid is what I'd ship: a persistent session for the interactive feel, plus a clean forked guest for anything that counts. Here's the session side.

from pandastack import Sandbox


def start_interview(candidate_id: str, starter_files: dict[str, str]) -> Sandbox:
    """One microVM per candidate, created when they join the room."""
    sbx = Sandbox.create(
        template="code-interpreter",
        ttl_seconds=4500,          # 75 min: the session plus slack, then it dies regardless
        metadata={"candidate": candidate_id, "kind": "interview-session"},
    )
    for path, src in starter_files.items():
        sbx.filesystem.write(f"/work/{path}", src)
    return sbx


def on_run_clicked(sbx: Sandbox, source: str) -> dict:
    """Called every time the candidate hits Run. Their code, their machine."""
    sbx.filesystem.write("/work/solution.py", source)

    # Three nested deadlines; the candidate can only reach the innermost one:
    #   timeout -s KILL 10  -> bounds the process inside the guest
    #   timeout_seconds=15  -> bounds the API call if the guest stops answering
    #   ttl_seconds=4500    -> bounds the VM itself, enforced outside the guest
    r = sbx.exec(
        "cd /work && timeout -s KILL 10 python3 solution.py",
        timeout_seconds=15,
    )
    return {
        "stdout": r.stdout[-16000:],   # truncate: someone will print in a loop
        "stderr": r.stderr[-4000:],
        "exit_code": r.exit_code,
        "timed_out": r.exit_code == 137,
    }


def end_interview(sbx: Sandbox) -> None:
    sbx.kill()   # background processes, temp files, and the miner all go with it

Note the layered deadlines. The guest-side kill handles the ordinary infinite loop; the call timeout handles a guest that stopped answering because someone exhausted its PIDs; the TTL handles everything else, enforced where the candidate can't reach it. Truncating stdout matters too — printing in a tight loop is the most common accidental denial-of-service in interview products, and it's aimed at your websocket, not your CPU.

Hidden tests live outside the guest, or they aren't hidden

The rule has no exceptions worth taking: hidden test cases never exist inside a machine the candidate can type into. Not in a file with restrictive permissions — they're root, or one step from it, because it's their VM. Not in an env var, not compiled to bytecode, not obfuscated. If the bytes are in the guest, the person in the guest can read them.

That leaves two patterns, really one with different plumbing. Either grade outside the guest — read the candidate's source out and run tests in a sandbox they never had access to — or inject tests only after their code is captured and frozen. Capture first, then inject: a candidate handed your hidden tests at minute 10 doesn't write a general solution, they write an if statement.

import json

from pandastack import Sandbox


def grade(candidate_id: str, session: Sandbox, tests_src: str) -> dict:
    # 1. Capture FIRST. After this line the answer is frozen; anything they type
    #    afterwards changes their session, not what we're grading.
    submitted = session.filesystem.read("/work/solution.py")
    archive_submission(candidate_id, submitted)

    # 2. Grade in a guest the candidate has never had a shell on. The hidden tests
    #    exist here and in your control plane -- never in their VM.
    grader = Sandbox.create(
        template="code-interpreter",
        ttl_seconds=300,
        metadata={"candidate": candidate_id, "kind": "grading"},
    )
    try:
        grader.filesystem.write("/work/solution.py", submitted)
        grader.filesystem.write("/work/test_hidden.py", tests_src)

        r = grader.exec(
            "cd /work && timeout -s KILL 60 python3 run_tests.py "
            "--tests test_hidden.py --out report.json",
            timeout_seconds=90,
        )
        if r.exit_code != 0 and not wrote_report(r):
            return {"status": "error", "detail": r.stderr[-2000:]}

        report = json.loads(grader.filesystem.read("/work/report.json"))

        # 3. Only counts and case names cross back to the candidate-facing UI.
        #    Never assertion text -- that's how a suite gets reverse-engineered.
        return {
            "passed": report["passed"],
            "total": report["total"],
            "failed_cases": [c["name"] for c in report["cases"] if not c["ok"]],
        }
    finally:
        grader.kill()

The last step is the one people skip. Stream full assertion output into the candidate's panel and you've published the hidden tests one failure message at a time; a patient person will binary-search the suite out of your error strings. Return counts and case names, keep the diffs for the reviewer. And rotate question banks assuming they leak — nobody needs to exfiltrate the problem they just spent an hour on.

Fairness, determinism, and the LLM in the next tab

Time limits are the part of interview infrastructure most likely to be quietly unfair. On a shared machine, whether an O(n log n) solution clears a two-second threshold depends on who else pressed Run that second. That isn't a flaky test — it's rejecting a person because of someone else's for-loop, and you'll never notice, because from your side it looks like a candidate who didn't pass.

A microVM per candidate gives you a fixed machine shape. Firecracker can't change vCPU or RAM at snapshot restore, so guest size is a property of the baked template: everyone gets the same CPU count and memory ceiling, and an OOM happens inside their guest against their own ceiling rather than being adjudicated by a host OOM killer picking whichever process looked biggest.

• Bake one template per round and use it for every candidate, so the only variable is their code.
• Prefer CPU time over wall clock for pass/fail, and treat wall clock as a safety net rather than the measurement.
• Set limits with headroom, not on a knife edge. If your threshold can't reliably separate a 1.8s run from a 2.1s run, it can't separate an efficient solution from an inefficient one either.
• Judge complexity in the conversation and use the timer for the obvious cases. A time limit is good at catching an accidental O(n²) and bad at being the reason someone doesn't get hired.

Then there's egress, which is where the anti-cheat conversation actually belongs. Each sandbox gets its own network namespace and routing, so policy is per-candidate rather than fleet-wide: default-deny, open one hole if setup needs it, close it before the session starts.

#!/usr/bin/env bash
# Egress policy for an interview guest. Applied per-sandbox -- each microVM has
# its own netns, so this is one candidate's network, not the fleet's.
set -euo pipefail

MIRROR_IP="${PACKAGE_MIRROR_IP:?internal package mirror, used during setup only}"

iptables -P OUTPUT DROP
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Cloud metadata: the first thing any escalation attempt reaches for.
iptables -A OUTPUT -d 169.254.0.0/16 -j DROP

# No lateral movement: your control plane, your test bank, the next candidate.
iptables -A OUTPUT -d 10.0.0.0/8     -j DROP
iptables -A OUTPUT -d 172.16.0.0/12  -j DROP
iptables -A OUTPUT -d 192.168.0.0/16 -j DROP

# Setup-only hole: an internal mirror, so a starter repo can install its deps.
# Nothing here reaches a model API, a pastebin, or the candidate's own server.
iptables -A OUTPUT -d "$MIRROR_IP" -p tcp --dport 443 -j ACCEPT

# Then, before the candidate joins, drop even that:
#   iptables -D OUTPUT -d "$MIRROR_IP" -p tcp --dport 443 -j ACCEPT
# The interview itself runs with no outbound network at all.
iptables -S OUTPUT

Be honest about what this buys. It does not stop cheating: a candidate with a phone is outside your threat model and always will be. It stops the cheats that route through your infrastructure — code calling a model API from inside the guest, a dependency phoning home, a script POSTing your hidden tests somewhere — and it closes the exfiltration path for your question bank, which is the asset you can actually protect. The general pattern is in /blog/controlling-network-egress-untrusted-code.

Network isolation is not proctoring. It stops your platform from being the vector; it does not stop the second laptop. Design an interview a pasted answer can't survive, and let the infrastructure do the part infrastructure is good at.

Replay: snapshot the session so the panel argues about facts

Hiring decisions get revisited: two interviewers remember the session differently, a candidate appeals, a manager wants the work behind a borderline score. Most platforms answer with a code paste and a transcript, which loses how the person got there.

Because the session is a VM, you can call sbx.snapshot() before sbx.kill() and keep the whole working state: the file tree including scratch files they deleted from the editor, the shell history, the installed packages. A reviewer later calls snap.fork() for their own copy — memory shared copy-on-write until they touch it, so two reviewers can each poke at a session without stepping on each other.

Two caveats, both non-negotiable. A session snapshot is a recording of a person's work: say so in the candidate-facing terms, give it a retention window, and delete it when the window closes. And restore review forks under the same egress policy as the interview — a snapshot restored with open network is a machine full of someone else's code that nobody is watching.

Honest limits

None of this makes interview infrastructure a solved problem. Per-candidate VMs cost more engineering than a container pool: session lifecycle, snapshot hygiene, storage budget for replays, and a plan for when a create fails while a candidate watches a spinner. A fixed guest shape improves determinism without eliminating variance — page cache and host contention still exist — so knife-edge time limits stay a bad idea on any substrate. And hardware isolation is stronger than a shared kernel, not perfect: hypervisor escapes are rare, not mythical.

And microVMs cannot help with the human end. If your interview is a puzzle with a known answer, the answer is on the internet, and the only durable defence is making the candidate explain what they wrote. What the infrastructure gets you is narrower and worth having anyway: candidates can't reach each other or your test bank, can't turn your fleet into someone's mining operation, and all get an identical machine with an identical clock — so when someone doesn't pass, it's because of their code.

For the batch side of this, /blog/microvm-autograder-student-code-isolation covers grading a whole class and /blog/online-judge-microvm-isolation covers adversarial contest judging. For the mechanics under the fast Run button and the per-run forks, see /blog/snapshot-and-fork-explained.