Proof checking has the best reputation in computing. A proof assistant is the one program everybody agrees is trustworthy: a small kernel, decades of logic behind it, and an answer that is not a probability. So when a team builds an RL pipeline that generates a few million candidate Lean proofs a day, the checker tends to get wired in like a linter. Pipe the file to lean, read the exit code, hand out the reward. What could a proof possibly do?
Quite a lot, it turns out, and in two separate ways that people routinely blur together. The first is that checking a proof runs code. Lean 4 elaborates a file by executing it: tactics, macros, custom elaborators and #eval are programs, and the ones defined in the submission run with the checker's privileges. The second is that the verdict can be forged. A proof that cannot break out of anything can still try to convince your pipeline it proved something it did not, and an RL policy is, by construction, a search process pointed at exactly that gap.
I build PandaStack, an open-source Firecracker microVM platform, and this post is about both halves: what a microVM per check buys you for side effects, why it buys you nothing for soundness on its own, and the two-stage layout I would use to get both. Most of the examples are Lean 4 because that is where most AI-math work currently lives; Coq/Rocq and Isabelle get their own section, because they have the same two problems wearing different clothes.
Why checking a proof runs arbitrary code
A Lean 4 source file is not a passive document that a kernel reads. It is a program for the elaborator, and the elaborator is extensible from inside the file. That is the feature that makes Lean pleasant to work in and Mathlib possible, and it means the honest description of lean Submission.lean is "compile and run the submitter's metaprograms, then type-check what they produce". The relevant entry points:
#evalevaluates a term at elaboration time, and if the term is anIOaction it performs it.#eval IO.Process.run { cmd := "sh", args := #["-c", "..."] }is a perfectly ordinary line of Lean.run_cmd,run_tacand friends run monadic code in the command and tactic monads, which can reachIOthrough the usual lifts.- Custom
macro,syntax,elabandtacticdefinitions in the submission run during elaboration of whatever follows them in the same file. A tactic is a program;by my_tacticexecutes it. initializedeclarations and@[extern]/@[implemented_by]attributes connect Lean names to compiled or native code. They matter less for direct side effects in a single-file check and a great deal for soundness, as below.- Lake. If you accept a whole project rather than one file,
lakefile.leanis itself a Lean program Lake runs,requireclones git repositories, and builds can compile C throughextern_libtargets. Accepting a lakefile is morally annpm installfrom a stranger.
None of this is a vulnerability in Lean, and nobody on the Lean side claims the elaborator is a sandbox. It is a compiler with a macro system, and compilers with macro systems run code. The mistake is only on our side, when we treat "check this proof" as a read-only operation and run it next to the reward model's credentials.
Resource bombs: the checker as a denial-of-service target
Even a submission with no IO at all can make the checker miserable, and in an RL setting you will see these constantly, mostly not from malice but because a policy that has learned decide sometimes works will try it on a proposition the size of a phone book.
- Elaboration blowup.
decideorsimpon a large goal, type-class resolution that wanders, unification that unfolds definitions forever. Lean hasmaxHeartbeats(a per-declaration budget measured in small allocations) andmaxRecDepth, and they help — but they are options, andset_option maxHeartbeats 0in the submission switches the heartbeat limit off. Any limit the input can configure is advice, not a limit. - Memory. A blown-up term or a runaway kernel reduction can eat RAM until something dies, and on a shared box that something is chosen by the host's OOM killer rather than by you.
- Disk and output. A metaprogram can write files or print until the log pipeline falls over;
#evalof a hugeStringis enough. - Wall clock. A tactic loop with no allocation can spin without ever exhausting heartbeats; only an external clock stops it reliably.
The only limits worth relying on are the ones the submission cannot reach: the VM's RAM, a timeout -s KILL around the process, the SDK's timeout_seconds around the exec, and the sandbox's own lifetime. A bomb inside a per-check microVM exhausts that VM's memory and that VM's clock, fails exactly one candidate, and takes nobody's batch with it.
The other half: a proof that tries to fool the checker
Here is the part isolation does not help with at all. Suppose the VM is perfect: nothing gets out, nothing persists, every bomb is contained. The submission can still aim at your verdict. This is the interesting threat model for RL specifically, because you are running an optimiser against the checker. Whatever cheap path to reward exists, the policy is being paid to find it — not hypothetically, but as the literal objective.
The usual suspects in Lean 4, roughly in order of how cheap they are:
sorry. Elaboration succeeds with a warning, and the theorem depends on the axiomsorryAx. If your pipeline reads the exit code,sorryis a valid proof of everything.- New axioms.
axiom cheat : Falseis legal Lean. So is a more tasteful axiom that happens to imply the goal. The kernel checks that the proof follows from the axioms; it does not check that you like the axioms. native_decide. It proves a decidable proposition by compiling and running the decision procedure, and the proof depends onLean.ofReduceBool, which trusts the compiler and everything linked into it. Combine it with@[implemented_by]or@[extern]in the same file — tell the compiler the procedure is implemented by a function that just returnstrue— and you can obtain a proof of a false statement. Newer toolchains may surface additional trust axioms here (Lean.trustCompiler, for instance); check which names your version reports rather than trusting this list.- Skipping the kernel.
set_option debug.skipKernelTC trueasks Lean to add declarations without kernel type-checking. The declarations then exist in the environment, and in the compiled.olean, unchecked. - Lying in the same process. A metaprogram can print "no errors", write a fake
#print axiomsresult, define a macro that intercepts the very command you append to audit the file, or exit 0 after doing whatever it liked. Anything you learn from the process that ran the submission, you learned from the submission. - Proving a different theorem. The cheapest cheat of all is proving something adjacent: a theorem with the right name and the wrong statement, a statement that elaborates differently because the submission redefined notation or an instance, or a goal made vacuous by a hypothesis that is never satisfiable.
The lesson is a structural one. A Lean file's output is computed by the file. The kernel is trustworthy; the process around the kernel, once it has run your opponent's metaprograms, is not. So the verdict has to come from somewhere that has not executed the submission at all.
What a trustworthy verdict actually requires
For Lean 4, a reasonable bar — and I want to be careful to call it reasonable rather than complete — is three checks, all performed in a context that never parsed the submission's syntax:
- Kernel replay. Take the compiled
.oleanand feed every declaration back through the kernel.lean4checkeris the community tool for exactly this, and it is what catchesdebug.skipKernelTCand anything else that put an unchecked declaration into the environment. For extra confidence you can export the environment and re-check it with an independent kernel implementation (more than one exists for Lean 4, written by different people in different languages; check which are maintained for your toolchain); a bug would then have to exist in two unrelated kernels at once. - Axiom audit. Collect the axioms the target theorem transitively depends on and allow only the standard three,
propext,Classical.choiceandQuot.sound. Anything else —sorryAx,Lean.ofReduceBool, a submission-declared axiom — is a reject, or at least a separate, lower-reward category you decide on deliberately. Do this with compiled code that loads the module as data and walks the declarations (Lean ships aCollectAxiomsutility you can call from a small executable), not by appending#print axiomsto a file that imports the submission, because imported syntax extensions are live in the importing file. - Statement comparison. Write the statements yourself, in a
Specmodule that is compiled into your environment and that the submission must import, and require the submission's theorem to have type exactlySpec.main_statement— compared as kernel terms, not as source text. Syntactic equality is conservative: a definitionally-equal restatement gets rejected, which in a reward pipeline is the right direction to be wrong in.
.olean is a memory image that Lean maps and trusts; as far as I know the loader is not designed to defend against a deliberately malformed file. So the audit stage is reading attacker-produced bytes with a parser that assumes they are honest. That does not change the logic of the checks, but it is a reason to run the audit in a disposable VM too, as the uploaded file's last stop, and not on your control plane. If that bothers you, the stronger option is an export-format check by an independent checker, which parses text rather than mapping a binary image.Why a microVM per check, and not a container or a seccomp profile
The side-effect half of the problem is the standard untrusted-code problem with one awkward property: the toolchain is large, native, and does a lot of mmap. Lean loads Mathlib's .olean files by mapping them, the compiler and interpreter are a sizeable C++ and C runtime, and Lake shells out to git and a C compiler. That is a big syscall surface to narrow with a filter, and it changes with every toolchain bump, which in an AI-math pipeline happens every time Mathlib moves.
A container shares the host kernel with the checker, so a kernel bug reachable from that syscall surface is a host compromise; a container is, at bottom, a polite suggestion to the kernel. A Firecracker microVM gives each check its own guest kernel behind KVM, so the boundary becomes the hypervisor's small device model rather than the whole Linux syscall table. Inside the guest you can be as root-ish and syscall-heavy as Lean wants, and none of it is your host's problem. It also gives you the right failure granularity: one candidate's resource bomb, kernel panic or filled disk ends one VM.
The usual objection to a VM per unit of work is boot time, and for proof checking it would be a fair one if every check booted Linux and then imported Mathlib from a cold page cache. That is where the template snapshot comes in.
The template: Mathlib built once, restored for every check
A Mathlib checkout with its compiled .olean files is several gigabytes, and building Mathlib from source takes far longer than anyone wants to pay per proof. Even with the prebuilt cache from lake exe cache get, you do not want each check downloading it. So build it into a custom template once: the toolchain, a pinned Mathlib, your Spec module and your audit executable, all owned by root and read-only to the user that will run submissions.
# templates/lean-mathlib/Dockerfile
FROM ubuntu:24.04
RUN apt-get update && apt-get install -y --no-install-recommends \
curl git ca-certificates build-essential \
&& rm -rf /var/lib/apt/lists/*
# Toolchain + Mathlib live under /opt, owned by root, read-only to the
# user that runs submissions. Pin everything: lean-toolchain, the Mathlib
# rev in lake-manifest.json, and your own Spec/Audit sources in proj/.
ENV ELAN_HOME=/opt/elan PATH=/opt/elan/bin:$PATH
RUN curl -sSfL https://raw.githubusercontent.com/leanprover/elan/master/elan-init.sh \
| sh -s -- -y --no-modify-path --default-toolchain none
COPY proj/ /opt/proj/
WORKDIR /opt/proj
# Prebuilt Mathlib .olean files from the cache, then YOUR modules:
# Spec (the statements you want proved) and the audit executable.
RUN lake exe cache get && lake build
RUN useradd --create-home --shell /usr/sbin/nologin checker \
&& mkdir -p /work && chown checker: /work \
&& chmod -R a-w /opt/elan /opt/proj# The default template RAM is 1 GiB; importing Mathlib wants several.
# Size both to what you measure, not to this example.
pandastack template build --name lean-mathlib -f templates/lean-mathlib/Dockerfile \
--context templates/lean-mathlib --memory-mb 8192 --size-mb 20480
# First create of a new template cold-boots (~3 s) and bakes a snapshot;
# every create after that restores it.On PandaStack every create restores the template's baked Firecracker snapshot rather than booting: 179 ms p50 and about 203 ms p99 for the whole create, of which the snapshot restore itself is around 49 ms. The guest's rootfs is a copy-on-write clone (XFS reflink or dm-snapshot), so the multi-gigabyte Mathlib tree is not copied per VM; each check sees its own writable view and its writes vanish with it. Guest memory is restored copy-on-write as well. There is no warm pool of idle VMs being kept alive between checks — each one is restored on demand — which is also why the snapshot matters more than the box size.
Two honest caveats. First, the restore is fast; import Mathlib inside the restored VM is a separate cost, Lean loading and mapping a large number of .olean files, and it can easily dominate the 179 ms. Measure it on your toolchain rather than assuming. Second, size the template: the default template RAM is 1 GiB, which is not enough to import Mathlib comfortably, and RAM is fixed at bake time because a snapshot cannot change it on restore. If you use the full import Mathlib, try importing only the modules your problem set needs; it is the cheapest speedup available.
The mechanics of restore-on-every-create are covered in The Snapshot-Restore Boot Path: Every Sandbox in Under 200ms, if you want to know what those 49 ms are made of.
No network: what PandaStack gives you and what you add
A proof checker never needs the internet. Mathlib is on the disk, the statement is on the disk, and the only output is a verdict. So the right posture is no route out at all, and I want to be precise about who provides it.
PandaStack gives every sandbox its own network namespace, veth pair and TAP device — 16,384 pre-allocated /30 subnets per agent — and the host firewall drops VM-to-VM traffic, the cloud-metadata link-local range and standard mining-pool ports. It does not have a per-sandbox "no network" switch today: outbound internet is open by default on the hosted platform. So the no-network layer is one you add inside the guest, which is cheap because the guest is yours.
The approach in the code below is to run every Lean process with unshare --net, which puts it in a fresh network namespace containing only a loopback interface, and to drop to an unprivileged checker user with setpriv after that. The unprivileged process cannot undo the namespace, and to get the network back it would first need a guest-kernel exploit, at which point it is root in a VM holding nothing but Mathlib and its own proof, with the hypervisor still between it and you. Do not try to get the same effect by taking the guest's interface down: the SDK's exec channel is SSH over that interface, and you would be cutting off the arm you are using to collect the verdict. Also check that your guest kernel supports network namespaces before relying on this — run a connect attempt under the same wrapper once per template build and fail loudly if it succeeds. The broader pattern, including egress allowlists for workloads that do need the network, is in Controlling Network Egress for Untrusted Code.
The other half of "no network" is having nothing worth sending. The checker VM gets the submission and nothing else: no API key, no reward-model endpoint, no other candidates. Your PANDASTACK_API_KEY lives in the orchestrating process on your side and is never written into the guest.
The checking loop: two VMs per proof
Here is the whole thing in Python. Stage 1 elaborates the untrusted file in a fresh VM and keeps only the .olean it produced. Stage 2 copies that file into a second fresh VM that has never executed submitted code, replays it through the kernel, and runs your audit executable. Stage 1's stdout is kept as a log for humans and never consulted for the verdict.
"""Check a batch of candidate Lean 4 proofs: two throwaway microVMs each."""
import json
from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox, SandboxTimeout
TEMPLATE = "lean-mathlib" # the Dockerfile above
STANDARD = {"propext", "Classical.choice", "Quot.sound"}
# Every Lean process runs as the unprivileged `checker` user, in its own
# network namespace (loopback only), under a hard wall clock. We do NOT
# take eth0 down: the SDK's exec channel is SSH over the guest's NIC.
JAIL = ("timeout -s KILL {secs} unshare --net "
"setpriv --reuid=checker --regid=checker --init-groups ")
COMPILE = ("cd /opt/proj && " + JAIL.format(secs=150) +
"lake env lean -o /work/Submission.olean /work/Submission.lean")
# Stage 2 never parses the submission's syntax. lean4checker replays every
# declaration in the .olean through the kernel; `audit` is OUR compiled
# Lean program: it loads the module as data and reports the axioms behind
# Submission.main_theorem and whether its type is exactly Spec.main_statement.
LEANPATH = "LEAN_PATH=/work:$(lake env printenv LEAN_PATH) "
REPLAY = ("cd /opt/proj && " + LEANPATH + JAIL.format(secs=300) +
"lake env lean4checker Submission")
AUDIT = ("cd /opt/proj && " + LEANPATH + JAIL.format(secs=60) +
".lake/build/bin/audit Submission Submission.main_theorem")
def run(sbx, cmd, secs):
try:
return sbx.exec(cmd, timeout_seconds=secs)
except SandboxTimeout:
return None
def check(proof_id: str, source: str) -> dict:
v = {"id": proof_id, "ok": False}
# Stage 1: elaborate the untrusted file. Anything can happen in here,
# including a metaprogram that prints "all good" and exits 0.
with Sandbox.create(template=TEMPLATE, ttl_seconds=300,
metadata={"proof": proof_id, "stage": "elab"}) as sbx:
sbx.filesystem.write("/work/Submission.lean", source)
sbx.exec("chown checker: /work/Submission.lean")
r = run(sbx, COMPILE, 180)
if r is None or r.exit_code != 0:
v["reason"] = "elab timeout" if r is None else f"elab exit {r.exit_code}"
v["log"] = "" if r is None else (r.stdout + r.stderr)[-4000:]
return v
olean = sbx.filesystem.read("/work/Submission.olean")
# Stage 2: a FRESH VM that has never run submitted code.
with Sandbox.create(template=TEMPLATE, ttl_seconds=300,
metadata={"proof": proof_id, "stage": "audit"}) as sbx:
sbx.filesystem.write("/work/Submission.olean", olean)
sbx.exec("chown checker: /work/Submission.olean")
replay = run(sbx, REPLAY, 330)
if replay is None or replay.exit_code != 0:
v["reason"] = "kernel replay failed"
return v
a = run(sbx, AUDIT, 90)
if a is None or a.exit_code != 0:
v["reason"] = "audit failed"
return v
report = json.loads(a.stdout)
extra = sorted(set(report["axioms"]) - STANDARD)
v.update(ok=report["statement_matches"] and not extra,
axioms=report["axioms"], nonstandard=extra)
return v
if __name__ == "__main__":
candidates = load_candidates() # [(proof_id, lean_source), ...]
with ThreadPoolExecutor(max_workers=32) as pool:
for verdict in pool.map(lambda c: check(*c), candidates):
print(json.dumps(verdict))A few things in there that are easy to get wrong:
- The verdict is computed from stage 2 only. Stage 1's exit code is used to reject early, never to accept; a nonzero exit is a failure, a zero exit merely earns the candidate a stage 2.
- Timeouts are layered.
timeout -s KILLinside the guest stops the Lean process;timeout_secondsonexecstops the SDK waiting even if the guest misbehaves, and raisesSandboxTimeout;ttl_secondsis an idle timeout that reaps a VM the orchestrator forgot about. It is not a hard lifetime, which is why the hard clock is the in-guesttimeout. - The context manager kills each VM on exit, including on exceptions, so a check that blows up takes its VM, its disk writes and its network namespace with it.
metadatatags each VM with the proof id and stage, which is what you will want when one of them misbehaves at 3 a.m. and you are trying to work out which of two million candidates it was.- Concurrency is a thread pool on your side. Each VM gets its own 8 burstable vCPUs fair-shared under contention, so the practical ceiling is host memory, which is why the template's RAM figure is the number to tune.
One VM per proof or one per batch?
Two VMs per proof is the clean design. It is not the only defensible one, and at millions of candidates a day the trade-offs are worth writing down:
| Layout | What it costs | What it shares | Where it fits |
|---|---|---|---|
| VM per proof, two stages | Two creates per candidate plus Mathlib import twice | Nothing between candidates | Rewards that matter: final evaluation, leaderboards, anything adversarial |
| VM per batch, process per proof | One create per batch; import Mathlib per process | The guest's filesystem and /tmp across the batch; one bomb can take the batch | High-volume RL rollouts where you re-verify winners in the strict layout |
| VM per batch, persistent REPL | Import Mathlib once per batch | One Lean process: a metaprogram can poison the environment for every later proof | Fast filtering of obviously broken candidates only; never the final verdict |
| Shared host, no VM | Cheapest | Everything, including your credentials | Proofs you wrote yourself |
The pattern I would actually run for RL is a funnel: cheap batched checking to discard the large majority of candidates that do not even elaborate, then the strict two-VM layout for every candidate that is about to earn reward. A false negative in the cheap stage costs you one sample; a false positive in the strict stage trains the policy to cheat. If per-batch VMs are part of your plan, give each proof its own working directory and its own checker process at minimum, and assume that anything one proof can write, a later proof in the same VM can read. For the general version of this rollout-isolation argument, see Running RL environments in microVMs: isolating rollout workers.
Coq/Rocq and Isabelle: same two problems
The details differ, and I am less deep in either than in Lean, so check these against your version. The shape does not differ.
- Coq/Rocq, code execution —
Declare ML Moduleloads compiled OCaml plugins into the running prover, which is native code with the prover's privileges, andLoad/Requirepull in other files. A submission that can choose its plugins can run anything. Fix the plugin set in your template and reject submissions that declare their own. - Coq/Rocq, soundness —
Print Assumptionslists the axioms a theorem depends on, includingAdmittedlemmas, and should be part of your audit.vm_computeandnative_computetrust a bytecode VM and a native compiler respectively, which is the same class of trust as Lean'snative_decide. For an independent verdict,coqchkre-checks compiled.vofiles outside the elaborator; under the Rocq 9 renaming the tooling names have been changing, so look up the checker for your release. - Isabelle, code execution —
ML ‹…›blocks run Isabelle/ML inside the prover session, and Isabelle/ML can reach the operating system (Isabelle_System.bashis the obvious example). A theory file is a program in the same sense a Lean file is. - Isabelle, soundness — the LCF-style kernel means theorems can only be built through trusted inference rules, but
sorry(underquick_and_dirty) and oracles exist, and the kernel tracks which theorems depend on them. Audit that dependency information for your target theorem rather than trusting a clean session exit, and keepquick_and_dirtyoff in the checking session.
The same two-stage idea applies to all three: isolate the session that runs the submission, and derive the verdict from an artefact checked by something that did not run it.
What this design does not protect you from
- Kernel bugs. If the Lean kernel accepts a false proof, every check here passes. The independent re-check with a second kernel implementation is the mitigation; reporting it upstream is the fix.
- A wrong specification. If
Spec.main_statementis not the theorem you meant — a missing hypothesis, an off-by-one in a definition, a vacuous premise — the pipeline will cheerfully certify proofs of your mistake. Formalisation errors are the most common way a "verified" result is wrong, and no VM helps. - Reward design. Rejecting
native_decidemakes some true theorems unrewardable; allowing it trusts the compiler. That is a policy choice you should make deliberately and write down, not one the checker should make by default. - Cost. Two VMs and two Mathlib imports per candidate is real compute at RL scale. The funnel above is the honest answer, not a promise that strict checking is free.
The rest of the evaluation story — harnesses, graders, execution-based scoring — has its own posts: Running AI Agent Eval Harnesses in Isolated microVMs covers the harness side, and LLM-as-Judge vs Execution-Based Evals: Why Your Code Evals Need a Sandbox covers why scoring by execution needs a sandbox at all.