Running RL environments in microVMs: isolating rollout workers

Most sandboxing problems are about code that might be malicious. Reinforcement-learning environments are a different animal: the code inside is not merely untrusted, it is under active optimization pressure to find whatever your environment does not handle. If there is a shortcut to reward that runs through your host, gradient descent will find it, patiently, across a few million episodes, without ever getting bored. That is not a hypothetical threat model. That is the training objective.

I'm Ajay — I build PandaStack, which runs Firecracker microVMs as a service, so I have an obvious bias and I'll flag it where it matters. This post is about the practical shape of running RL rollout workers and agent-training environments (gym-style envs, terminal-bench-style task environments, tool-use loops) each inside their own microVM: why the workload is unusually hostile, how snapshot-restore turns "reset the environment" into a millisecond operation instead of a provisioning job, how forking lets you branch a mid-episode state N ways, and where this is genuinely overkill.

Why RL environments are the most hostile workload you can host

Ordinary untrusted code is written once by someone with an intent — a customer, a model, an attacker. It tries a thing, it succeeds or fails, it goes away. An RL environment sits in a loop with a policy that is being explicitly rewarded for finding high-return actions, including the ones you did not enumerate. Every misconfiguration in your environment is a reward signal waiting to be discovered, and the search is automated, parallel, and relentless.

The failure modes are funnier than they are exotic. A policy trained to make a test suite pass discovers that deleting the test file makes it pass. A coding agent learns that `git checkout -- .` reverts its own broken change and restores a green build, so it optimizes toward doing nothing at high confidence. A terminal-task policy discovers that `sudo` exists in the image and that many tasks become trivially solvable once you are root — which is a perfectly correct observation about the reward function and a completely useless policy. And my personal favourite class: the environment writes the grading script into the same filesystem the agent can write to, and the policy learns to edit the grader. It scores 100%. It has learned nothing except that you were careless.

Those are the polite failures, because they stay inside the episode. The impolite ones leak. A rollout worker that shares a kernel with 63 siblings and finds a container escape does not just corrupt one episode's reward, it corrupts your whole training run and possibly your host fleet. A worker that can reach a shared package mirror or a shared results cache can poison state that other workers read, and you will spend a week convinced your loss curve has a numerical bug. A worker with outbound internet can exfiltrate, phone home, or — much more likely and much more embarrassing — solve the benchmark task by downloading the reference solution from GitHub.

The reset problem: why the usual sandboxes get awkward here

RL has a specific operational demand that ordinary sandboxing doesn't: `reset()` is called constantly, and it must return the environment to a state that is byte-for-byte equivalent to the last time you called it. If a policy can leave residue across an episode boundary — a lingering process, a mutated config, a warm cache, a file in `/tmp` — then episode N+1 is not a sample from the same distribution as episode N, and your training signal is quietly contaminated. Worse, this contamination is correlated with policy behaviour, which is the exact thing you are trying to measure.

So you have three unappealing options with conventional tooling. Reset in-process, which is fast but trusts the environment code to clean up after an adversary — it will not. Tear down and recreate a container, which is cleaner but re-pays image setup, dependency load, and service warmup on every episode, and still shares a kernel. Or provision a fresh VM per episode, which is genuinely clean and, at classic VM boot times, absurd: you cannot afford a thirty-second boot in a loop that wants thousands of resets per minute.

The microVM argument is that the third option stopped being absurd. If creating a hardware-isolated VM costs roughly the same as an HTTP round trip, "throw the machine away" becomes a viable implementation of `reset()`, and the whole category of residue bugs disappears by construction rather than by discipline.

Snapshot-restore is the reset

The mechanism that makes this work is snapshot-restore. You boot your environment once, let it finish everything slow — install dependencies, warm the interpreter, start the database, load the fixture data, import torch, whatever your env's cold path is — and then freeze the running machine to a memory file plus a state file. From then on, "create an environment" means restoring that frozen machine, not booting one.

On PandaStack that's the normal creation path rather than an optimization you opt into: every create restores a baked snapshot on demand, at p50 179ms and p99 around 203ms end to end, of which the Firecracker restore step itself is about 49ms. The first-ever spawn of a template with no snapshot yet is a real cold boot at roughly 3 seconds — you pay that once, when the template gets baked, not per episode. Whether you use PandaStack or run Firecracker yourself, the shape is the same: bake once, restore forever.

Two things follow that matter more for RL than for typical sandbox workloads. First, the restored machine is warm — it is not a fresh boot that then has to import numpy, it is the exact process state you froze, including page cache and loaded libraries. Your env's expensive setup is amortized across every episode that will ever run. Second, the reset is total. Not "we cleaned up the files we know about" but "that machine's kernel, memory, and filesystem no longer exist." A policy cannot leave a message for its future self in a machine that has been deleted.

Determinism, or: the bug that only happens on worker 41

A VM boundary buys you a clean starting state. It does not buy you determinism — that part is still your job, and it is the part people skip. Snapshot-restore actually helps more than it looks like it should, because a restored snapshot starts from an identical memory image every time, which eliminates a surprising amount of ordering nondeterminism that a fresh boot would reintroduce (service start order, lazily-populated caches, hash seeds chosen at process start).

But the classic offenders survive: thread-count-dependent floating-point reductions, wall-clock timestamps baked into outputs, unpinned package resolution, and network calls whose latency changes which branch of a timeout you take. Pin them inside the image, before you bake, and verify by running the same seed twice and comparing a hash of the trajectory. If you cannot get two identical traces on one machine, you have no business scaling to four thousand workers and hoping.

# =================================================================
# Run this INSIDE the env image, before you bake the snapshot.
# Anything left floating here reappears later as an unreproducible
# episode and three days of "it only fails on worker 41".
# =================================================================
set -euo pipefail

# 1. Sources of nondeterminism that are cheap to pin.
export PYTHONHASHSEED=0        # dict/set iteration order
export OMP_NUM_THREADS=1       # BLAS thread count changes float reductions
export MKL_NUM_THREADS=1
export TZ=UTC                  # any task that formats a date is seed-sensitive
export LC_ALL=C.UTF-8          # sort order, string collation

# 2. Freeze the package set. "pip install -U" at episode time is a
#    nondeterminism generator with a friendly name.
pip install --require-hashes -r requirements.lock

# 3. Make the grader unwritable by the policy. Yes, this is obvious.
#    Yes, everyone learns it the hard way at least once.
install -o root -g root -m 0500 grade.py /opt/grader/grade.py
chown -R root:root /opt/grader

# 4. Prove reproducibility BEFORE trusting a single reward number:
#    same seed twice, compare the trajectory hash.
for i in 1 2; do
  python -m env.rollout --seed 1234 --steps 200 --policy replay.jsonl \
    | sha256sum | cut -d' ' -f1 > "/tmp/trace-$i.sha"
done

if diff -q /tmp/trace-1.sha /tmp/trace-2.sha >/dev/null; then
  echo "deterministic: $(cat /tmp/trace-1.sha)"
else
  echo "NOT deterministic -- fix this before scaling to 4,000 workers" >&2
  exit 1
fi

One more determinism trap that is specific to snapshots and bites people in production: a restored guest wakes up with the clock it had at bake time. If your environment cares about wall-clock time — TLS certificate validity, token expiry, anything that formats "now" into an observation — you need the clock resynced on restore. It's a solved problem, but it is not a free one, and it is the kind of thing that makes an env look nondeterministic when it is actually just time-travelling.

Fork: branch a mid-episode state N ways

Restore-as-reset gives you cheap episode starts. Fork gives you something RL people usually cannot have at all: cheap branching from an arbitrary point mid-episode. Take a machine that is 40 steps into a task, with all the mess that implies — half-edited files, a running dev server, a database with rows in it — and produce eight independent copies of it that then diverge. Copy-on-write memory and a reflinked rootfs mean the copies share pages with the parent until something writes, so branching is cheap in both time and RAM. On PandaStack a same-host fork lands in 400–750ms; a cross-host fork, which has to move the snapshot between machines, is 1.2–3.5s.

That primitive maps onto several things you probably already want. Best-of-N sampling where all N candidates start from the identical world state, not from a re-executed approximation of it. Tree search / MCTS over environment states, where expanding a node is a fork rather than a replay of the whole action prefix. Counterfactual credit assignment: run the same prefix, branch, take action A in one child and action B in another, compare terminal reward. And plain old debugging — freeze the exact machine where the policy did the deranged thing, and poke at it afterwards at human speed.

Per-worker network isolation (the part people forget)

Kernel isolation gets all the attention, but for RL the network is usually the more interesting boundary, because network access is where a policy finds shortcuts rather than crashes. On a Firecracker-based setup each sandbox gets its own network namespace and tap device rather than sharing a bridge — on PandaStack that's a pool of 16,384 pre-allocated /30 subnets per agent host, which is why per-worker networking is the default rather than a premium feature. That structure lets you make per-worker decisions instead of fleet-wide ones.

What you actually want to enforce, roughly in order of how often it matters:

• No worker-to-worker reachability. Rollout 12 should not be able to see rollout 13's HTTP server, its Redis, or its port scan. Separate namespaces make this the default instead of a firewall rule you forgot on one host.
• Default-deny egress, with an explicit allowlist. A task that needs PyPI gets PyPI, through a pinned mirror. A task that needs nothing gets nothing. Otherwise "solve the bug" becomes "search GitHub for the upstream patch", which is a real strategy that a real policy will really find.
• No shared mutable caches across trust boundaries. A shared package cache or shared results store that any worker can write is a cross-episode communication channel and a poisoning vector. Mount caches read-only, or give each worker its own copy-on-write view.
• No access to your control plane. The training orchestrator's API, the metrics endpoint, the object store holding your checkpoints — none of it should be routable from inside a rollout. A policy that can write to the reward store has solved RL in a way you will not enjoy explaining.
• Egress logging even when you allow egress. If you cannot tell after the fact which rollout talked to what, you cannot investigate the run where the reward curve suddenly went vertical.

Throughput and density math

The question I get is always "fine, but does it keep up?" Do the arithmetic rather than trusting a vibe. At a p50 create of 179ms, a single-threaded driver does about 5.6 resets per second — not enough. But resets are I/O-bound and independent, so they parallelize almost perfectly: 32 concurrent reset workers put you in the neighbourhood of 180 resets/second, and 64 puts you around 360. For most agent-training loops, where a single episode involves an LLM forward pass and a handful of multi-second tool calls, the environment reset stops being anywhere near the bottleneck long before that. Your GPU is the bottleneck. It was always going to be the GPU.

For density, the binding constraint is memory, not the VM abstraction and not networking — with 16,384 pre-allocated subnets per host, you will run out of RAM first, by a wide margin. Copy-on-write helps materially here: workers restored from the same baked snapshot share identical memory pages until they write, so a hundred rollouts of the same environment are not a hundred independent copies of your dependency tree. The honest caveat is that this sharing degrades as episodes progress and each worker dirties more pages, so size for the steady state of a long episode, not for the first second after restore.

Here is the comparison that actually decides the architecture, framed as isolation-per-rollout-worker. I'm describing the container case qualitatively and generically; specific runtimes differ a lot, so verify the details against your runtime's own docs rather than taking my word for it.

• Isolation boundary — Container per rollout: a shared host kernel with namespaces, cgroups, and seccomp; strong in practice, but kernel bugs are a shared blast radius across every worker on the box. MicroVM per rollout: a separate guest kernel behind hardware virtualization, so an escape has to beat the VMM, not just a syscall filter.
• Reset semantics — Container per rollout: teardown and recreate; clean if the image is immutable, but anything the policy mutated in a shared mount or shared kernel state can survive. MicroVM per rollout: restore a frozen machine image; residue is impossible because the previous machine no longer exists.
• Reset cost — Container per rollout: fast to start, but you re-pay service warmup and any lazy loading your env does on first use. MicroVM per rollout: restoring a pre-warmed snapshot skips warmup entirely — PandaStack creates at p50 179ms / p99 ~203ms with the restore step itself around 49ms.
• Mid-episode branching — Container per rollout: no native primitive; you approximate it by replaying the action prefix, which is slow and only correct if the env is perfectly deterministic. MicroVM per rollout: fork the running machine — 400–750ms same-host on PandaStack, 1.2–3.5s cross-host — and let the copies diverge.
• Network isolation — Container per rollout: usually a shared bridge by default, with per-worker policy as something you configure and can misconfigure. MicroVM per rollout: a dedicated network namespace and tap per worker as the structural default (16,384 pre-allocated /30 subnets per agent host on PandaStack).
• Kernel-level tasks — Container per rollout: the guest sees the host kernel, so tasks involving modules, sysctls, mounts, or /proc tampering are either blocked or dangerous. MicroVM per rollout: the policy gets its own kernel and can wreck it freely, which is exactly what you want for systems-flavoured task environments.
• Operational weight — Container per rollout: everyone already knows how to run these; your existing tooling works today. MicroVM per rollout: more machinery underneath (kernel image, rootfs, snapshot lifecycle, per-VM networking), which is either your problem or your vendor's — be honest about which.

Wiring it into a training loop

Concretely, here's the gym-shaped wrapper. The important detail is that `reset()` does not clean the environment — it deletes the machine and restores a new one. That one decision removes an entire category of contamination bug, and it costs you a couple hundred milliseconds.

import json
from pandastack import Sandbox


class VmEnv:
    """One rollout worker == one microVM. reset() throws the machine away."""

    TEMPLATE = "code-interpreter"

    def __init__(self, worker_id: int):
        self.worker_id = worker_id
        self.sbx = None
        self.steps = 0

    def reset(self, seed: int):
        # We do not "clean up" the previous episode. We delete the machine.
        # A policy that corrupted /usr, filled the disk, left a daemon on
        # :8080, or edited the grader cannot follow us across this line.
        if self.sbx is not None:
            self.sbx.kill()

        self.sbx = Sandbox.create(
            template=self.TEMPLATE,
            ttl_seconds=900,          # backstop: a leaked worker reaps itself
            metadata={
                "role": "rl-rollout",
                "worker": str(self.worker_id),
                "seed": str(seed),
            },
        )
        self.steps = 0
        self.sbx.filesystem.write("/work/seed", str(seed))
        self.sbx.exec(f"env-init --seed {seed}", timeout_seconds=60)
        return self.observe()

    def observe(self) -> str:
        return self.sbx.exec("env-observe", timeout_seconds=15).stdout

    def step(self, action: str) -> dict:
        # The action IS a shell command. That is the whole threat model,
        # and it is why this runs behind a hypervisor and not in a thread.
        self.steps += 1
        r = self.sbx.exec(action, timeout_seconds=60)
        return {
            "obs": r.stdout[-8000:],      # truncate: policies love cat'ing logs
            "stderr": r.stderr[-2000:],
            "exit_code": r.exit_code,
            "latency_ms": r.duration_ms,
        }

    def reward(self) -> float:
        # Grader lives at a root-owned, mode-0500 path baked into the image.
        g = self.sbx.exec("/opt/grader/grade.py --json", timeout_seconds=120)
        if g.exit_code != 0:
            return 0.0
        return float(json.loads(g.stdout)["score"])

    def close(self):
        if self.sbx is not None:
            self.sbx.kill()
            self.sbx = None

Now the branching version, which is where the fork primitive earns its place. Build the task environment once, freeze it, then fork it N ways so every candidate rollout starts from a byte-identical world instead of from a replayed approximation of one. Note the `sync` before snapshotting — flush the env's state to disk rather than assuming a live process's memory rides along untouched.

import json
from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox


def build_task_base(task) -> Sandbox:
    """Pay the expensive setup ONCE per task, then freeze it."""
    base = Sandbox.create(
        template="code-interpreter",
        persistent=True,
        metadata={"role": "rl-task-base", "task": task.id},
    )
    base.filesystem.write("/work/task.json", json.dumps(task.spec))
    setup = base.exec("bash /opt/task/setup.sh", timeout_seconds=900)
    if setup.exit_code != 0:
        base.kill()
        raise RuntimeError(f"task {task.id} setup failed: {setup.stderr}")

    base.exec("sync", timeout_seconds=30)   # flush state before we branch it
    base.snapshot()                          # durable restore point for later
    return base


def rollout(base: Sandbox, policy, task, branch: int, horizon: int = 24):
    """One candidate trajectory, forked from the shared mid-task state."""
    env = base.fork()          # 400-750ms same-host; pages shared copy-on-write
    trajectory = []
    try:
        obs = env.exec("env-observe", timeout_seconds=15).stdout
        for t in range(horizon):
            action = policy.act(obs, temperature=0.8, seed=10_000 * branch + t)
            r = env.exec(action, timeout_seconds=60)
            obs = r.stdout[-8000:]
            trajectory.append(
                {"t": t, "action": action, "exit_code": r.exit_code, "obs": obs}
            )
            if r.exit_code == 0 and "TASK_DONE" in obs:
                break

        g = env.exec("/opt/grader/grade.py --json", timeout_seconds=120)
        score = float(json.loads(g.stdout)["score"]) if g.exit_code == 0 else 0.0

        # Cheap reward-hacking tripwire: did the policy touch the grader,
        # the task spec, or the git history instead of solving the task?
        audit = env.exec(
            "sha256sum /opt/grader/grade.py /work/task.json", timeout_seconds=15
        ).stdout
        if audit.split() != task.expected_digests:
            score, trajectory = 0.0, trajectory + [{"flag": "tampered"}]

        return {"branch": branch, "score": score, "trajectory": trajectory}
    finally:
        env.kill()             # the branch dies; the frozen base survives


def best_of_n(policy, task, n: int = 8):
    base = build_task_base(task)
    try:
        with ThreadPoolExecutor(max_workers=n) as pool:
            results = list(
                pool.map(lambda i: rollout(base, policy, task, i), range(n))
            )
        return sorted(results, key=lambda r: r["score"], reverse=True)
    finally:
        base.hibernate()       # keep the warm base for the next batch

Two details worth stealing regardless of what you run this on. First, the tripwire: hash the grader and the task spec at the end of every episode and zero out any trajectory where they changed. It costs one exec and it catches the single most common reward hack. Second, `hibernate()` on the base rather than `kill()` — the expensive part is the environment setup, and keeping the frozen machine around means the next batch of forks starts warm instead of re-running your 900-second setup script.

When this is overkill

Let me be straight, because the honest version is more useful than the sales version. If your environment is a pure-function simulator — CartPole, a board game, a physics sim, anything where the "agent" emits a float vector and the env is code you wrote — do not put it in a VM. Run it in-process, vectorized, thousands of copies per core. The policy cannot break out of a numpy array. Adding a hypervisor there buys you nothing and costs you throughput and complexity, and I would talk you out of it.

The line is crossed when the action space includes executing code you did not write. Shell commands, arbitrary Python, package installs, tool calls that hit real services, browser control, anything terminal-bench-shaped. At that point the policy is not choosing among your enumerated actions, it is choosing among everything a Linux machine can do, and your isolation boundary needs to be a machine boundary. Same for kernel-adjacent task environments, multi-tenant training platforms where different teams' policies share hosts, and any run where an anomalous reward spike needs to be explainable rather than merely celebrated.

The reason this argument is even available now is timing: hardware isolation used to cost you a boot, and a boot used to cost you seconds. Snapshot-restore moved that to a couple hundred milliseconds and made forking a running machine a routine operation. So "one VM per rollout worker" stopped being a luxury architecture and became roughly the price of a careful `reset()`. Given that the thing on the other side of the boundary is being actively optimized to find whatever you left open, that seems like a reasonable price.