Fan Out Monte Carlo Trials by Forking a Warm MicroVM

Monte Carlo and parameter-sweep workloads have a shape you'll recognize the moment you've run one at scale: an expensive setup followed by a cheap, embarrassingly-parallel inner loop. You load a big pricing model, warm a market dataset into memory, seed a physics state, or spin up a game engine to its start-of-turn position — and then you want to run thousands of independent trials from that exact same warm starting point. Different random seeds. A grid of parameters. A fan of scenario branches. The setup is the same every time; only the trial differs.

The naive implementation cold-boots a fresh VM per trial and re-runs the setup inside each one. That's the wasteful part. If warming the state takes ten seconds and you want ten thousand trials, you've just signed up for a day of pure setup you didn't need to pay. I'm Ajay; I built PandaStack. This post is the pattern that kills that waste: warm the state once, snapshot it, then fork a microVM per trial so every fork inherits the warm state via copy-on-write and diverges on its own. Plus the one footgun that turns ten thousand "random" trials into ten thousand copies of the same trial.

The expensive setup, run ten thousand times

Say each trial needs a warm base: parse the model, build the in-memory structures, hydrate a dataset, initialize the simulation state. Call that setup cost S — often seconds, sometimes gigabytes of RAM. Each trial itself is fast: draw some random numbers, run the simulation forward, record a result. Call that T, and assume T is much smaller than S.

Cold-boot-per-trial pays S + T for every single trial. Across N trials that's N × (S + T), and since S dominates, you're paying to rebuild the same warm state N times. It's not just slow — it's the same deterministic work, repeated, with nothing learned between repetitions. The setup on trial 9,999 produces byte-for-byte the state that trial 1 already built. You are, quite literally, paying a computer to have the same idea ten thousand times.

The insight is that the warm state is identical across trials by construction — that's the whole point of a controlled experiment. Only the trial's inputs (the seed, the parameter) vary. So the warm state should be built once and shared, and only the divergence should cost anything. That's exactly what snapshot-plus-fork gives you.

Snapshot once, fork per trial

Boot one VM. Run your expensive setup inside it: load the model, warm the dataset, initialize the simulation to its common starting state. Now snapshot it. You have a frozen image with everything already paged into memory and ready to run — the equivalent of a save-game right at the branch point of your experiment.

Every trial is then a fork of that snapshot, not a fresh boot. A same-host fork lands in 400–750ms and — this is the part that makes it economical — shares the parent's memory copy-on-write. The multi-gigabyte warm state is mapped into each fork via MAP_PRIVATE, not copied. Pages only get duplicated when a fork actually writes to them, and each fork diverges independently: nothing one trial writes is visible to another, because the moment a page is touched the kernel hands the writer its own private copy. The rootfs is cloned with a reflink, so disk is copy-on-write too.

So a thousand forks of a warm 2 GiB base don't cost a thousand copies of 2 GiB. They cost roughly one copy of the warm state plus whatever each trial actually dirties — which, for a trial that draws a seed and runs a simulation forward, is usually small. That's the density argument for fork-per-trial over cold-boot-per-trial, and it falls straight out of the copy-on-write memory model.

from pandastack import Sandbox
import json, concurrent.futures as cf

# --- Phase 1: build the warm base ONCE ---
# Boot one VM, run the expensive setup, snapshot the warm state.
base = Sandbox.create(template="code-interpreter", ttl_seconds=3600)

setup = """
import pickle, numpy as np
# Expensive setup: load a model, warm a dataset, build sim state.
# (Stand-in: a big calibrated parameter block we don't want to rebuild per trial.)
model = {'drift': 0.07, 'vol': 0.2, 'horizon_days': 252, 'paths': 5000}
with open('/workspace/model.pkl', 'wb') as f:
    pickle.dump(model, f)
print('warmed model')
"""
base.filesystem.write("/workspace/setup.py", setup)
print(base.exec("python3 /workspace/setup.py", timeout_seconds=300).stdout)

# Freeze the warm state. Every fork inherits it in memory, copy-on-write.
snap = base.snapshot()

# --- Phase 2: fork ONE VM per trial, each with a DISTINCT seed ---
def trial(seed: int) -> dict:
    sbx = snap.fork()  # ~400-750ms same-host; warm state shared CoW, no copy
    try:
        # The seed is the ONLY thing that differs between trials.
        # Reseed inside the fork -- see the frozen-RNG caveat below.
        run = f"""
import pickle, json, numpy as np
model = pickle.load(open('/workspace/model.pkl', 'rb'))
rng = np.random.default_rng({seed})            # distinct seed per fork
dt = 1.0 / model['horizon_days']
z = rng.standard_normal((model['paths'], model['horizon_days']))
logret = (model['drift'] - 0.5*model['vol']**2)*dt + model['vol']*(dt**0.5)*z
terminal = np.exp(logret.sum(axis=1))          # terminal price multiples
result = {{'seed': {seed}, 'mean': float(terminal.mean()),
          'p95': float(np.percentile(terminal, 95))}}
json.dump(result, open('/workspace/out.json', 'w'))
"""
        sbx.filesystem.write("/workspace/run.py", run)
        r = sbx.exec("python3 /workspace/run.py", timeout_seconds=120)
        if r.exit_code != 0:
            return {"seed": seed, "error": r.stderr[-2000:]}
        return json.loads(sbx.filesystem.read("/workspace/out.json"))
    finally:
        sbx.kill()  # the fork and its diverged pages evaporate

# Fan out: distinct seed per trial, forks run in parallel.
seeds = range(1000, 1000 + 64)
with cf.ThreadPoolExecutor(max_workers=16) as pool:
    results = list(pool.map(trial, seeds))

means = [r["mean"] for r in results if "mean" in r]
print("trials:", len(means), "grand mean:", sum(means) / len(means))
base.kill()

Results come back as structured JSON through the filesystem API rather than scraped from stdout, so a trial that prints diagnostic noise can't corrupt your aggregation. Each fork is killed the moment its result is read; the pages it diverged are freed, and the warm state it saw was a copy-on-write view it could never write back to the parent or to the next trial.

The frozen-RNG footgun: reseed, or run the same trial 10,000 times

Here is the mistake that will get you, and it's a good one, because the failure is silent. A snapshot freezes the entire machine state — including the state of every random number generator that was already initialized when you took the snapshot. If you seed your RNG during setup and then snapshot, every fork inherits that exact RNG state. Every fork's first `random()` call returns the same number. Every trial draws the same "random" sequence. You fan out ten thousand forks, run them all, aggregate the results — and your Monte Carlo has the variance of a single sample, computed ten thousand times, very fast, in parallel, on real hypervisors. It looks like it worked. The histogram is a single spike wearing a mustache.

The fix is simple and non-negotiable: reseed the RNG inside each fork, after the fork, with a value distinct to that trial. Do not seed before the snapshot and assume the forks will diverge — they inherit frozen entropy and will not. Derive each fork's seed from something unique to the trial: the trial index, the sandbox ID, or a counter you pass in. The code above passes an explicit distinct seed into every fork for exactly this reason.

Two related traps live next door. First, if you want reproducibility, derive seeds deterministically (seed = base_seed + trial_index) so a rerun produces the same trials — random-per-fork and reproducible-per-fork are both fine, but pick one on purpose. Second, watch for RNGs you didn't know you had: a library that lazily seeds from the clock on first use will read the frozen snapshot clock and hand every fork the same seed. When in doubt, reseed explicitly rather than trusting anything to reseed itself.

Fanning out, collecting results, and tearing down

Fan-out width is bounded by host memory and CPU, not by the sandbox plumbing — a single agent pre-allocates 16,384 /30 subnets, so network namespaces are never the bottleneck. Run as many forks concurrently as your CPU and RAM budget allows, and queue the rest. Because the forks share the warm state copy-on-write, the memory ceiling is roughly the warm footprint plus the sum of what the in-flight trials have dirtied, not the number of forks times the whole warm state.

Collect each trial's result through the filesystem API as structured data and aggregate on the outside — the fork is a worker that produces one record, not a place you keep state. Kill each fork as soon as you've read its result so its diverged pages are freed promptly; an untrusted or runaway trial should also carry a per-exec timeout and a create TTL so a wedged simulation can't leak a VM. When the whole sweep is done, kill the parent too — though you can keep it alive to fork the next batch from the same warm state without rebuilding it.

• Cold-boot per trial — Pays the setup cost S on every trial: N × (S + T). Each VM re-parses the model and re-warms the dataset from scratch. Memory is N full copies of the warm state. Trivially isolated, and trivially wasteful — the same deterministic setup rebuilt N times.
• Fork per trial (PandaStack) — Pays S once to build the warm snapshot, then ~400–750ms per same-host fork. Every fork inherits the warm state via copy-on-write memory, so RAM is roughly one warm copy plus per-trial divergence, not N copies. Each trial diverges independently and is torn down cheaply. The catch: you MUST reseed per fork or every trial is identical.

For competing sandbox and microVM platforms the snapshot-and-fork primitive exists in various forms, but the copy-on-write memory sharing across a fan-out, the reseed-after-fork discipline, and the exact latency numbers vary by vendor — verify those against their docs before you commit.

Honest limits

• Same-host forks (400–750ms) share the parent's resident memory copy-on-write; keep a sweep on the parent's agent for this fast path. Cross-host forks are 1.2–3.5s because the memory image has to move over the network first.
• The first spawn of a brand-new template is a cold boot (~3s) that bakes the snapshot; every create after that restores it in ~49ms and p50 179ms (p99 ~203ms). Bake your warm base once, not per trial.
• Copy-on-write is a density win on the fork, not over the lifetime. A trial that rewrites most of its RAM eventually copies most of those pages — the savings are largest for short, lightly-diverging trials, which is what most Monte Carlo trials are.
• The fork clones the machine, not the outside world. If a trial writes to a shared external database or calls a paid API, every fork does it. Copy-on-write protects in-VM memory and disk, not third-party systems.
• Snapshot freezes the clock along with everything else. Trials that read wall-clock time all see the bake-time clock until they diverge — reseed clock-derived inputs the same way you reseed RNGs, and don't rely on the snapshot clock being current.

The shape of the solution: one VM builds the expensive warm state and gets snapshotted; every trial is a fork of that snapshot with the state already warm in memory; each fork reseeds its RNG to a trial-distinct value, runs, and returns a structured result; and teardown is cheap because copy-on-write meant a short trial never allocated much in the first place. You pay the setup cost once instead of N times, fan out as wide as your host memory allows, and get thousands of clean, identical-start, independently-diverging trials — as long as you remember to reseed. PandaStack's core is open source under Apache-2.0, so you can run the agent on your own KVM hosts, fork a warm VM, and watch the trials diverge page by page yourself.