Isolating Smart Contract Simulation and Forked-Chain Testing in MicroVMs

The workload looks harmless on a whiteboard. You spin up a local node forked from mainnet at a specific block, replay a bundle of transactions against it, and read out the state diff. Maybe you're checking whether a proposed governance action does what its author claims. Maybe you're fuzzing an invariant on a contract a user just pasted into your app. Maybe you're a security review pipeline that dry-runs every contract someone submits. In all three cases the shape is the same: a short-lived EVM with real mainnet state, a pile of code you did not write, and an answer you need to be able to trust.

I'm Ajay; I built PandaStack. This post is about why forked-chain simulation is one of the most under-isolated workloads I see in the wild, and about a specific architecture for it: one ephemeral Firecracker microVM per simulation, pinned to an exact block height and node version, with exactly one permitted upstream and no credentials anywhere inside the guest. Along the way: why the test scripts are scarier than the bytecode, how a snapshot taken after the fork has warmed makes parallel scenario exploration cheap, and why a mismatched result should be a finding rather than a shrug.

What the workload actually is

The tooling here is mature and the pattern is well established. Foundry's local node (anvil), Hardhat's built-in network, and similar dev nodes all support forking: you point the node at an archive RPC endpoint, optionally pin a block number, and it lazily fetches account state, storage slots, and code from upstream as your transactions touch them. You get a private, writable copy of mainnet at a moment in time. Flags, defaults, and caching behaviour differ between tools and move between releases, so treat everything I say about them as directional and verify against the current docs for whichever node you standardize on.

What people build on top of that primitive falls into a few buckets, and they escalate in how much untrusted material is involved.

• Bundle simulation — replay a sequence of transactions against a forked head and report the state diff, gas, and revert reasons. The transactions may be adversarial, but the harness is yours.
• Invariant and property fuzzing — run thousands of randomized call sequences against a contract and assert a property never breaks. The contract may be arbitrary, and the fuzz loop is unbounded by nature.
• Dry-running user-submitted contracts — a review tool, an audit pipeline, or a product feature where anyone can paste a repo and get a report. Now you're running someone else's build scripts, someone else's dependencies, and someone else's test suite.
• Upgrade and migration rehearsal — fork production state, run the upgrade, then run the existing test suite against the upgraded system to see what breaks. Sensitive because it touches the real deployment story.
• Regression corpora — replay a library of historical scenarios on every commit, which means the same simulation runs constantly and any nondeterminism turns into a flaky test that everyone learns to re-run.

The default place all of this ends up is a shared CI runner or a long-lived simulation box. It works right up until the moment the thing you're simulating is hostile, at which point the boundary you were relying on turns out to be a folder name.

Untrusted twice over: the bytecode and the scripts around it

The EVM is genuinely a good sandbox for what it covers. Bytecode can't open a socket, can't read a file, and can't call `exec`. Its worst behaviour is confined to chain state and gas. If contract bytecode were the only untrusted input, you could run everything in one long-lived node process and sleep fine.

But nobody simulates bare bytecode. A real submission is a repository. It has a package manifest, a lockfile, a build step that compiles Solidity, deployment scripts, fixtures, mocks, and a test suite. Those are plain programs running on the host with the host's privileges. The compiler plugin runs on the host. The deploy script runs on the host. The test helper that "just impersonates an account" runs on the host. And the dependency install step runs arbitrary lifecycle hooks on the host, which is the shortest known path from `npm install` to an incident report.

The EVM sandbox contains the contract. Nothing contains the `postinstall` hook except the machine you handed it.

• Install-time code execution — package lifecycle hooks run before a single line of your test suite does. `--ignore-scripts` helps and is worth doing, but plenty of legitimate toolchains break without hooks, so you end up re-enabling them for exactly the repos you understand least.
• Build plugins — Solidity toolchains support plugins and custom compilation pipelines. A plugin is a program you're executing because a config file told you to.
• Test helpers with host reach — test suites shell out constantly: to a compiler, to a formatter, to a coverage tool, to `curl`. Any of those call sites is a place where a helpful-looking script does something else.
• Fixture downloads — "fetch the reference deployment artifacts before running" is a normal thing for a test suite to do, and it is also a normal thing for a payload to do.
• The report itself — if your pipeline renders the simulation's output into a report, the untrusted code controls that output. Treat it as data to be escaped, not markup to be trusted.

None of this is exotic supply-chain wizardry. It's the ordinary behaviour of ordinary developer tooling, pointed at a repository you have no reason to trust. The mitigation isn't to audit every hook; it's to make the host that runs them disposable and empty.

The RPC key problem, or: the funniest way to lose money

Here's the part that makes this workload specifically dangerous rather than generically dangerous. A forked node needs an archive RPC endpoint, and archive endpoints are expensive and authenticated. So the URL — with the API key in it — ends up in the environment of the process running the simulation. Meanwhile, deployment rehearsal is a first-class use case, so a lot of these same pipelines also carry a deployer private key, usually as `PRIVATE_KEY`, because that is the variable name every tutorial uses.

So the untrusted test script, running as a normal host process, has `process.env` in scope. Not a clever exploit. Not a sandbox escape. Just `os.environ`, in a language with a networking library, on a machine with outbound internet. The API key is a billing problem: someone else now runs their indexer on your archive plan and you find out via an invoice. The private key is not a billing problem.

The structural fix has two halves, and both are boring in the good way. First, the guest gets no credentials — none, not scoped ones, not read-only ones. Second, the guest's upstream RPC is a broker on the host side of the boundary that holds the key, attaches it, forwards the request, and returns the response. The guest sees an unauthenticated local URL. Exfiltrating it gets you an address that only exists inside a network namespace that no longer exists.

That broker is also the right place to enforce everything else you care about: only JSON-RPC read methods, only the chain you meant, a request budget so a pathological fuzz run can't spend your monthly quota in an afternoon, and a log of exactly which upstream calls a simulation made — which turns out to be genuinely useful for debugging cache-miss slowness later.

from pandastack import Sandbox
import json

def simulate(bundle_b64: str, pin: dict) -> dict:
    """Run ONE forked-chain scenario in a machine that contains nothing else.

    Note what is NOT in this guest: your archive-node API key, any deployer
    private key, and anyone else's simulation. The fork's upstream is a
    host-side broker that holds the credential; the guest sees a bare local
    URL and could exfiltrate it to no effect whatsoever.
    """
    sbx = Sandbox.create(
        template="code-interpreter",
        ttl_seconds=900,          # a fuzz loop does not get to run until Tuesday
        metadata={
            "kind": "evm-sim",
            "chain": pin["chain"],
            "block": str(pin["block_number"]),
            "node_version": pin["node_version"],
        },
    )
    try:
        sbx.filesystem.write("/work/bundle.tar.b64", bundle_b64)
        sbx.filesystem.write("/work/pin.json", json.dumps(pin, sort_keys=True))

        # Unpack, then install WITHOUT lifecycle hooks. This is not a security
        # boundary -- it's hygiene. The boundary is the VM around all of it.
        sbx.exec(
            "cd /work && base64 -d bundle.tar.b64 > b.tar && tar xf b.tar && "
            "npm ci --ignore-scripts --no-audit --no-fund",
            timeout_seconds=300,
        )

        # Start the forked node against the broker gateway. Pin the block so the
        # result is a statement about a specific state, not about 'recently'.
        # (Fork flag names differ across anvil/hardhat and across releases --
        #  check your node's docs; this is the shape, not the gospel.)
        sbx.exec(
            "cd /work && setsid sh -c '"
            "anvil --fork-url http://10.200.0.1:8545"
            f" --fork-block-number {pin['block_number']}"
            " --host 127.0.0.1 --port 8545"
            "' > /work/node.log 2>&1 & sleep 4; tail -3 /work/node.log",
            timeout_seconds=60,
        )

        # The untrusted scenario. It talks only to 127.0.0.1:8545.
        run = sbx.exec(
            "cd /work && node scripts/run-scenario.js --out /work/result.json",
            timeout_seconds=600,
        )
        if run.exit_code != 0:
            return {"status": "failed", "stderr": run.stderr[-4000:]}

        # Structured result through the filesystem API -- never scraped from
        # stdout, which the untrusted code controls completely.
        return {"status": "ok",
                "pin": pin,
                "result": json.loads(sbx.filesystem.read("/work/result.json"))}
    finally:
        sbx.kill()   # node, node_modules, chain cache, and any surprises: gone

Egress: exactly one upstream and nothing else

A forked-chain simulation has an unusually crisp network requirement, and you should exploit that. It needs to reach one RPC endpoint. That's the entire list. It does not need package registries at run time (install already happened, and you can pre-bake dependencies into the template), it does not need to resolve arbitrary DNS, and it certainly does not need to POST anywhere.

That makes deny-by-default egress unusually easy to actually ship here, as opposed to the usual situation where the allowlist grows until it's a denylist wearing a hat. On PandaStack each sandbox gets its own network namespace, so the policy lives at the boundary rather than inside code you don't trust.

# A simulation needs ONE upstream: the RPC broker on the host side.
# Everything else -- package registries, DNS, that helpful telemetry endpoint
# a dependency added last week -- is dropped at the namespace boundary.
# (Run on the agent host; each sandbox has its own ns-<id> namespace.)
NS="ns-<sandbox-id>"
BROKER="10.200.0.1"      # host-side gateway; holds the archive API key

ip netns exec "$NS" iptables -P OUTPUT DROP
ip netns exec "$NS" iptables -A OUTPUT -o lo -j ACCEPT
ip netns exec "$NS" iptables -A OUTPUT -d "$BROKER" -p tcp --dport 8545 -j ACCEPT
ip netns exec "$NS" iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT

# Result: the guest can fetch mainnet state through the broker and do nothing
# else. A test script that decides to phone home gets a connection timeout,
# which -- pleasingly -- is indistinguishable from your CI just being slow.

# Verify from inside the guest before you trust it:
#   curl -s -m 5 -X POST http://10.200.0.1:8545 \
#     -d '{"jsonrpc":"2.0","id":1,"method":"eth_blockNumber","params":[]}'  # works
#   curl -s -m 5 https://example.com                                          # hangs, then dies

Determinism: a simulation result is a claim about a specific machine

Simulation output only means something if you can reproduce it. "This transaction bundle nets out fine" is not a fact — it's a fact about a chain state, an EVM spec version, and a node implementation. Change any of those and the same bundle can revert, or succeed for a different reason, or consume noticeably different gas. If you can't say which machine produced a result, you can't defend the result.

The failure mode is subtle and demoralizing rather than dramatic. Someone reports that a scenario used to pass. You re-run it. It passes. You shrug and close the ticket, and three weeks later it fails again, and by now the node has been upgraded twice and the fork block has drifted forward by a hundred thousand blocks because nobody pinned it. There is no way to tell whether you found a real regression or just moved.

• Pin the block height explicitly. Forking "latest" means your regression suite is a function of what happened on chain this morning. Pin it, record it, and bump it deliberately as its own reviewable change.
• Pin the chain spec. Which hardfork/EVM version the node emulates changes opcode availability, gas schedules, and occasionally semantics. It must be an input to the run, not a property of whichever node binary got installed.
• Pin the node client and version. Different dev nodes disagree at the margins, and a single node's behaviour changes across releases. A snapshot-backed template makes this an artifact rather than an aspiration.
• Pin the compiler. Solidity version, optimizer settings, and optimizer runs all change deployed bytecode, which changes gas and can change behaviour. Bytecode identity belongs in the run record.
• Eliminate ambient nondeterminism. Seed the fuzzer explicitly and record the seed. Don't read the wall clock — pass block timestamps in. Watch for anything that varies with parallelism, which reorders under load and produces a flake nobody can reproduce locally.
• Hash the output. Compute a digest over the canonicalized state diff and store it with the run. A re-run producing a different digest under an identical pin is a real finding, and it should page someone rather than getting re-run until green.

The environment half of that list is where snapshots earn their keep. A PandaStack template's first spawn is a genuine cold boot — around 3 seconds — during which the node client, compiler, and toolchain are installed and warmed. That state is captured once, and every subsequent create restores it: 179ms p50, roughly 203ms p99, with about 49ms of that being the restore step itself. Every simulation therefore starts on a byte-identical machine. Same node binary, same compiler, same libraries, same locale. When a result changes, you know the environment didn't.

// Everything that can change the answer becomes part of the run record.
// If two runs disagree and their pins are identical, that is a FINDING.
// If the pins differ, that is just two different questions being asked.
type SimulationPin = {
  chain: "mainnet" | "sepolia" | string;
  chainId: number;
  blockNumber: number;        // never "latest" -- that's a moving target
  blockHash: string;          // guards against a reorg under your feet
  evmVersion: string;         // hardfork/spec the node emulates
  nodeClient: string;         // e.g. "anvil" | "hardhat"
  nodeVersion: string;        // exact release, not a range
  solcVersion: string;
  optimizer: { enabled: boolean; runs: number };
  fuzzSeed: string;           // recorded, not generated per-run
  snapshotGeneration: string; // the PandaStack template snapshot that ran it
};

type SimulationRun = {
  pin: SimulationPin;
  bundleSha: string;          // hash of the submitted repo/bundle
  deployedBytecodeSha: string;
  resultDigest: string;       // sha256 over the canonicalized state diff
  rpcCallCount: number;       // from the broker, not from the guest
  durationMs: number;
};

// Re-running a scenario means restoring the SAME snapshot generation and
// replaying the SAME pin, then comparing digests. Match -> explained.
// Mismatch -> a real behavioural change, surfaced as an alert instead of
// as a flaky test someone learns to click 'retry' on.
export function isRegression(a: SimulationRun, b: SimulationRun): boolean {
  return samePin(a.pin, b.pin) && a.resultDigest !== b.resultDigest;
}

Snapshot after the fork is warm, then fan out

Now the performance argument, which is also a determinism argument in disguise. A forked node is slow to become useful because forking is lazy: state is fetched from upstream the first time a transaction touches it. Your first run against a complex protocol makes hundreds of upstream calls to pull in code, storage slots, and balances. Subsequent runs against the same fork hit a warm cache and fly.

So if you naively boot a fresh node per scenario, every single scenario pays that cold-cache tax again, against an RPC provider that charges you for the privilege and rate-limits you for the fun of it. Exploring 200 scenarios means 200 cold caches. This is the part of the workload that quietly becomes your biggest line item.

The fix: warm the fork once, in one trusted VM, by replaying a representative transaction set so the node pulls in the state your scenarios actually touch. Then snapshot that VM — with the node process running and its cache resident in memory. Every scenario is now a fork of that snapshot. Same-host forks land in 400–750ms and share the parent's memory copy-on-write, so the warm cache is mapped rather than copied; pages only diverge where a scenario actually writes. Cross-host forks are 1.2–3.5s because the memory image has to travel, so keep a fan-out on the parent's agent when you can.

The determinism payoff is the good part: every scenario starts from bit-identical chain state, not merely equivalent chain state. Scenario 7 cannot be affected by scenario 6, because scenario 6 was a sibling fork whose writes went to its own copy-on-write pages and then evaporated.

import concurrent.futures as cf, json
from pandastack import Sandbox

# --- Phase 1: warm the fork ONCE, in a trusted VM, then freeze it ---
base = Sandbox.create(template="code-interpreter", persistent=True)
base.filesystem.write("/work/warm.js", WARMUP_SCRIPT)   # replays a representative set
base.exec("cd /work && node warm.js", timeout_seconds=600)

# Snapshot AFTER the node's state cache is populated. This is the whole trick:
# the expensive part (hundreds of archive RPC calls) is now baked into memory.
warm = base.snapshot()

# --- Phase 2: one fork per scenario, all starting from identical warm state ---
def explore(scenario: dict) -> dict:
    sbx = warm.fork()          # 400-750ms same-host; cache shared copy-on-write
    try:
        sbx.filesystem.write("/work/scenario.json", json.dumps(scenario, sort_keys=True))
        r = sbx.exec(
            "cd /work && node scripts/run-scenario.js "
            "--scenario scenario.json --out /work/result.json",
            timeout_seconds=300,
        )
        if r.exit_code != 0:
            return {"id": scenario["id"], "status": "failed", "stderr": r.stderr[-2000:]}
        return {"id": scenario["id"], "status": "ok",
                "result": json.loads(sbx.filesystem.read("/work/result.json"))}
    finally:
        sbx.kill()             # this scenario's chain writes never existed

# Fan out. Each agent pre-allocates 16,384 /30 subnets, so the ceiling here is
# host CPU and RAM -- not sandbox networking, and not your RPC rate limit,
# because the state these forks read was fetched exactly once.
with cf.ThreadPoolExecutor(max_workers=16) as pool:
    for out in pool.map(explore, SCENARIOS):
        print(out["id"], out["status"])

Resource fences, because fuzzing has no natural end

An invariant fuzzer's job description is "keep going." Left alone, it will use every core you own, and it will do so while being completely correct about what it was asked to do. Add an accidental infinite loop in a test helper, or a contract whose gas metering doesn't bound the host-side work of simulating it, and you have a run that consumes a shared box until someone notices the dashboards.

On a shared CI runner this is a noisy-neighbour problem with an unpleasant twist: the kernel's OOM killer doesn't reap the process that caused the pressure, it reaps whichever one looks tastiest. One user's pathological fuzz campaign gets an unrelated team's build killed mid-write. Per-run microVMs bound this structurally, because the guest has a fixed vCPU and RAM allocation baked into its snapshot and a hard TTL on top.

• Set a TTL on every create, always. It is the backstop that works when your timeout logic is the thing that hung. A wedged simulation is reclaimed rather than lingering.
• Set a per-exec timeout too, tighter than the TTL, so you get a diagnosable failure rather than a vanished machine.
• Bound the fuzzer explicitly — iteration count or wall-clock budget, recorded in the run record. "How long did we fuzz" is part of what the result means.
• Budget the broker. A per-run cap on upstream RPC calls turns a runaway cache-miss loop into a clean failure instead of a bill.
• Cap concurrency per submitter, not just globally. One user should not be able to fill your fleet by submitting two hundred scenarios, and this is much easier to enforce when the unit of work is one VM.
• Treat resource exhaustion as a result. A submission that cannot be simulated within its budget is a report you can return, not an outage you absorb.

Shared CI runner vs container-per-run vs microVM-per-simulation

• Risk: credential exposure — Shared CI runner: the archive RPC key, and often a deployer private key, sit in the environment of the same process that runs a stranger's test suite; `os.environ` is the whole exploit. MicroVM per simulation: the guest holds no credentials at all, and its RPC upstream is an unauthenticated local address served by a host-side broker that attaches the key.
• Risk: install-time code execution — Shared CI runner: package lifecycle hooks and build plugins run on a long-lived machine with cached credentials, an SSH agent, and reachable neighbours. MicroVM per simulation: hooks run on a disposable guest with its own kernel, a separate install phase, and nothing worth stealing on disk.
• Risk: exfiltration — Shared CI runner: general outbound internet is typically required for the runner to function, so "deny by default" never actually ships. MicroVM per simulation: a simulation needs exactly one upstream, so a per-namespace default-DROP with a single allow rule is genuinely enforceable and easy to verify.
• Risk: cross-run contamination — Shared CI runner: caches, node_modules, and leftover chain state persist between jobs, so run N can be influenced by run N-1's writes. MicroVM per simulation: each run is a fork of the same warm snapshot, so chain state is bit-identical at start and every write dies with the VM.
• Risk: nondeterministic results — Shared CI runner: node client, compiler, and OS libraries drift with every image rebuild, so a changed result is ambiguous between regression and drift. MicroVM per simulation: the environment is a pinned snapshot generation recorded in the run record, so an identical pin producing a different digest is unambiguously a finding.
• Risk: resource exhaustion — Shared CI runner: an unbounded fuzz loop starves co-resident jobs, and the OOM killer picks its victim by memory footprint rather than by fault. MicroVM per simulation: fixed vCPU and RAM per guest plus a hard TTL, so a runaway run exhausts only itself and returns a clean failure.
• Cost of isolation — Shared CI runner: nothing extra per job, which is exactly why everyone starts here. MicroVM per simulation: on PandaStack a create is 179ms p50 (~203ms p99) because every create restores a baked snapshot rather than cold-booting (~3s, paid once at bake time), and a same-host fork of a warm chain state is 400–750ms. That is a rounding error next to compiling a Solidity project.

Other sandbox and microVM platforms give you a comparable per-VM boundary — a hypervisor boundary is a hypervisor boundary. Where offerings differ is the warm-state fork model, copy-on-write memory sharing across a fan-out, and the specific latencies; verify those against each vendor's own documentation before you build a cost model on them.

Honest limits

A few things this architecture does not do, stated plainly so you don't discover them in production.

• It doesn't make a simulation correct. A forked node approximates mainnet; it does not reproduce mempool dynamics, ordering, or anything that depends on other actors. A passing simulation is evidence, not proof.
• It doesn't protect a secret you deliberately put inside the guest. Isolation contains execution — if you inject a signing key into a sandbox that runs untrusted code, you have injected a signing key into untrusted code.
• Re-baking the warm base invalidates the old snapshot, so treat the warm fork as a versioned artifact. Bumping the pinned block is a deliberate, reviewable change that produces a new generation.
• Cross-host forks cost 1.2–3.5s versus 400–750ms same-host, because the memory image moves over the network. Schedule a fan-out on the parent's agent when latency matters.
• The broker is now a trusted component you own. It holds the credential, so it needs the same care as any other secret-bearing service — including method allowlisting and per-run budgets.

The summary

Forked-chain simulation gets treated as a testing workload when it's really an untrusted-code workload with a testing workload sitting on top. The EVM contains the bytecode. It does not contain the build scripts, the dependency hooks, or the test helpers, and those are ordinary programs with a filesystem, a network stack, and — on most setups I've seen — an archive RPC key and sometimes a deployer key sitting in the environment next to them.

One ephemeral microVM per simulation fixes the shape rather than patching the instances. The guest carries no credentials and reaches exactly one upstream through a host-side broker, so exfiltration returns a local address and a connection timeout. The environment is a pinned snapshot generation recorded alongside the block height, chain spec, node version, and compiler settings, so a differing result is a finding instead of a shrug. Warm the fork once and fork it per scenario, and the expensive cold-cache tax is paid a single time while every scenario still starts from bit-identical state. And fixed vCPU, fixed RAM, and a hard TTL mean an unbounded fuzz loop exhausts its own machine and nobody else's.

The same pattern shows up anywhere untrusted code meets valuable data — /blog/microvm-quant-backtesting-isolation covers the warm-dataset fork for backtesting, and /blog/microvm-per-tenant-billing-usage-metering covers determinism and fail-closed semantics when the output is money. Here the cost is about a fifth of a second per run. The alternative is explaining to someone how a `postinstall` hook came to know your private key.