Replaying and Debugging Webhooks in Disposable MicroVMs

You have a folder full of production webhook payloads — a few thousand real Stripe events, GitHub push deliveries, Shopify order hooks — captured off the wire because at some point a handler misbehaved and you were smart enough to keep the evidence. Now you want to replay them against a candidate build of your handler to reproduce the bug, or to prove a fix actually fixes it. The naive move is to point those payloads at a staging copy of the service and watch what happens. The problem with that move is that some of those payloads are not just data — they're triggers for code, and code has side effects. Replaying the wrong event against the wrong build can charge a card, fire a Slack notification to a real customer, or — in the genuinely bad case — run a handler whose new codepath does something you didn't intend on a box that can reach things you care about. This post is about replaying webhooks the safe way: one disposable microVM per payload, side effects trapped inside a VM that's about to be deleted, and a clean deterministic starting state every single time.

Why "just replay it in a prod-ish env" is dangerous

Replay is not a read-only operation. A webhook payload is an instruction to run your handler, and your handler was written to do things — write rows, call APIs, enqueue jobs, send email. Point a captured payload at an environment that can reach real systems and you've re-triggered every one of those side effects, retroactively, on data that's months stale. The failure modes stack up fast:

• Side effects hit real systems: a replayed `payment_intent.succeeded` re-runs your fulfilment code, a replayed GitHub push re-triggers a deploy, a replayed Shopify order emails a customer a receipt for something they bought in March. Staging that shares any credential with prod inherits prod's blast radius.
• Secrets are sitting right there: the environment you replay in has API keys, database URLs, and signing secrets so the handler can work. A handler build with a bug — or a handler you're debugging precisely because it started doing something weird — is one errant line away from reading those and putting them somewhere you can't take them back from.
• Poison payloads are real: not every captured payload is benign. Some were crafted by whoever was probing you when you started capturing. A payload that steers a handler into `rm -rf`, an infinite loop, a fork bomb, or a 40GB allocation takes down whatever host runs it — and 'whatever host' should never be a machine that matters.
• Non-determinism poisons the diff: replaying against a long-lived shared environment means run N sees the leftover state of runs 1 through N-1. A handler that appended a row, cached a value, or left a file changes the starting conditions for the next replay, so you can't tell whether a behavior difference came from your code change or from accumulated cruft.

The shape of the fix: one disposable microVM per payload

The move is to give every replayed payload its own throwaway Firecracker microVM. Bake a template that has your candidate handler build and its dependencies already installed, snapshot it once, then restore a fresh VM from that snapshot for each payload you replay. Deliver the payload into the guest as a file, exec the handler against it, capture stdout plus the exit code plus any side-effect artifacts the handler left on the guest filesystem, and then delete the VM. Nothing the handler did — no row it wrote, no file it touched, no process it forked, no charge it tried to make on a network it couldn't reach — survives the delete.

This buys you three things that a shared staging environment can't. First, containment: the handler runs behind a hardware virtualization boundary with its own guest kernel, so a poison payload that steers it into `rm -rf /` deletes a rootfs that was disposable anyway. Second, determinism: every replay starts from the exact same baked snapshot, so run 50 sees precisely what run 1 saw — no leftover state, no ordering effects, a clean slate you can actually diff against. Third, a real air gap for side effects: the guest gets its own network namespace, so you decide exactly what it can reach (usually: nothing), and a handler that tries to POST a receipt to Stripe's live API just gets a connection refused instead of a real charge.

The safest place to replay a payload that might delete production is a VM you were about to delete anyway.

Fork-per-payload: a clean, deterministic starting state

There are two ways to get a fresh VM per payload, and the difference matters when your handler needs state — a seeded database, a warmed cache, a fixture directory. The first way is a plain create: restore a fresh VM from the baked template snapshot for each payload. The second way is fork: stand up one VM, set up its state exactly how you want the replay to start (seed the DB, load the fixtures, warm whatever needs warming), snapshot it, and then fork that snapshot once per payload. Every fork is a copy-on-write clone of that identical prepared state — same rows, same cache, same everything — so your replays are deterministic down to the starting byte.

Fork is the better fit for replay specifically because the whole point is reproducibility. If payload A and payload B both start from a forked snapshot of the same seeded state, then any difference in how the handler behaves is attributable to the payloads (or to the build), not to drift in the environment. Copy-on-write means the fork doesn't duplicate gigabytes of seeded database — the children share the parent's memory and disk pages until they write, so you can fan out a lot of replays from one warm parent cheaply.

The replay loop in Python

Here's the core loop with the Python SDK: for each captured payload, restore a fresh microVM from the template that has your candidate handler baked in, write the payload into the guest as a file, exec the handler against it with a hard timeout, then read back the handler's output and any side-effect artifacts it left behind. Set PANDASTACK_API_KEY in your environment first. The `ttl_seconds` on create is a backstop so a handler that hangs kills its own VM; the `timeout_seconds` on exec is the circuit breaker for a payload that steers the handler into an infinite loop.

import json
from pandastack import Sandbox

def replay_one(payload: dict, build_template: str) -> dict:
    # Fresh, hardware-isolated microVM restored from the snapshot that has
    # THIS candidate handler build baked in (~179ms p50 create). The handler
    # runs behind its own guest kernel, so a poison payload can't reach the host.
    sbx = Sandbox.create(
        template=build_template,
        ttl_seconds=60,                    # backstop: VM self-destructs after 60s
        metadata={"purpose": "webhook-replay"},
    )
    try:
        # Deliver the captured payload into the guest as a plain file.
        sbx.filesystem.write("/replay/event.json", json.dumps(payload))

        # Exec the handler against it with a hard timeout. It reads the event,
        # writes its response to /replay/out.json and any side effects it
        # tried to make (that we allow) to /replay/effects/.
        result = sbx.exec(
            "python3 /app/handler.py < /replay/event.json > /replay/out.json",
            timeout_seconds=15,
        )

        # Capture everything we want to diff: exit code, stdout/stderr, the
        # response body, and the side-effect artifacts the handler emitted.
        # This is the ONLY thing that escapes the VM before we delete it.
        out = None
        if result.exit_code == 0:
            out = json.loads(sbx.filesystem.read("/replay/out.json"))
        return {
            "exit_code": result.exit_code,
            "stdout": result.stdout,
            "stderr": result.stderr[-2000:],
            "response": out,
        }
    finally:
        # Delete. Every side effect the handler made dies here. Next payload
        # starts from the same clean baked snapshot — deterministic replay.
        sbx.delete()

Two details do the real work. The handler reads `/replay/event.json` and writes its response — and, crucially, its side effects — to known paths under `/replay/` instead of firing them at real systems. In your candidate build's replay mode you point the handler's outbound calls at the filesystem: instead of calling Stripe, it appends what it would have called to `/replay/effects/stripe.jsonl`. Now the side effects are captured artifacts you can inspect and diff, not events that actually happened. And because the VM has no route to Stripe's live API anyway, a bug that skips your replay-mode guard fails closed with a connection error rather than an actual charge.

Capturing and diffing behavior across builds

Reproducing a bug is only half the job — the other half is proving a fix works, and that means diffing behavior between two builds against the same payloads. The pattern is to run the whole captured corpus through the old build and the new build, each in its own disposable VM per payload, and compare the captured results payload-by-payload. Because both runs start from an identical baked snapshot, any difference in the output is a difference your code change caused, full stop. There's no shared-environment noise to explain away.

What you diff, per payload:

• Exit code and status: did the handler that used to crash on this payload now exit cleanly (the fix working), or did a payload that used to pass now start failing (a regression the fix introduced)?
• The response body: the JSON the handler produced. A well-formed diff here tells you exactly which fields the new build computes differently.
• The side-effect artifacts: the captured `/replay/effects/` files — the API calls it would have made, the rows it would have written, the jobs it would have enqueued. This is where a subtle bug hides: same response, but the new build now double-fires an outbound call, or stops emitting one it should.
• stderr and timing: a payload that used to run in 20ms and now hits your exec timeout is a performance regression a response diff would miss entirely.

Run the two builds in parallel — each payload's VM is independent, so there's no reason to go serial. A create is ~179ms p50 (around 203ms at p99), so fanning a few thousand payloads across two builds is a matter of pooling concurrency, not waiting on a single shared box. The first cold boot of a brand-new template build is ~3 seconds, but that happens once and gets baked into the snapshot, so every replay after that pays only the restore cost.

Bisecting a regression by replaying against two builds

When a payload behaves differently between builds and you don't yet know which change caused it, the disposable-VM setup turns bisection into a mechanical loop. You have the one payload that reproduces the regression and a range of commits between the last-known-good build and the current broken one. Bake a template snapshot for a candidate commit, replay the reproducing payload against it in a fresh VM, read the exit code and the effect artifacts, and score it good or bad. That's a `git bisect` step — except the 'test' is a real replay of a real captured payload against a real handler build, in an environment that resets to identical state every iteration.

The determinism is what makes this trustworthy. Ordinary bisection over a flaky, stateful integration environment is miserable because a 'bad' result might be leftover state from the previous run rather than the commit under test — you end up re-running steps and second-guessing the answer. Restoring a fresh VM from a per-commit baked snapshot removes that entirely: each bisection step starts from the same clean slate, the payload is byte-identical, and the only variable is the build. When the loop converges, the commit it lands on is genuinely the one that changed the behavior, and you have the captured effect artifacts from both sides of the boundary to prove exactly what changed.

Teardown, and why it's the whole point

The teardown isn't cleanup you tack on at the end — it's the mechanism. Deleting the VM after each payload is what makes the next replay deterministic and what makes a poison payload harmless. Because the VM is disposable, you never write defensive reset logic between replays: you don't truncate tables, you don't clear caches, you don't hunt for files a previous handler left behind. If a handler corrupted its environment, mined a block, or leaked a hundred file descriptors, it did so inside a VM that's already gone. A few operational rails keep the whole thing honest:

• Always set both timeout_seconds on exec and ttl_seconds on create. The exec timeout kills a handler stuck in a loop on a poison payload; the TTL reaps a VM you forgot to delete, so a crash in your replay driver doesn't leak VMs.
• Deny egress by default. A replay handler has no legitimate reason to reach the live internet — point its outbound calls at the filesystem and give the guest network namespace no route out, so a missed replay-mode guard fails closed instead of firing a real side effect.
• Never inject prod credentials into the replay guest. The handler is being debugged precisely because it might do something wrong; don't hand it the live signing secret or database URL to do it with. Replay mode reads payloads, not secrets.
• Prefer fork-per-payload when state matters. Snapshot one seeded, warmed parent and fork it per payload so every replay starts byte-identical; a same-host fork is 400–750ms (cross-host 1.2–3.5s) and shares memory copy-on-write, so the seeded state isn't recopied.

If your replay setup needs a real database rather than fixtures on disk — say the handler does genuine SQL you want to inspect — you can stand up a managed database per replay batch (create is 30–90s, so you do it once per batch, not once per payload) and point the forked handler VMs at it, then throw the whole database away when the batch finishes. That's heavier than filesystem effect-capture and you should reach for it only when you actually need query-level fidelity; for most replay-and-diff work, capturing what the handler would have written to a file and diffing that is faster and cleaner.

When you don't need a VM per payload

This is real infrastructure — you own the template bakes, the payload plumbing, and the effect-capture wiring. Be honest about whether your handler warrants it. If your handler is a pure function — parse the payload, compute a result, return it, no side effects, no network, no filesystem — then you don't need a VM at all; you need a unit test that feeds it the captured payloads directly in-process, and it'll run in milliseconds. The whole argument here rests on the handler having side effects worth containing and worth diffing.

Reach for disposable microVMs when replaying a payload actually does something — charges, deploys, emails, database writes, outbound API calls — so a stray one hitting a real system is a genuine risk; when your captured payloads include untrusted or adversarial traffic that could steer a handler into destructive behavior; when you need byte-identical determinism across a large corpus to trust a behavior diff; or when you're bisecting a regression and need each step to start from a provably clean slate. In those cases, a fresh guest kernel per payload gives you the one thing a shared staging environment can't: the confidence to replay a payload that might delete production, on a machine where 'delete production' just means deleting a VM you were done with.