Snapshot the Failure, Not the Log Line

Everyone who has chased a one-in-four-hundred test failure knows where the time actually goes. It is not the analysis. Once you are standing in front of the broken machine the answer usually arrives in ten minutes. The expensive part is getting back there: re-running the suite two hundred times, adding a print statement, re-running it two hundred more, discovering the print statement was in the wrong place, and losing a Thursday.

That loop exists because of an assumption we inherited from deterministic bugs — that a failure can be summoned on demand. When it cannot, the only artefact you keep is whatever you happened to be logging at the time, which is to say a projection of the machine's state chosen in advance by someone who did not know what was going to break. A log line tells you what happened. A memory snapshot taken at the moment of failure is the state: the heap, the stack, the process table, the open file descriptors, the half-written temp file, the database the test was talking to.

On a platform where freezing a guest is a routine operation rather than an operations project, "on failure, snapshot instead of tearing down" is a plausible default for a CI harness. This post is the shape of that harness, the trap in it that will eat your evidence, and — the part I care most about getting right — exactly what a restored guest does and does not give you, because it is emphatically not deterministic replay.

The harness: snapshot instead of tearing down

The mechanism is a pause window. The platform pauses the guest, writes its memory and vCPU state out, takes a reflink copy of the rootfs inside that same window so the disk and the memory agree with each other, and resumes. The sandbox carries on running afterwards; a snapshot is non-destructive. What you get back is an identifier you can hand to a create call later.

The one design decision worth making deliberately is when you take it. If you run the suite synchronously and snapshot after it exits, you capture the aftermath: the files, the logs, the residue, the database rows. That is often enough. If the failure mode is a hang, the aftermath is not what you want — you want the process still sitting in whatever syscall it is stuck in. So run the suite detached inside the guest, poll for a completion marker, and snapshot both when it fails and when it fails to finish.

import json, os, time
from pandastack import Sandbox

COMMIT = os.environ["GIT_COMMIT"]
SEED = os.environ.get("TEST_SEED", "0")
SUITE = "tests/integration/test_ledger.py::test_concurrent_settle"

sbx = Sandbox.create(
    template="code-interpreter",
    # A TTL, not a kill() in a finally block. The reason is the next section.
    ttl_seconds=3600,
    metadata={"kind": "ci-run", "commit": COMMIT, "seed": SEED},
)
sbx.exec("git clone --depth 50 https://github.com/acme/ledger /work", check=True)
sbx.exec(f"cd /work && git checkout {COMMIT} && pip install -e '.[test]'", check=True)

# Detached, so a HANG is a state we can freeze rather than a client-side
# timeout. The exit code lands in a file; its absence means "still running".
sbx.exec(
    "cd /work && setsid nohup sh -c "
    f"'python -m pytest -x -q {SUITE} > /run.log 2>&1; echo $? > /run.exit' "
    "</dev/null >/dev/null 2>&1 &"
)

deadline = time.time() + 600
code = None
while time.time() < deadline:
    out = sbx.exec("cat /run.exit 2>/dev/null || true").stdout.strip()
    if out:
        code = int(out)
        break
    time.sleep(5)

if code == 0:
    sbx.kill()                      # green run: nothing to keep
else:
    # 1. Provenance goes INSIDE the guest, before the freeze. The snapshot
    #    registry records id, sandbox_id, template, size and created_at --
    #    there is no label field -- so the only durable place to write
    #    "which commit, which seed, which attempt" is the machine itself.
    sbx.filesystem.write("/failure.json", json.dumps({
        "commit": COMMIT, "seed": SEED, "suite": SUITE,
        "exit_code": code,               # None == it hung
        "captured_at": time.time(),
    }))
    snap_id = sbx.snapshot()        # pause -> mem + state + disk -> resume
    print(f"FAILED (exit={code}) snapshot={snap_id}")
    # 2. Deliberately no kill() here. Read the next section before you add one.

Two details are load-bearing. The provenance file is written into the guest before the freeze because the control-plane snapshot registry has no metadata column — it stores the snapshot id, the source sandbox id, the template, a size and a timestamp, and that is all. Putting the commit and the seed inside the machine means the evidence carries its own chain of custody, and a restored guest can tell you what it is without you having to keep a side table. The second detail is what is missing: there is no `kill()` on the failure path.

The trap: deleting the sandbox deletes your evidence

This is the part that would have cost you a week, so it gets its own section. Deleting a sandbox explicitly — `sbx.kill()`, or a `DELETE` on the sandbox — cascade-deletes every snapshot taken from that sandbox. The local bytes, the mirrored copy in object storage, and the registry row all go. The reasoning is sound as a default: you asked for the machine to be gone, so its derived artefacts go with it. It is nonetheless the exact opposite of what a failure-capture harness wants, and a `try/finally: sbx.kill()` around your test run will quietly destroy the snapshot you took four lines earlier.

Once the source sandbox is gone, the snapshot is an orphan, and orphans are reclaimed on a grace period rather than immediately — seven days by default on our fleet, swept every fifteen minutes, both configurable on the agent. That grace period is your retention policy whether you meant to choose one or not. If a failure snapshot needs to outlive it, either restore it and extract what you need, or treat the seven days as a deadline for turning the finding into a deterministic test, which is what you wanted to do anyway.

Why a whole machine beats a core dump

The traditional answer to "capture the failing state" is a core dump, and core dumps are genuinely useful. They are also one process's address space and nothing else, which is a poor fit for the failures that are actually hard.

• Scope. A core dump is the memory of the process that died. A VM snapshot is every process on the machine — the test runner, the database it was talking to, the sidecar, the queue worker, the thing holding the lock. Distributed-ish bugs inside one machine stop being distributed.
• The filesystem comes too. Disk is captured inside the same pause window as memory, so the half-written file, the lock file, the sqlite journal and the log the process had not flushed are all consistent with the memory image. A core dump has no opinion about your filesystem.
• It runs. This is the real difference. A core dump is a corpse you inspect with a debugger that has to be told what the binary looked like. A restored snapshot is a live machine that happens to be wrong, and you can type into it: run a query, strace something, read /proc, poke the state and watch what happens next.
• No debuginfo scavenger hunt. Matching a core dump to the exact binary and symbols that produced it is its own small misery, especially six weeks later. The snapshot contains the binary, the symbols, the interpreter, the site-packages tree and the environment variables, because it contains the machine.
• The price is size. A core dump is one process. A snapshot is the guest's RAM — on our base template that is 4 GiB per capture, because guest memory is baked into the template rather than requested per create. If your CI goes red thirty times a day, that is a real number and it is why the retention question above is not academic.

Handing out copies without anyone racing anyone

Restoring is an ordinary create that names a snapshot instead of booting a template. The restored guest comes back on the instruction it was frozen on, in a fresh network namespace of its own. You can do it as many times as you like, in parallel, from the same snapshot — which is the thing that makes this pleasant socially as well as technically. Three engineers each get their own copy of the broken machine, nobody steps on anybody's breakpoint, and nobody has to ask whether they are allowed to restart the process to test a theory.

import json
from pandastack import Sandbox

# Everyone who wants a look runs this. Each call is an independent guest.
scene = Sandbox.create(
    template="code-interpreter",
    from_snapshot="<snapshot-id>",
    ttl_seconds=7200,
    metadata={"kind": "debug", "engineer": "ajay"},
)

print(json.loads(scene.filesystem.read("/failure.json")))   # what am I looking at
print(scene.exec("ps auxf").stdout)                          # who was still alive
print(scene.exec("cat /run.log | tail -50").stdout)          # the failure, verbatim
print(scene.exec("ls -la /tmp /work/.pytest_cache").stdout)  # the residue

# A hang: the process is still in the snapshot, still stuck where it was.
print(scene.exec("pid=$(pgrep -f pytest | head -1); cat /proc/$pid/stack /proc/$pid/wchan 2>/dev/null; ls -l /proc/$pid/fd").stdout)

There is one API distinction here that catches people, and it is worth stating plainly because the words work against you. `fork()` and `fork_tree()` are not the same operation. `fork()` pauses the parent, copies its rootfs, and boots each child fresh from that disk: the children get the parent's filesystem, and none of its running processes. That is the right primitive for "pre-install dependencies once, then fan out workers", and completely the wrong one for debugging, because the state you care about was in RAM. `fork_tree(count)` snapshots the parent once and restores the children from that snapshot, so they inherit memory and disk both. Children are capped at sixteen per call.

Fanning out from a pre-failure state

The second use of the same machinery is narrowing a nondeterministic failure in time. Snapshot the guest just before the window you suspect, then fan out sixteen children from that one state and let each run the suspicious section. Every child starts from a byte-identical machine, so the environment is controlled for in a way that re-running on a shared runner never is, and what you get back is a count rather than a shrug.

from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox

parent = Sandbox.get("<sandbox-id>")   # already at the pre-failure state

# One snapshot, sixteen restores of it, in parallel. 16 is the per-call cap.
kids = parent.fork_tree(count=16, metadata={"kind": "bisect"})

def attempt(child):
    try:
        r = child.exec("cd /work && python -m pytest -x -q tests/integration/test_ledger.py")
        return r.exit_code
    finally:
        child.kill()      # no snapshots taken from these, so kill is safe

with ThreadPoolExecutor(max_workers=len(kids)) as pool:
    codes = list(pool.map(attempt, kids))

print(f"{sum(1 for c in codes if c != 0)}/{len(codes)} reproduced")

Move the snapshot point earlier and later and the number moves with it, which localises the failure to a region of the run without you having to reason about it. Be clear-eyed about what the number means, though: the children are not replays. Each one resumes with its own scheduling, its own entropy and its own view of the network, so you are sampling a distribution rather than re-running a trace. A result of zero out of sixteen is still information — it says the variance you are hunting entered before your snapshot point, not after it.

What survives a restore, precisely

Here is the honest inventory, because everything above is only useful if you know which parts of the machine you are allowed to trust.

Preserved: guest RAM in full, so the heap, the stacks, every thread's registers and anything anonymous the process had mapped. Kernel state, because the kernel lives in that RAM — the process table, open file descriptors, pipes, Unix sockets, the page cache, the state of a process blocked in a syscall. The disk as of the same pause window, which was not always true: memory-only snapshots restored over a fresh template disk and silently lost every file written after the template was baked, and capturing the rootfs inside the pause window is what fixed it. And the vCPU state, so execution resumes at the instruction it stopped on.

Not preserved, in rough order of how much trouble each will cause you:

• Anything crossing the guest boundary over TCP. The guest's kernel comes back believing those sockets are still ESTABLISHED. The peer has never heard of this connection, and the host-side NAT mapping is new. So you get retransmits and then a reset or a long silence, and the failure surfaces as an application-level mystery rather than a clean "connection closed". This is the single most common source of confusion with restored guests.
• Sessions with anything external. A database connection, an OAuth session, a licence checkout, a websocket, a lease with a coordinator — all of them are gone from the other side's point of view, even though your process still holds a handle it thinks is fine.
• The wall clock. CLOCK_REALTIME is frozen at the instant of the snapshot; there is no RTC in the microVM and the templates do not run NTP. The platform re-syncs it on every restore, resume and wake by running the guest equivalent of `date -u -s @<epoch>` — best-effort, second precision, and it fires after the guest is reachable. So the clock inside your restored crime scene is correct-ish now and was frozen a moment ago, which is exactly the kind of discontinuity a timestamp-sensitive bug will notice.
• Scheduling and entropy from here on. The instant you resume, thread interleaving, timer expiry and randomness are all fresh. Two restores of the same snapshot diverge immediately and legitimately.
• Anything host-side. The hypervisor log, the console log, the platform's own metrics — these live on the host, not in the guest, and they are not in the snapshot. If you want them, collect them separately at capture time.

A restored guest is a machine that is correct about its own past and wrong about the world. Everything inside the boundary is evidence; everything crossing it is a claim that expired.

This is not rr, and the difference matters

If you have used rr or Pernosco you will recognise a family resemblance and it is worth being precise about where it ends. Those tools record the nondeterministic inputs to a process — syscall results, signal delivery, thread interleaving — so that a later run is bit-for-bit the same run, and then they let you execute backwards. That is a genuinely stronger guarantee than anything described here, and if your bug fits inside one process and rr works on your hardware, use rr.

• Scope: rr records one process tree. A snapshot captures the whole machine, including the database, the sidecar and the kernel state that a single-process recorder cannot see.
• Guarantee: rr replays the same execution every time. A restore resumes forwards and diverges immediately — same starting state, different run.
• Direction: rr gives you reverse execution, which is the feature people actually fall in love with. A snapshot gives you no such thing; you can only start again from the frozen instant.
• Cost model: rr pays recording overhead on every run, including the vast majority that pass. Snapshotting pays nothing until something fails, then pays for storage until you delete it.
• Prerequisites: rr leans on hardware performance counters, which a guest is not automatically given. Before you plan a workflow around running it inside a microVM, run rr's own check in the guest and find out — the answer is a property of your hypervisor and CPU, not something a blog post can promise you.

They also compose, which is the underrated option. If rr does run in your guest, the snapshot preserves the recording plus the entire machine around it, so "the recording, its inputs, the database it was pointed at, and the config that produced it" becomes a single artefact instead of four things that have to be kept in sync. And if rr does not run there, snapshot-restore is still a large upgrade on a log file, which is what you were comparing against in reality.

What it costs to keep the evidence

A snapshot is roughly the guest's RAM in bytes, so the base template's 4 GiB is your unit of account. Creating one is synchronous and takes tens of seconds on a multi-GB guest — both SDKs raise the per-call HTTP timeout to three minutes for exactly this reason — and if a replication bucket is configured the mirror upload to object storage happens synchronously too, so that a restore on a different host can find the bytes. That mirroring is what makes "restore it on whatever host has room" work; without it a restore has to land back on the host that took the snapshot.

The published rate card prices running guests — per vCPU-hour and per GiB-hour — plus provisioned volume storage. Snapshot bytes are not a line item on it today. That is a statement about the current rate card rather than a promise about the future, and the operational point stands regardless: the seven-day orphan grace is the thing actually bounding your storage, so decide what you want that number to be before your first red Friday rather than after.

When not to bother

• The failure is deterministic. If `pytest -x` reproduces it locally on the second attempt, you do not have a reproduction problem and this is machinery for a problem you do not have.
• The cause is entirely in the request. If the interesting variable is one webhook payload, capture and replay the payload — far cheaper than four gigabytes of RAM, and easier to put in a test.
• The cause is outside the guest. A host-level problem, a network partition, or an upstream API behaving badly will not be in the snapshot, and the snapshot will make you very confident about the wrong machine.
• Your red rate is high. If a hundred runs a day fail, you have a triage problem before you have a debugging problem, and snapshotting all of them turns it into a storage problem as well. Snapshot the quarantined ones.
• You need reverse execution. Nothing here is a substitute for it. That is a different tool, and the honest answer is to go and use it.

The underlying shift is small and worth naming. We tolerate lossy failure evidence because capturing the whole machine used to be an ordeal — a heavyweight VM, a manual freeze, a storage conversation. When a freeze is an API call and a restore is a routine create, the default can invert: keep the machine, throw away the ones you did not need, and stop asking people to guess in advance which variable will turn out to matter.

Start with one test. Take the flakiest thing in your suite, wire the snapshot-on-red path around it, and the next time it goes red, restore it and have a look before you retry. If the first thing you find is something you would never have logged, you have your answer about whether the pattern is worth generalising.