Snapshots and Forks: Copy-on-Write for Running Machines

A snapshot freezes a running microVM — its entire RAM, CPU registers, and virtual device state — to disk, so it can be brought back to life exactly where it left off. A fork clones that frozen machine using copy-on-write: the child shares the parent's memory pages and disk blocks until it writes to them, so branching a live environment costs around 400ms instead of a full boot. On PandaStack, both operations sit on the same Firecracker primitive, and the same machinery is what makes every sandbox create land in 179ms at the median. This post walks through what each captures, why copy-on-write makes it fast, and how to fork a running agent into parallel branches with a few lines of Python.

What a microVM snapshot actually captures

A microVM is a real machine: a guest kernel, its page tables, a CPU with registers and a program counter, and emulated devices (a virtio block device, a network card, a serial console). A Firecracker snapshot serializes all of that at a single instant.

Concretely, a PandaStack snapshot is three artifacts:

• vm.mem — the guest's entire physical RAM, byte for byte. This is the big one: a 2 GiB guest produces a 2 GiB memory file.
• vm.state — the VMM's serialized state: vCPU registers, the interrupt controller, the clock, and every virtio device's configuration. Small, but it is what lets the guest resume mid-instruction rather than reboot.
• rootfs — the disk. Snapshots clone the ext4 rootfs as a copy-on-write image rather than copying every block.

The key property is that this is not an image you boot from — it is a paused running machine. Restore it and the guest does not see a reboot. Processes that were mid-execution keep running, the page cache is warm, open files stay open. There is no init, no systemd, no kernel handshake. That is why restore is measured in tens of milliseconds, not the seconds a cold boot takes.

Snapshot-restore is the basis of 179ms creates

Most sandbox platforms keep a warm pool of idle VMs so a create can just hand you one. PandaStack does not. There is no pool of pre-booted machines burning RAM. Instead, every create restores a baked template snapshot on demand. The first time a template is ever spawned, the agent does a real cold boot (~3s) and captures a snapshot of the booted, ready guest. Every create after that restores that snapshot.

That is what gets a fresh sandbox to a usable state with a p50 of 179ms. Restoring the memory file plus device state and resuming the vCPUs is fast on its own, and the surrounding create pipeline — allocating a pre-built network namespace, reflinking the rootfs, launching Firecracker — is engineered to overlap with it. The deep mechanics of that path are covered in the snapshot-restore internals docs.

Snapshot-restore is the foundation. Fork is the same primitive pointed at a different starting point: instead of restoring the generic template snapshot, you snapshot a specific running sandbox and restore that. Create starts from the clean template baseline; fork branches your live machine.

Copy-on-write memory: MAP_PRIVATE and page faults

Naively, restoring a 2 GiB snapshot means reading 2 GiB off disk before the guest can run. That would be slow and would make forking proportionally expensive in both time and RAM. Copy-on-write removes that cost.

On restore, PandaStack maps the memory file with MAP_PRIVATE. The mapping is lazy and copy-on-write at the page level:

• The guest's RAM is backed by the snapshot file, but nothing is eagerly copied. Pages are faulted in by the kernel only when the guest first touches them.
• A read fault maps the page straight from the shared snapshot file — no copy. Multiple restores of the same snapshot share those identical pages in the page cache.
• A write fault triggers the copy: the kernel makes a private copy of just that 4 KiB page for this guest, and the original stays untouched for everyone else.

The consequence for forking is direct. A child fork starts out sharing essentially all of its memory with the parent snapshot. It only allocates private RAM for the pages it actually modifies. Two forks that diverge slowly stay cheap for a long time, because most of their RAM is still the shared, read-only baseline.

When the memory file lives in remote object storage rather than on the local host, PandaStack can stream pages on demand over the network using userfaultfd (UFFD), so restore does not block on downloading the whole memory image first. That is an optional path; the copy-on-write semantics the guest sees are the same.

Copy-on-write disk: reflink rootfs

Memory is half the machine; the disk is the other half. Cloning a multi-gigabyte rootfs by copying bytes would dominate fork latency. PandaStack uses XFS reflinks instead.

A reflink (cp --reflink) creates a new file that points at the same underlying data extents as the original. The clone is an O(metadata) operation — it writes a bit of filesystem bookkeeping and returns, regardless of how big the file is. The two files share data blocks until one of them writes, at which point the filesystem copies just the affected blocks for the writer. Same copy-on-write idea as memory, applied to the block device.

So a fork is two copy-on-write clones happening together: MAP_PRIVATE on the memory file and a reflink on the rootfs. Neither copies real data up front. The child diverges from the parent block by block and page by page, only as it actually changes things.

What fork gives you: parallel agent branches

The reason fork matters for AI agents is that agents constantly hit branch points. The model has built up real context — a cloned repo, an indexed codebase, a half-finished migration, a loaded dataset — and now there are several plausible next moves. Without forking, exploring each one means either serializing (try A, undo, try B) or replaying the whole expensive setup N times.

Fork turns that into a branch. You snapshot the agent's exact running state and clone it into N independent sandboxes. Each branch inherits everything — files on disk, processes, the warm in-memory state — and then evolves independently. You let each branch run its candidate action, evaluate the results, and keep the winner. The losing branches are just deleted; copy-on-write means they never cost much in the first place.

Because every fork is itself a full microVM with its own guest kernel and hardware isolation, branches cannot interfere with each other or with the parent. And because you can snapshot before forking, you can always resume the parent and branch again from a known-good point. This is the same model as branching in version control, except the thing you are branching is a live, executing machine.

Same-host vs cross-host fork cost

Fork latency depends on where the child lands relative to the parent's data.

• Same-host fork (~400ms): the parent's memory is already resident on the machine and the rootfs can be reflinked locally. There is no data movement — just a reflink, a memory map, and a resume. This is the default and the fast path.
• Cross-host fork (slower): if the child must run on a different host, the snapshot's memory and disk artifacts have to be moved across the network before the target can restore. Reflink only works within a single filesystem, and the parent's resident RAM is not on the destination machine, so cross-host trades latency for placement flexibility.

Because of this, PandaStack's scheduler defaults a fork to the parent's host — the placement that keeps memory and rootfs local. Cross-host forking is opt-in, for when you specifically need a branch to land elsewhere (capacity, isolation, region). For fan-out into many parallel branches, keeping them on the parent's host is both faster and lets every branch share the parent's page cache.

Create, work, then fork into branches (Python SDK)

Here is the full loop with the Python SDK: create a sandbox, do some work to build up state, snapshot it as a clean checkpoint, then fork that state into several parallel branches and run each independently. Set PANDASTACK_TOKEN in your environment first; install with pip install pandastack.

from pandastack import Sandbox

# 1. Create a sandbox from a template (p50 ~179ms via snapshot-restore)
box = Sandbox.create(template="base", ttl_seconds=3600)

# 2. Do real work — this state is what we'll branch from
box.exec("git clone --depth 1 https://github.com/acme/service /work")
box.exec("cd /work && npm ci")            # warm node_modules, build caches
box.filesystem.write("/work/plan.md", "refactor the auth module")

# 3. Snapshot the running machine as a clean checkpoint we can resume
checkpoint = box.snapshot()
print("checkpoint:", checkpoint)

# 4. Fork the live environment into 3 parallel branches (~400ms each,
#    same-host). Each inherits the cloned repo, installed deps, plan.md.
branches = box.fork_tree(count=3)

# 5. Explore a different candidate in each branch, independently
strategies = [
    "git checkout -b try-jwt   && ./scripts/apply jwt",
    "git checkout -b try-oauth && ./scripts/apply oauth",
    "git checkout -b try-paseto && ./scripts/apply paseto",
]
for branch, strategy in zip(branches, strategies):
    result = branch.exec(f"cd /work && {strategy} && npm test", timeout_seconds=300)
    print(branch.id, "->", "pass" if result.exit_code == 0 else "fail")

# 6. Keep the winner, drop the rest. Copy-on-write means the losers
#    barely cost anything.
for branch in branches:
    branch.kill()

# The parent still holds the checkpoint — resume and branch again anytime.
box.kill()

fork_tree(count=N) is just a convenience over calling fork() N times; use box.fork() when you want a single branch. Each returned object is a full Sandbox you can exec into, read and write files in, snapshot again, or fork further into a deeper tree. For the broader lifecycle — pausing, hibernate/wake for scale-to-zero, and fork parent-child trees — see the snapshots and forks docs.

When not to reach for fork

Fork is the right tool for branching shared, expensive state. It is the wrong tool in a few cases worth naming honestly:

• Independent, clean-start jobs: if each task wants a fresh template with no shared context, just call Sandbox.create() N times. A 179ms create from the template is simpler than forking and gives you a guaranteed-clean baseline.
• External side effects that don't fork: forking clones the machine, not the outside world. If a branch sends an email, charges a card, or writes to a shared external database, every branch will do it. Copy-on-write protects in-VM state, not third-party systems.
• Stateful network connections to outside services: a forked TCP socket to an external API is now held open by two machines. In-guest sockets resume fine; connections to the outside world generally need to be re-established per branch.
• Long-lived divergence: the cheaper a fork stays, the more it shares. If branches are going to diverge almost completely and run for a long time, the copy-on-write savings erode and a plain create is cleaner.

The mental model that holds up: create restores the template baseline, snapshot freezes your specific running machine, and fork branches that frozen machine with copy-on-write so the branches are cheap until they diverge. For agents that need to explore several paths from a hard-won context, that branch-and-keep-the-winner loop is the whole point — and it runs in roughly the time it takes to read this sentence.