Copy-on-Write Memory: Why Forking a VM's RAM Is Cheap

Cloning a running machine means cloning its RAM. A 2 GiB guest has 2 GiB of live memory — page tables, a warm page cache, a loaded model context, a half-built index — and a fork has to reproduce all of it. Do that by copying bytes and a fork costs seconds and doubles your memory bill per branch. PandaStack doesn't copy the bytes. It clones the memory with copy-on-write: the child's RAM is backed by the parent's memory image, nothing is duplicated up front, and the kernel only makes a private copy of a page the instant that page is written. The clone is O(metadata), not O(bytes) — which is why a same-host fork lands in 400–750ms regardless of how much RAM the guest holds. This is the memory half of copy-on-write. Its disk twin — reflink and dm-snapshot on the rootfs — is covered in /blog/copy-on-write-rootfs; this post is about the RAM.

The problem: a fork has to clone gigabytes of RAM

A microVM is a real machine with real physical memory. A Firecracker snapshot serializes that memory into a file — vm.mem — that is the guest's entire RAM, byte for byte. A 2 GiB guest produces a 2 GiB vm.mem. To fork that machine, the child needs its own copy of that RAM: it has to be able to write to its memory without the parent (or any sibling branch) seeing the change, and vice versa.

The naive implementation reads all 2 GiB off disk and hands the child a private duplicate. That is correct and catastrophically slow: multiple seconds of I/O on the fork path, and a full 2 GiB of fresh RAM committed per branch. Fork ten branches and you've moved 20 GiB and committed 20 GiB of memory to hold ten machines that are, at the instant of forking, byte-for-byte identical. Almost all of that work is wasted, because almost none of those pages will ever be written by the child before it's discarded.

The requirement is the same one that copy-on-write disk solves, pointed at RAM instead: give each child a private, writable view of a multi-gigabyte memory image, instantly, without duplicating it. Defer the copy to first write, and only for the pages that are actually written.

How copy-on-write memory works at the process level

This isn't a VM invention — it's how Unix fork() has worked for decades, and the VM case is the same trick one layer up. When a process calls fork(), the kernel does not copy the parent's address space. It marks every writable page in both parent and child as read-only and copy-on-write, and duplicates only the page tables (cheap metadata). Parent and child now point at the same physical pages. As long as both only read, nothing is copied — they genuinely share the same RAM.

The copy happens on the first write. When either process stores to a CoW page, the CPU's MMU sees the page is read-only and raises a page fault. The kernel's fault handler recognizes it as a CoW fault, allocates one fresh physical page, copies the 4 KiB of contents into it, remaps just that page as writable in the faulting process, and lets the store retry. The other process keeps the original. One page diverges; everything else stays shared.

Here is that fault path made concrete — an mmap of a backing file mapped MAP_PRIVATE (copy-on-write), and what the kernel does on a read touch versus a write touch:

/* Map a 2 GiB memory image copy-on-write. No bytes are read yet. */
void *ram = mmap(NULL, 2UL << 30, PROT_READ | PROT_WRITE,
                 MAP_PRIVATE,        /* <-- writes stay private to us */
                 fd_vm_mem, 0);

char c = ram[offset];   /* READ fault:  kernel maps the shared page in
                           straight from vm.mem. No copy. Every process
                           mapping this file shares this one physical page. */

ram[offset] = 0x42;     /* WRITE fault: page is read-only + CoW, so the MMU
                           traps. Kernel allocates ONE fresh 4 KiB page,
                           copies the old contents in, remaps it writable
                           for us only. The shared original is untouched. */

The whole point of MAP_PRIVATE is in that last comment: modifications are private to the mapping and never written back to the file. The file is the shared, read-only baseline; each writer accumulates its own thin layer of diverged pages on top. Everyone who maps the same file shares the pages nobody has written.

Snapshot-restore maps vm.mem MAP_PRIVATE

PandaStack restores a baked Firecracker snapshot on every create — there is no warm pool of idle VMs — and the mechanism is exactly the mmap above. Instead of reading vm.mem into a fresh 2 GiB buffer before the guest can run, the agent maps the memory file MAP_PRIVATE and resumes the vCPUs. The mapping is lazy: no pages are faulted in until the guest actually touches them.

• The guest's RAM is backed by the snapshot's vm.mem, but nothing is eagerly copied. Pages fault in on demand as the guest reads and writes.
• A read fault maps the page straight from the shared vm.mem — no copy. Multiple restores of the same snapshot on one host share those identical pages in the kernel page cache, so a second, third, and tenth restore of a template ride pages the first restore already brought in.
• A write fault copies just that one 4 KiB page private to this guest. The original page in vm.mem stays pristine for every other restore and every future create.

That is a large part of why a create restores a baked snapshot in roughly 49ms and 179ms at the p50 (p99 ~203ms) instead of the ~3s a first cold boot takes — the guest resumes against a lazily-mapped memory image rather than waiting on a 2 GiB read. The full create pipeline is broken down in /blog/how-firecracker-boots-fast and /docs/internals/snapshot-restore; here the relevant fact is that memory is mapped, not copied, on every single create.

Why a fork clones GBs of RAM in O(metadata)

A fork is snapshot-restore pointed at a specific running machine instead of the generic template. PandaStack snapshots the parent's live memory and restores it into the child with the same MAP_PRIVATE mapping. The child's entire address space starts out sharing the parent's pages; it commits private RAM only for the pages it modifies. Because the map is O(metadata) — page tables and bookkeeping, not bytes — the size of the guest's RAM barely affects fork time. A 2 GiB guest and a 4 GiB guest fork in about the same wall-clock, because in neither case are the bytes being moved.

So the fork cost is not a function of how much memory the guest has. It is a function of how far the child diverges. A branch that reads a lot and writes little stays almost entirely shared with the parent for its whole life and costs almost nothing in RAM. A branch that rewrites most of its address space eventually pays for most of those pages — but it pays gradually, page by page, exactly as it dirties them, and never up front.

Contrast the two approaches directly:

• Full memory copy: O(bytes). Read and duplicate all 2 GiB before the child runs. Seconds of I/O on the fork path. Commits a full 2 GiB of fresh RAM per branch, immediately, whether or not the branch ever touches those pages. Ten branches = ten full copies.
• Copy-on-write memory: O(metadata). Map vm.mem MAP_PRIVATE and resume. No data movement up front. Commits RAM only for pages a branch actually writes. Ten branches that barely diverge share almost all of one machine's memory footprint — the cost is the sum of what each branch dirtied, not ten times the whole.

This is why fork latency tracks locality rather than RAM size. A same-host fork runs in 400–750ms: the parent's memory is already resident on the machine, so the child's MAP_PRIVATE mapping shares physical pages that are already in RAM, the rootfs reflinks locally, and the fork is a memory map, a metadata disk clone, and a resume. A cross-host fork takes 1.2–3.5s because the parent's resident RAM isn't on the destination host — the vm.mem artifact has to move over the network before the target can map it. PandaStack's scheduler defaults a fork to the parent's host for exactly this reason; cross-host is opt-in for when you need a branch to land elsewhere.

Nobody copies the RAM. A fork maps the parent's memory image copy-on-write and resumes — so branching a live machine costs about what a create costs, not what a reboot costs, and the price is paid one 4 KiB page at a time as the branches actually diverge.

Where this pays off: fork-tree-of-thought and best-of-N

The reason cheap memory cloning matters for AI agents is that agents constantly hit branch points. The model has built up expensive, warm in-memory state — a cloned repo, an indexed codebase, a loaded dataset, a half-finished migration, a filled KV cache — and now there are several plausible next moves. Without cheap forking, exploring each move means either serializing (try A, roll back, try B) or replaying the entire costly setup N times from a clean create.

Copy-on-write memory turns that into a branch. You fork one warm VM into N children. Each child inherits the parent's exact RAM — the warm caches, the loaded context, the running processes — and then explores a different path. Because every child shares all the untouched memory with the parent, the fan-out is cheap: N branches that each dirty a little cost roughly the parent's footprint plus the small per-branch diverged set, not N full machines. You let each branch run its candidate, evaluate the results, keep the winner, and delete the losers. The losing branches barely cost anything, because copy-on-write meant they never allocated much in the first place.

That is best-of-N branching — and the deeper fork-tree-of-thought pattern, where you fork, evaluate, and fork the winner again into a tree of exploration. Each level shares the level above it via CoW, so a whole tree of branches stays anchored on one warm baseline. The agent-facing view of this is in /blog/snapshot-fork-tree-of-thought and /blog/snapshot-and-fork-explained; the point here is that it's economical because the memory is shared, not copied.

The fork in code

The whole loop is a few lines with the Python SDK: create a warm sandbox, build up expensive state, then fork it into parallel branches that each share that state via copy-on-write memory. Set PANDASTACK_API_KEY in your environment first; install with pip install pandastack.

from pandastack import Sandbox

# 1. Create a warm sandbox (p50 179ms via snapshot-restore) and load it up
#    with the expensive state we want to branch from.
parent = Sandbox.create(template="base", ttl_seconds=3600)
parent.exec("git clone --depth 1 https://github.com/acme/service /work")
parent.exec("cd /work && npm ci")          # warm node_modules + build cache
parent.filesystem.write("/work/goal.md", "make the failing test pass")

# 2. Fork the LIVE machine into 4 children. Each MAP_PRIVATE-maps the
#    parent's RAM, so all 4 share the warm caches + loaded context.
#    Same-host fork is 400-750ms each and copies zero bytes up front.
candidates = [
    "apply --strategy=retry-with-backoff",
    "apply --strategy=null-guard",
    "apply --strategy=widen-types",
    "apply --strategy=reorder-init",
]
for patch in candidates:
    child = parent.fork()                   # branch shares parent's memory (CoW)
    child.exec(f"cd /work && ./fix {patch}")
    result = child.exec("cd /work && npm test", timeout_seconds=300)
    print(child.id, "->", "pass" if result.exit_code == 0 else "fail")
    # Losing branches barely cost anything: CoW meant they never
    # allocated the untouched pages they inherited.
    if result.exit_code != 0:
        child.kill()

# The parent still holds the pristine warm state — fork again anytime.
parent.kill()

Each returned child is a full Sandbox — its own guest kernel and hardware-virtualization boundary — that you can exec into, read and write files in, snapshot, or fork further into a deeper tree. Nothing a child writes is visible to the parent or its siblings, because the moment it writes a page, copy-on-write hands it a private copy. The isolation is the microVM boundary; the cheapness is the CoW memory underneath it.

Memory CoW vs disk CoW vs UFFD streaming

It's worth being precise about three related mechanisms that all wear the copy-on-write name, because they operate on different things and it's easy to conflate them:

• Copy-on-write MEMORY (this post): a MAP_PRIVATE mmap of the snapshot's vm.mem. Shares physical RAM pages between parent and children until a page is written, at which point a 4 KiB page is copied private. This is what makes forking a guest's RAM O(metadata).
• Copy-on-write DISK (/blog/copy-on-write-rootfs): XFS reflink or dm-snapshot on the rootfs.ext4. Shares disk extents/blocks between the template and each sandbox until a block is written. This is what makes cloning a multi-GB disk O(metadata). Same principle, different substrate — blocks instead of pages.
• UFFD streaming (/blog/userfaultfd-explained): the on-demand-from-remote variant of memory paging. When vm.mem lives in object storage rather than on the local host, a userfaultfd handler serves the guest's page faults by fetching chunks over the network instead of from a local file. It removes the multi-GB local memory download; the copy-on-write semantics the guest sees are unchanged. It's the streaming source for the same page-fault machinery this post describes.

A fork is the memory and disk mechanisms working together: MAP_PRIVATE on the RAM and a reflink on the rootfs, both deferring their copies to first write. Neither moves real data up front, which is the whole reason a live machine branches in the time it takes to restore one — not the time it takes to reboot one.

The honest caveats

• Divergence has a cost. The clone is cheap; a branch that rewrites most of its RAM eventually copies most of those pages. The O(metadata) win is on the fork, not the lifetime — a long-lived, write-heavy branch ends up roughly as large as a full copy. The savings are largest for short-lived branches and slowly-diverging trees.
• It's a density optimization, not the isolation model. Copy-on-write memory makes forks cheap; the KVM boundary and a separate guest kernel per microVM are what isolate one sandbox from another. Each fork is a full Firecracker microVM for that reason — sharing pages does not mean sharing a security domain.
• Forking clones the machine, not the outside world. If a branch sends an email, charges a card, or writes to a shared external database, every branch does it. CoW protects in-VM memory and disk, not third-party systems. External network connections generally need re-establishing per branch.
• The parent's RAM must be reachable to stay cheap. Same-host forks share resident memory and are fast (400-750ms); cross-host forks (1.2-3.5s) pay to move the vm.mem artifact first, because MAP_PRIVATE can only share pages that are actually present on the host.

The mental model that holds up: a fork's memory is a reference, not a copy. The child's RAM is the parent's memory image mapped copy-on-write, shared until the child writes, at which point only the written pages diverge — one 4 KiB page at a time. That is what turns "clone gigabytes of RAM" into a few hundred milliseconds of metadata, and it's the exact same copy-on-write principle that its disk twin applies to blocks in /blog/copy-on-write-rootfs. PandaStack's core is open source under Apache-2.0, so you can run the agent on your own KVM hosts and watch the page-fault-driven divergence yourself — fork a warm VM, dirty a few pages in the child, and time how little it actually costs. For the broader story, /blog/snapshot-and-fork-explained and /docs/concepts/snapshots-and-forks.