Shared Pages & Copy-on-Write: Packing MicroVMs Densely

The cheapest page of memory is the one you never had to copy. That one sentence is most of what makes a Firecracker microVM platform economical, and this post is about why. When PandaStack restores a sandbox, it does not cold-boot a fresh Linux into a fresh, unique set of dirty pages. It restores a baked snapshot, and it maps that snapshot's memory copy-on-write over a shared, read-only backing. So a hundred sandboxes spawned from the same template don't start with a hundred private copies of an identical guest kernel and warmed-up runtime — they start sharing one physical copy of it, and only diverge, page by page, as each guest actually writes. This is the same copy-on-write idea we use for the rootfs disk via reflink, applied to RAM. This post walks the mechanic, where it stops paying off, how KSM (kernel same-page merging) is the general-purpose but costlier cousin, and how the whole thing stacks with overcommit and on-demand paging to drive real density.

Cold boot manufactures unique pages; restore inherits shared ones

Consider two ways to get a hundred ready guests on one host. The first is the obvious one: cold-boot each microVM. Every guest runs its own kernel init, brings up its own userland, loads its own libc and interpreter, warms its own caches. Functionally the hundred guests end up nearly identical — same kernel version, same libraries, same warmed runtime — but the host doesn't know that. Each guest wrote those pages independently, into its own address space, so the kernel is holding a hundred separate physical copies of what is, byte for byte, the same memory. A hundred copies of the same idea, and the host pays RAM for every one.

The second way is snapshot-restore. You bake one guest to a ready state exactly once, capturing its physical RAM into a memory image. Then every create maps that same image copy-on-write. Now the hundred guests don't each manufacture their own copy of the common pages — they all point at the one shared image. The identity that cold boot threw away, restore preserves: because the pages came from a single source file, the kernel can back them with a single physical copy and let everyone read it. The sharing isn't discovered after the fact; it's structural. You never made the duplicates in the first place.

This is why 'no warm pool' and 'dense memory' go together on PandaStack rather than fighting each other. Every create restores a baked snapshot — p50 179ms, p99 ~203ms, with the snapshot-load step around 49ms; only the very first spawn of a fresh template pays the ~3s cold boot to bake the snapshot everyone else inherits. That architectural choice is also the memory-sharing choice: the same baked image that makes restore fast is the shared backing that makes restore cheap.

The mechanic: MAP_PRIVATE and write-triggers-a-copy

The whole trick lives in one mmap flag. When the agent restores a snapshot, it maps the guest's memory image with MAP_PRIVATE — a private, copy-on-write view over a shared file. 'Private' here means private on write, not private on read. Every guest that maps the same image with MAP_PRIVATE shares the underlying physical pages for reading. The kernel only breaks the sharing when a guest writes: the write faults, the kernel allocates one fresh physical page, copies the original 4 KiB into it, re-points that single guest's page table at the private copy, and lets the write land. Every other guest keeps reading the original, untouched, out of the one shared copy.

Read that mechanic carefully and the accounting falls out of it. A read fault on a shared page costs no new RAM — it maps the existing shared page. A write fault costs exactly one page, and only for the page that was written. So the divergence between two sibling guests grows precisely as fast as they modify memory, and no faster. A guest that boots from the snapshot and idles has written almost nothing, so it's sharing almost everything. A guest hammering a big in-memory dataset has forked off a private copy of most of it. The cost is a faithful bill for exactly what each guest actually changed.

You can see the same flag family at both layers. Below, MAP_PRIVATE gives copy-on-write RAM over the shared snapshot; cp --reflink gives copy-on-write disk over the shared rootfs; both defer the copy to first write:

/* Memory: map the shared snapshot image copy-on-write. Every guest that maps
 * the same vm.mem this way READS from one shared set of physical pages. */
void *mem = mmap(NULL, mem_size,
                 PROT_READ | PROT_WRITE,
                 MAP_PRIVATE,          /* <-- copy-on-write: shared until written */
                 vm_mem_fd, 0);
/* A read fault maps the shared page. A WRITE fault copies just that one 4 KiB
 * page private to this guest; every other guest still sees the original. */

# Disk: the same idea for the rootfs. reflink clones share data extents until
# a write copies a single block -- O(metadata), not O(data).
$ cp --reflink=always rootfs.ext4 clone.ext4   # ~milliseconds, ~no new space

# Memory: watch the shared-vs-private split of a running guest.
# Shared_Clean = pages read straight from the shared backing (free to share).
# Private_Dirty = pages this guest has written, so CoW forked them off (the real bill).
$ grep -E 'Shared_Clean|Private_Dirty' /proc/<firecracker-pid>/smaps_rollup
Shared_Clean:     1521664 kB   # shared with the template -- one physical copy
Private_Dirty:      74236 kB   # what THIS guest actually dirtied since restore

Where sharing stops helping: the working set dirties

Copy-on-write sharing is not a perpetual-motion machine, and it's worth being blunt about where it runs out. The sharing is large exactly while guests are reading their common image and small once they've rewritten it. A guest that loads a 1 GiB dataset into RAM and mutates it in place will, page by page, copy that gigabyte private to itself. At that point it shares essentially nothing of that region with its siblings, and you're paying full freight for it. The O(1)-per-guest illusion holds for the unwritten majority of memory and dissolves for the working set the guest actively churns.

So the savings are shaped by your workload, not fixed. Fleets of mostly-idle, mostly-reading guests — code interpreters waiting for the next request, agents parked between tool calls, preview environments serving static bytes — share enormously, because their post-boot memory is overwhelmingly the untouched common image. Fleets of guests each churning a large private working set share little after warm-up, because each one has dirtied its own pages. The mechanism is the same; the payoff tracks how much your guests read versus write. Don't promise yourself a fixed multiplier — measure Private_Dirty across a representative fleet and let that set your expectations.

KSM: the general-purpose cousin, and why it costs more

Copy-on-write from a shared snapshot gives you sharing for free because the pages started identical — the kernel never had to go looking for duplicates. KSM (kernel same-page merging) solves the harder, general version of the problem: find identical pages that were produced independently and merge them after the fact. It's what you reach for when your guests didn't come from a common backing but still happen to hold matching pages — the cold-boot scenario from earlier, where a hundred guests each manufactured their own copy of the same libc.

The way KSM works is a background kernel thread (ksmd) that scans pages you've marked mergeable, hashes them, and when it finds two with identical content, collapses them into one write-protected shared page and frees the duplicate. The moment any owner writes, copy-on-write forks it back apart — same write-triggers-a-copy contract as MAP_PRIVATE, just reached by scanning instead of by structure. It genuinely recovers memory that snapshot-CoW can't, because it can merge pages across guests that share no common origin at all.

# KSM is opt-in and scans continuously. The knobs and the payoff live in sysfs:
$ cat /sys/kernel/mm/ksm/run            # 0 = off, 1 = scanning
$ cat /sys/kernel/mm/ksm/pages_shared   # unique pages kept (the dedup masters)
$ cat /sys/kernel/mm/ksm/pages_sharing  # duplicate pages reclaimed by merging
$ cat /sys/kernel/mm/ksm/pages_to_scan  # pages per scan cycle -- CPU vs. reclaim dial

# pages_sharing >> pages_shared means good dedup. But ksmd is spending CPU
# every cycle to find it -- and merging pages across tenants has a security cost.
$ echo 1 | sudo tee /sys/kernel/mm/ksm/run   # turn scanning on (measure before you trust it)

The trade-offs are why KSM is a deliberate choice rather than an always-on default. First, CPU: ksmd burns cycles continuously hashing and comparing pages, and you're spending that CPU whether or not there's much to merge — it's a standing tax on the host in exchange for reclaimed RAM. Second, and more serious for a multi-tenant sandbox platform, security: merging identical pages across guests opens a side channel. Because a write to a merged page triggers a copy-on-write fault that takes measurably longer than a write to an unshared page, a hostile guest can time writes to infer whether another tenant holds a page with specific content — the basis of published cross-VM attacks like memory-deduplication timing side channels and the Rowhammer-adjacent work that abused KSM to locate victim pages. That's a real reason a security-first platform may prefer structural CoW sharing (which stays within a single guest's own private mapping) over cross-tenant page merging.

Snapshot copy-on-write shares pages because they started the same; KSM shares pages by scanning until it finds ones that ended up the same. One is free and structural; the other spends CPU forever and merges across tenant boundaries you might not want merged.

Cold boot vs. snapshot CoW vs. KSM, side by side

It helps to line the three approaches up against the questions that actually matter — how the sharing arises, what it costs, and whether it crosses tenant boundaries:

• How sharing arises — Cold boot per VM: it doesn't; each guest manufactures its own private copy of identical pages. Snapshot CoW (MAP_PRIVATE): structural — every guest maps the same baked image, so identical pages are shared by construction from the first instruction. KSM: discovered — a background scanner finds already-duplicate pages and merges them after they exist.
• CPU cost — Cold boot: none for sharing (there is none), but you pay a full ~3s boot per guest. Snapshot CoW: none ongoing — sharing is a property of the mapping, the only cost is the one-time bake. KSM: a standing tax — ksmd scans and hashes continuously whether or not it finds matches.
• When the copy happens — Cold boot: never shared, so never a CoW copy; it's all private from birth. Snapshot CoW: on first write to a shared page, one 4 KiB page is copied. KSM: on first write to a merged page, the same per-page copy — plus the merge itself happened earlier during a scan.
• Tenant boundary — Cold boot: fully isolated, nothing shared. Snapshot CoW: sharing is read-only backing within each guest's private mapping; no guest can write another's page. KSM: merges pages across guests, which is exactly what creates the dedup timing side channel.
• Best fit — Cold boot: one-off or wildly heterogeneous guests. Snapshot CoW: many guests from the same template (PandaStack's whole model). KSM: heterogeneous guests with incidental page overlap where you'll accept CPU + side-channel risk for reclaimed RAM.

The pattern: snapshot copy-on-write wins precisely because PandaStack's guests share an origin. When your fleet is many restores of the same handful of templates, you don't need a scanner to discover sharing that structure already guarantees — and you avoid paying for the scan and inviting the cross-tenant side channel that comes with it.

Stacking with overcommit and on-demand paging

Copy-on-write page sharing is one lever. Real density comes from stacking it with two others that attack the same gap between configured and resident memory from different angles. The first is overcommit: the Linux kernel is willing to promise more guest memory than the host physically has, on the bet that guests are statistically idle relative to their configured ceiling. CoW sharing makes that bet dramatically safer, because a shared page isn't a promise the host has to keep separately for each guest — it's one physical page answering for all of them. The more your guests share, the less real RAM your overcommitted promises actually draw down.

The second is on-demand paging via userfaultfd. Copy-on-write keeps identical pages from being duplicated; UFFD streaming keeps untouched pages from being resident at all. Instead of loading the whole memory image up front, the agent pages it in lazily — a guest touches an address, the kernel raises a fault, and a handler fetches the surrounding 4 MiB chunk from object storage over an HTTP Range GET. Pages the guest never touches are never fetched. Zero-elision goes further: a baked header records which chunks are non-zero, so a fault in an all-zero region is zero-filled with no fetch. Put together, an idle guest's memory converges on genuinely cheap: not duplicated (copy-on-write), not resident (demand-paged), not even fetched (zero-elided). The UFFD mechanics are in /blog/userfaultfd-explained; the overcommit accounting is in /blog/overcommit-microvm-density.

And the no-warm-pool architecture is what makes all three compound cleanly. Warm idle VMs are the worst thing for the memory bet — they hold resident pages and consume the RAM you were counting on lending to active guests. Because a create is a ~179ms snapshot restore, an idle sandbox can simply be snapshotted and deleted, freeing its host RAM entirely, then recreated in under 200ms when the next request lands. Idle stops being a resident-memory tax. Fork inherits the same economics: a same-host fork lands in 400–750ms by mapping the parent's memory copy-on-write, and a cross-host fork runs 1.2–3.5s because the bytes have to travel first.

The takeaway

Sort a guest's memory by who's really paying and the density stops looking like a trick. The bulk of an idle guest's RAM is copy-on-write-shared with its template — one physical copy across every sibling, because they all restored from the same baked image and haven't written it. A thin sliver is privately dirtied, and that's the honest per-guest bill, growing exactly as fast as the guest modifies memory. The rest may not even be resident under demand paging. There's no fabricated packing ratio here, just an accounting of which pages cost anything — and the answer is 'the ones a guest actually wrote,' which for mostly-idle, mostly-reading fleets is a small corner of the whole.

The discipline is remembering where it stops: a write-heavy guest churning a large working set copies its way back to full cost, and KSM's cross-tenant merging buys extra sharing at a CPU and side-channel price a security-first platform may not want to pay. Structural copy-on-write over a shared snapshot avoids both traps — the sharing is free because the pages started identical, and it never crosses a tenant boundary. PandaStack's core is open source under Apache-2.0, so you can watch the Shared_Clean-vs-Private_Dirty split on your own hosts and size your overcommit against your own load. For the disk side of the same idea, see /blog/copy-on-write-rootfs; for the overcommit story, /blog/overcommit-microvm-density; and for the demand-paging internals, /blog/userfaultfd-explained.