How Copy-on-Write Page Tables Work (and Why VM Fork Is Fast)

Copy-on-write sounds like it's about copying memory. It isn't — it's about permission bits in a page table. The whole trick lives in one leaf entry of a tree the CPU walks on every memory access: flip that entry from writable to read-only, point two address spaces at the same physical page, and the hardware will trap the instant either one tries to write. That trap is the hook. Everything else — the fresh page, the memcpy, the remap — happens in the handler, only for pages actually written. This post goes one level below the fork and page-fault-lifecycle explainers (/blog/copy-on-write-memory-fork-explained and /blog/firecracker-cow-page-fault-lifecycle): not just "the page is shared until you write it," but what the page-table entry actually holds, how fork rewrites it in both parent and child, what the refcount is for, why the TLB shootdown is the sneaky cost, and how a 2 MiB hugepage changes the arithmetic. This is the machinery that makes a PandaStack microVM fork in 400-750ms.

The page table is a tree the CPU walks on every access

A virtual address doesn't name a byte of RAM — it names a path through a tree. On x86-64 the MMU splits a 48-bit virtual address into four 9-bit indices plus a 12-bit page offset, and walks four levels of tables (PML4 → PDPT → PD → PT) to reach a leaf entry, the page-table entry (PTE), that finally names a physical page frame. Each level is itself a page of 512 entries; the CPU indexes one entry per level, following physical pointers down until it hits the PTE. The bottom 12 bits — the offset — are added to the frame's physical base to land on the exact byte. This is done in hardware, on every load and store, which is why the translation must be fast and why the TLB (a cache of recent virtual-page → physical-frame results) exists to skip the walk when it can.

The reason the tree matters for copy-on-write is that sharing memory between two address spaces means their two page-table trees have leaf entries pointing at the same physical frame. Nothing else needs to be identical — the parent and child can walk completely different upper-level tables and still arrive at the same frame. So "sharing a page" is precisely: two PTEs, in two trees, holding the same physical frame number. And "diverging a page" is: rewriting one of those two PTEs to point somewhere new.

The PTE: a frame number and a fistful of bits

A PTE is a 64-bit word, but only some of it is the physical frame number. The rest are flag bits the MMU reads on every access and the kernel reads in its fault handler. The ones that matter for copy-on-write are a small set:

• Present (P) — is there a frame behind this entry at all? If clear, any access traps: the page must be faulted in (from a file, from zero, from swap). This is the bit that makes demand paging and lazy mmap work.
• Writable (R/W) — may this page be written? If clear, a store traps even though the page is present and readable. This single bit is the entire copy-on-write mechanism: shared CoW pages are present + readable but NOT writable.
• Dirty (D) — set by hardware the first time a page is written. The kernel uses it to know which pages need writing back (for file mappings) and, more relevant here, it's meaningless on a read-only page because a write can never reach it — the CoW fault fires first.
• Accessed (A) — set by hardware on any access; drives page-reclaim aging. Orthogonal to CoW but part of the same word.
• User/Supervisor (U/S) — may user-mode touch this, or kernel only? A guardrail the MMU checks alongside the others.

The elegant part is that the kernel doesn't need a separate "copy-on-write" bit. It reuses the writable bit. A CoW page is simply one that is present and readable but has its writable bit cleared, while the kernel privately remembers (in its own page metadata, not the PTE) that this read-only-ness means "copy me on write" rather than "this page is genuinely read-only forever." When the write fault arrives, the handler distinguishes the two cases by consulting the VMA's permissions: if the mapping was created writable but the PTE is read-only, it's a CoW fault; if the mapping itself is read-only, it's a real protection violation and the process gets a SIGSEGV.

What fork actually does to the page tables

When a process forks — or when the kernel maps a snapshot MAP_PRIVATE, which is the same maneuver one layer up — it does not copy the physical pages. It walks the parent's page tables and, for every writable private page, does three things: clear the writable bit in the parent's PTE, install a PTE in the child pointing at the same physical frame with the writable bit also clear, and increment that frame's reference count. After the walk, both trees describe the same physical memory, both mark every shared page read-only, and the only new memory allocated is the page-table pages themselves — metadata, not data.

That refcount is doing real work. A physical frame now has two (or ten, after a fan-out of forks) PTEs referencing it. The kernel keeps a per-frame count so it knows when a CoW fault can skip the copy entirely: if a write faults on a page whose refcount is 1, nobody else is sharing it anymore, so the kernel just flips the writable bit back on in place — no new page, no memcpy. If the refcount is greater than 1, the page is genuinely shared and must be copied. This is why the last surviving branch of a fork tree, once its siblings die, stops paying CoW costs on the pages it inherited: the refcounts have dropped back to 1 and its writes become free re-enable-in-place operations.

1. Walk the parent's writable private PTEs. For each one, clear the R/W bit so the next write will trap.
2. Install a matching PTE in the child's tree pointing at the SAME physical frame, also with R/W clear.
3. Increment that frame's reference count — the kernel now knows N mappings share it.
4. Duplicate only the page-table pages (a few KiB per GiB of address space). No guest data is touched.
5. Both address spaces resume, sharing every unwritten page by reference. Divergence is deferred entirely to write faults.

The write-protection fault, PTE by PTE

Now the write lands. A store instruction targets a virtual address; the MMU walks the tree, reaches the leaf PTE, sees Present set but R/W clear, and refuses the write. It raises a page-fault exception carrying the faulting address and an error code whose bits say present + write + user. The CPU freezes the instruction and traps into the kernel's fault handler. From there:

1. The handler reads the faulting address and error code, walks to the PTE, and sees present-but-read-only. It checks the VMA: the mapping was created writable, so this is a copy-on-write fault, not a violation.
2. It reads the frame's reference count. If the count is 1, it takes the shortcut: set the R/W bit back on in this PTE, flush this page's TLB entry, done — no copy. If the count is greater than 1, it proceeds to copy.
3. It allocates one fresh physical frame (a minor fault — from the free-page pool, no I/O) and memcpy's the 4 KiB of the shared page into it.
4. It rewrites THIS address space's PTE to point at the new frame with the R/W bit set. The other sharers' PTEs are untouched — they still point at the original, still read-only.
5. It decrements the original frame's reference count (one fewer sharer) and flushes the stale TLB entry for this page so the CPU won't keep using the old read-only translation.
6. It returns. The CPU re-executes the identical store. This time the PTE says writable, translation succeeds, the byte lands in the private copy, and hardware sets the Dirty bit.

The total cost is a trap, one page allocation, one 4 KiB memcpy, two or three PTE writes, and a TLB flush — all in RAM, all microseconds, all for exactly one page. Read a gigabyte of shared memory and you pay nothing. Write ten pages of it and you pay for ten frames. The reference is the default and the copy is the exception, taken lazily and locally, one PTE at a time.

The kernel is a very patient landlord: everyone shares the same page until someone scribbles on the wall, then they get their own copy and the bill. The rent is one 4 KiB page, due only on the first write, and never a cent before.

The hidden cost: TLB shootdowns

There's a cost the tidy story skips. The TLB caches virtual→physical translations per CPU core so the MMU can skip the four-level walk. When the kernel rewrites a PTE — clearing the writable bit on fork, or repointing it to a new frame on a CoW copy — every core that has the old translation cached is now wrong. Flushing the local core's TLB is cheap. The problem is the other cores: a multi-vCPU guest, or a host running many threads over the same address space, may have the stale entry cached on several cores at once. There's no hardware broadcast to invalidate them, so the kernel sends an inter-processor interrupt to each affected core telling it to flush the entry itself. That cross-core dance is the TLB shootdown.

A shootdown is an IPI plus a synchronous wait for the target cores to acknowledge — hundreds of nanoseconds to low microseconds, dwarfing the memcpy for a single 4 KiB page. It's usually invisible, but it's the reason a workload that write-faults a huge freshly-shared region in a tight loop can spend more time coordinating TLBs across cores than actually copying bytes. For a forked microVM it mostly shows up as a brief flurry right after resume, as the guest first-writes its hot pages and each write-fault repoints a PTE and shoots down its stale translation. It settles quickly, because once a page is private and writable no further fault or shootdown touches it.

Hugepages change the granularity of everything

Everything above assumed a 4 KiB page. Swap in a 2 MiB hugepage and the same machinery runs at a coarser grain, with a trade-off that cuts both ways. A hugepage collapses what would have been 512 leaf PTEs into one entry at the page-directory level (the PD entry points straight at a 2 MiB frame instead of at a page table). That means 512× fewer PTEs to walk, 512× fewer TLB entries to cover the same memory, and far fewer faults to page a region in — a real win for restore, where fewer faults means the guest reaches a running state sooner. But the copy-on-write unit is now the whole 2 MiB page. Write a single byte to a shared hugepage and the fault handler copies all 2 MiB, because that's the smallest thing the PTE can describe.

• 4 KiB pages — fine-grained CoW. A one-byte write copies exactly 4 KiB. Divergence tracks precisely what the branch actually dirtied, so near-identical forks share almost everything. The cost is more PTEs, more TLB pressure, and more (individually cheap) faults to page a region in.
• 2 MiB hugepages — coarse-grained CoW. A one-byte write copies the whole 2 MiB frame — 512× the data for one byte touched. But there are 512× fewer PTEs and TLB entries, and paging a region in takes far fewer faults. Great for read-mostly or densely-written guests; wasteful for a branch that dirties one byte in each of many otherwise-shared hugepages.

So hugepages trade CoW precision for translation efficiency. For a workload that touches memory in big contiguous runs, the coarse granularity barely costs anything and the fault/TLB savings are pure win. For a fork tree where each branch scribbles sparsely across a large shared region, 2 MiB CoW amplifies every write into a 2 MiB copy and the sharing you hoped for evaporates. PandaStack treats hugepage-ness as a snapshot property baked at template time, precisely because it changes the restore and divergence economics — and because a hugepage snapshot restores only through the userspace-fault (UFFD) path, which is a separate story in /blog/userfaultfd-explained.

How this maps onto Firecracker guest memory

A Firecracker snapshot serializes the guest's entire physical RAM to a file, vm.mem. To restore, PandaStack's agent maps that file MAP_PRIVATE and resumes the vCPUs — which is exactly the fork maneuver from above, applied to a guest's memory. MAP_PRIVATE installs PTEs that are present-and-read-only, pointing straight at the vm.mem page-cache frames, and defers every private copy to the first write. The guest resumes against a lazily-mapped image: no gigabyte read blocks the boot, and each early access is a cheap minor fault that either wires up a shared frame (read) or mints a private one (write).

• A guest read of an untouched page is a minor fault that installs a PTE pointing at the shared vm.mem frame — no copy, refcount bumped. A second, tenth, hundredth restore of the same snapshot on the host shares those very frames out of the page cache, so many microVMs from one template physically share read-only RAM until they diverge.
• A guest write is a write-protection fault: the handler copies one page (or one 2 MiB hugepage) private to that guest and repoints its PTE. vm.mem stays pristine for every other restore and every future create.
• Forking a running guest snapshots its live memory and restores it into the child with the same MAP_PRIVATE map. The child's PTEs point at the parent's resident frames, refcounts bump, and the child diverges one page at a time exactly as it dirties memory.

This is why a create that restores a baked snapshot lands in roughly 49ms of restore inside a ~179ms p50 (p99 ~203ms) instead of the ~3s a first cold boot costs — the guest resumes against mapped memory, not a full read. It's why a same-host fork runs in 400-750ms: the parent's frames are already resident, so the child's PTEs share warm physical memory and the fork is a page-table install plus a metadata disk clone plus a resume. And it's why a cross-host fork takes 1.2-3.5s: the parent's frames aren't on the destination, so vm.mem must travel the network before any PTE can point at it. Because the sharing is O(page-table metadata) rather than O(bytes), the ceiling on a host is memory and CPU, not fork mechanics — one agent pre-allocates 16,384 /30 subnets precisely because the networking, not the memory cloning, would otherwise be the scarce resource.

Watching the divergence yourself

You don't have to take the mechanism on faith. The kernel exposes per-mapping page accounting in /proc/<pid>/smaps, and the columns that matter for CoW are Shared_* versus Private_Dirty. Right after a fork or restore, almost everything is shared; as the guest writes, pages migrate from the shared columns into Private_Dirty — one CoW fault per page moved.

# Per-mapping CoW accounting for a Firecracker process. Rss = resident,
# Shared_* = pages still shared (unwritten, refcount > 1),
# Private_Dirty = pages this process copied on write (its divergence).
$ pid=$(pgrep -f 'firecracker.*vm.mem')
$ grep -E 'Rss|Shared_|Private_Dirty' /proc/$pid/smaps | \
    awk '{s[$1]+=$2} END{for(k in s) printf "%-16s %8d kB\n", k, s[k]}'

Rss:              2103072 kB   # guest is "using" ~2 GiB of RAM ...
Shared_Clean:     1984256 kB   # ... but almost all of it is SHARED, unwritten
Private_Dirty:      98304 kB   # only ~96 MiB actually diverged (got copied)

# Cause a divergence and watch Private_Dirty climb by exactly what you dirtied:
$ dd if=/dev/urandom of=/dev/null bs=1M count=1 2>/dev/null  # (inside guest)
#   each freshly-written page: one write-protection fault -> one private copy

The observable payoff shows up through the SDK too. Fork a warm sandbox, dirty a little in the child, and the child's memory footprint is the parent's shared baseline plus only what it wrote — the losing branches of a best-of-N fan-out barely cost anything because copy-on-write meant they never allocated the untouched pages they inherited.

from pandastack import Sandbox

# Warm parent: expensive state we want many branches to share for free.
parent = Sandbox.create(template="base", ttl_seconds=3600)
parent.exec("git clone --depth 1 https://github.com/acme/service /work")
parent.exec("cd /work && npm ci")          # warm caches now live in guest RAM

# Fork into branches. Each child MAP_PRIVATE-maps the parent's memory:
# its page tables point at the parent's resident frames (refcount++),
# and it copies a page ONLY when it writes one. 400-750ms same-host.
for strategy in ["retry-backoff", "null-guard", "widen-types"]:
    child = parent.fork()                  # shares parent RAM by reference (CoW)
    child.exec(f"cd /work && ./fix --strategy={strategy}")
    r = child.exec("cd /work && npm test", timeout_seconds=300)
    # A losing branch dirtied a handful of pages, so it cost a handful of
    # 4 KiB copies -- not a copy of the whole 2 GiB it inherited.
    if r.exit_code != 0:
        child.kill()

parent.kill()  # parent still holds the pristine shared baseline

The honest caveats

• Divergence is O(pages-written) over the lifetime. The clone is O(page-table metadata); a long-lived, write-heavy guest eventually takes a CoW fault for most of its pages and ends up roughly as large as a full copy. The win is on the clone and on short-lived, slowly-diverging branches — measure your own write patterns.
• The first write-burst is not free. Each first-write is a trap, a copy, a PTE rewrite, and a TLB shootdown across the vCPUs sharing the address space. It's microseconds each, but a tight loop dirtying a large freshly-shared region pays a visible spike of minor faults and IPIs right after resume.
• Hugepages amplify sparse writes. A 2 MiB CoW granularity copies 2 MiB for a one-byte write. Fewer PTEs and faults, but wasteful for branches that scatter tiny writes across many otherwise-shared hugepages. It's a per-workload trade, baked at snapshot time, not a free upgrade.
• Cheap sharing needs resident frames. Same-host forks (400-750ms) share the parent's already-resident memory; cross-host forks (1.2-3.5s) must move vm.mem first, because a PTE can only point at a frame that's actually present on the host. Keeping a fork on the parent's host is what keeps the page tables pointing at warm memory.

The mental model that holds up: copy-on-write is a permission bit, sharing is a refcount, and divergence is a repointed leaf PTE. A restored or forked guest's RAM is the snapshot image mapped read-only; every read rides a shared frame and only a write trips the write-protection fault that mints a private page — one 4 KiB (or 2 MiB) page, one PTE rewrite, one TLB flush, at the exact moment of divergence. That is what turns "clone gigabytes of RAM" into a few hundred milliseconds of metadata. PandaStack's core is open source under Apache-2.0, so you can run the agent on your own KVM hosts, fork a warm guest, and watch Private_Dirty climb by exactly the pages you dirtied. For the product view of forking see /blog/copy-on-write-memory-fork-explained; for the fault lifecycle, /blog/firecracker-cow-page-fault-lifecycle; for the disk twin, /blog/copy-on-write-rootfs; for the streaming variant, /blog/userfaultfd-explained.