Copy-on-Write and the Page-Fault Lifecycle, Step by Step

A restored microVM has gigabytes of RAM it never copied, and it doesn't notice. It resumes, runs, reads its memory, and only when it writes to a page for the first time does anything actually get duplicated — a single 4 KiB page, allocated and copied in the microseconds it takes the CPU to trap into the kernel and back. That trap is a page fault, and the specific kind that copies the page is a copy-on-write fault. This post walks the whole lifecycle: what a page table is, what the MMU does on every memory access, what a page fault actually is (and the crucial difference between a minor and a major one), how mmap with MAP_PRIVATE arranges copy-on-write, and exactly what the kernel does the instant a shared page is written. This is the kernel-CoW path — distinct from the userspace demand-paging in /blog/userfaultfd-explained, and the mechanism under the product-level fork story in /blog/copy-on-write-memory-fork-explained.

Virtual memory, page tables, and the MMU

Every address a program uses is a lie the kernel tells it — a virtual address that does not name a physical location in RAM. The translation from virtual to physical is done in hardware, on every single memory access, by the CPU's Memory Management Unit (MMU). What the MMU consults is a page table: a per-address-space tree of entries, one leaf per 4 KiB page, that maps a virtual page number to a physical page frame plus a handful of permission bits — present, writable, user/kernel, and so on.

The unit of all of this is the page, almost always 4 KiB. Memory is managed, mapped, protected, and faulted a page at a time; the page is the atom of the whole system. When the MMU translates an address, it walks the page table (or hits the TLB, a small cache of recent translations) and checks the permission bits against the access. If the mapping is present and the access is allowed, translation succeeds and the instruction proceeds — no kernel involvement at all. The interesting cases are when it doesn't succeed.

What a page fault actually is

A page fault is what happens when the MMU cannot satisfy a memory access from the page table alone. Maybe the page-table entry says not-present — there's no physical frame behind this virtual address yet. Maybe the entry is present but marked read-only and the instruction tried to write. Either way the CPU cannot complete the access, so it raises an exception: it stops the instruction mid-flight, saves the faulting address and the reason, and traps into the kernel's page-fault handler. The handler inspects why the fault happened and decides what to do — allocate a frame, copy a page, read from disk, or kill the process for a genuinely illegal access.

The critical thing is that a page fault is not an error. It's the kernel's hook for lazy, on-demand memory management. Demand paging, mmap, the page cache, swap, and copy-on-write all work by deliberately leaving page-table entries not-present or read-only so that the first access traps, and the handler does the real work exactly then — never sooner. A fault is the kernel being asked a question at the last possible moment.

Minor vs major page faults

Not all faults cost the same, and the difference is the whole performance story. A fault is minor if the kernel can resolve it without touching a slow backing store — the page is already in physical RAM (in the page cache, or a shared frame another mapping brought in) and the handler only has to wire up a page-table entry, or allocate-and-zero a fresh anonymous page. No disk, no network. A fault is major if the page's contents have to come from a slow source — read from a file on disk, paged back from swap. The handler blocks the thread until the I/O completes.

• Minor fault — cost: microseconds. The bytes are already in RAM (page cache, a shared frame, or a to-be-zeroed page); the handler just updates the page table. No I/O. This is the common case for a warm host, and it's what a copy-on-write fault is.
• Major fault — cost: much slower, dominated by I/O. The page's contents must be fetched from disk or swap before the mapping can be completed; the faulting thread blocks on that read. Same mechanism, wildly different latency — a memory access that quietly turned into a disk read.

You can watch the split directly. `perf stat` and `/proc/<pid>/stat` both count minor and major faults separately, and the ratio tells you whether a workload is paging comfortably out of RAM or thrashing against a backing store:

$ perf stat -e minor-faults,major-faults ./restore-and-run

       524,288      minor-faults      # pages wired up from RAM: cheap
            37      major-faults      # pages read from disk: the slow few

# Same counters, per-process, live:
$ ps -o min_flt,maj_flt -p $(pgrep firecracker)
 MINFL  MAJFL
611937     42        # a healthy restore: almost all faults are minor

How mmap with MAP_PRIVATE arranges copy-on-write

Copy-on-write is not a special allocator; it's a particular way of setting the page-table permission bits so the fault handler does the copy for you. The setup is an mmap of a backing file with the MAP_PRIVATE flag. MAP_PRIVATE means: reads see the file's contents, but writes are private to this mapping and are never written back to the file. To deliver that, the kernel maps the file's pages read-only and shared — every mapping of the file points at the same physical page cache frames — and defers any private divergence until a write actually occurs.

So on setup, nothing is copied. The mapping exists, its page-table entries are either not-present (fault them in from the file on first read) or present-but-read-only (shared with the page cache). A read touch is a minor fault that wires up the shared frame. A write touch trips the read-only bit and traps — and that trap is the copy-on-write fault. Here is the whole arrangement in one snippet, with the two touch cases annotated:

/* Map a 2 GiB memory image copy-on-write. Nothing is read or copied yet;
   this call is O(metadata) — it just installs a VMA + page tables. */
void *ram = mmap(NULL, 2UL << 30, PROT_READ | PROT_WRITE,
                 MAP_PRIVATE,          /* writes stay private to us */
                 fd_vm_mem, 0);

char c = ram[off];       /* READ touch -> MINOR fault. Kernel wires up the
                            shared, read-only page-cache frame straight from
                            the file. No copy. Every mapping of this file
                            shares this one physical frame. */

ram[off] = 0x42;         /* WRITE touch -> COPY-ON-WRITE fault. The frame is
                            read-only, so the MMU traps. The handler allocates
                            ONE fresh 4 KiB frame, copies the old bytes into
                            it, flips this entry to writable pointing at the
                            new frame, and re-runs the store. The shared
                            original is untouched; only WE diverged. */

The last comment is the entire mechanism. MAP_PRIVATE makes the file a shared, read-only baseline, and each writer accumulates a thin private layer of diverged pages on top of it. Everyone who maps the same file shares every page nobody has written yet — and the sharing is not a copy that happens to be identical, it's the same physical frame referenced from multiple page tables.

The copy-on-write fault, step by step

This is the heart of the post: the exact sequence from a store instruction to a writable private page. It's the same lifecycle whether the read-only sharing came from mmap MAP_PRIVATE or from a classic fork() (which marks every writable page in parent and child read-only and CoW, then duplicates only the page tables). When the first write lands on such a page:

1. A store instruction targets a virtual address. The MMU walks the page table for that page.
2. The page-table entry is present but read-only (the CoW bit is set). The write is not permitted, so the MMU raises a page-fault exception and traps into the kernel — the store does not complete.
3. The kernel's fault handler reads the faulting address and the fault reason, and recognizes a write to a copy-on-write page: this is a CoW fault, not a bug.
4. The handler allocates one fresh physical frame (a minor fault — from RAM, no I/O) and copies the 4 KiB of the shared page into it.
5. It rewrites this mapping's page-table entry to point at the new frame and flips the bit to writable — for this address space only. The original shared frame is untouched and still read-only for everyone else.
6. The handler returns. The CPU re-executes the exact store instruction that faulted. This time the entry is writable, translation succeeds, and the write lands in the private copy.
7. The program continues, none the wiser. One page diverged; every other page it shares stays shared.

That's it. The cost of a CoW fault is a trap, a page allocation, a 4 KiB memcpy, and a page-table write — all in RAM, all a minor fault, all microseconds. And it happens only for pages that are actually written. A process that reads a gigabyte of shared memory and writes ten pages of it pays for ten copies, not for the gigabyte. The reference is the default; the copy is the exception, taken lazily and locally.

Nobody decides up front to copy the memory. The kernel marks it read-only, waits for someone to write, and only then — quietly, one 4 KiB page at a time — hands that writer a private copy and lets the store retry. Sharing is the default; divergence is what you pay for.

Why this makes fork and restore O(metadata), not O(memory)

Once you see the lifecycle, the performance claim is almost a tautology. Cloning an address space — whether that's fork() duplicating a process or restore mapping a snapshot's RAM — does no data copying at clone time. It installs page tables and flips permission bits. That work is proportional to the number of mappings (metadata), not to the number of bytes of memory (data). A 2 GiB address space and a 4 GiB one clone in about the same wall-clock, because in neither case are the bytes moving. The bytes move later, page by page, only where the clone actually writes.

The naive alternative — eagerly copy every byte on clone — is O(memory): read and duplicate the whole image before the new machine can run, committing a full copy of RAM whether or not it's ever touched. Copy-on-write turns that into O(metadata) up front plus O(divergence) over the lifetime. Contrast the two:

• Eager full copy — cost: O(memory). Duplicate all N bytes before the clone runs. Seconds of I/O; a full copy of RAM committed immediately, per clone, whether or not those pages are ever written.
• Copy-on-write — cost: O(metadata) at clone time, O(pages-written) over the lifetime. Map the image read-only, share every frame, and let CoW faults copy only the pages actually written. A clone that barely diverges shares almost all of one machine's memory; ten near-identical clones cost roughly one footprint plus each one's small diff.

So clone cost stops being a function of RAM size and becomes a function of divergence. A short-lived branch that reads a lot and writes little stays almost entirely shared and costs almost nothing. A long-lived, write-heavy clone eventually copies most of its pages — but it pays gradually, exactly as it dirties them, never in one up-front lump. The O(metadata) win is on the clone; a workload that rewrites all its memory does ultimately pay for it, one minor fault at a time.

How Firecracker snapshot-restore rides this exact mechanism

A Firecracker snapshot serializes the guest's entire physical RAM to a file — vm.mem — byte for byte; a 2 GiB guest yields a 2 GiB vm.mem. To restore, PandaStack's agent doesn't read that file into a fresh buffer. It maps vm.mem with MAP_PRIVATE and resumes the vCPUs. That single decision is the whole page-fault lifecycle above, applied to a guest's memory: the guest's RAM is the snapshot file, mapped read-only and shared, diverging one page at a time as the guest runs.

• The guest resumes immediately against a lazily-mapped image. No 2 GiB read blocks the boot — pages fault in on demand as the guest touches them, each early fault a cheap minor fault.
• A guest read of an untouched page is a minor fault that wires up the shared vm.mem frame — no copy. Because the frame lives in the host page cache, a second, tenth, hundredth restore of the same snapshot on that host shares the very same physical frames the first restore paged in. Many microVMs from one template physically share read-only memory until they diverge.
• A guest write is a copy-on-write fault: one 4 KiB frame copied private to that guest. vm.mem stays pristine for every other restore and every future create. The guests diverge only where they actually write.

This is why a create that restores a baked snapshot lands in roughly 49ms of restore inside a p50 of about 179ms (p99 around 203ms), rather than the ~3s a first cold boot costs — the guest resumes against mapped memory instead of waiting on a full read. And it's why a same-host fork runs in 400–750ms: the parent's RAM is already resident, so the child's MAP_PRIVATE map shares physical frames that are already warm, and the fork is a memory map plus a metadata disk clone plus a resume. A cross-host fork takes 1.2–3.5s because the parent's resident frames aren't on the destination — vm.mem has to travel the network before the target can map it, which is a different problem solved by the userspace paging in /blog/userfaultfd-explained.

Kernel CoW vs userfaultfd: two different fault answerers

It's worth drawing the line between the mechanism in this post and the one in /blog/userfaultfd-explained, because both are about page faults but they answer them from different places:

• Kernel copy-on-write (this post) — the kernel answers the fault itself. On a MAP_PRIVATE mapping of a LOCAL vm.mem, a read fault wires up a shared page-cache frame and a write fault allocates-and-copies a private page. Everything happens inside the kernel's fault handler, from RAM, as minor faults. This is the fast path when the memory image is already on the host.
• userfaultfd paging (/blog/userfaultfd-explained) — the kernel hands the fault to a user-space handler. Used when vm.mem is NOT local: the guest faults, the kernel posts the event to PandaStack's handler, which fetches the missing chunk from object storage over the network and installs it. It changes where a faulted page comes from (remote, on demand) without changing the copy-on-write semantics the guest sees.
• The relationship — userfaultfd is the streaming source; kernel CoW is the local-sharing-and-divergence machinery. A streamed page, once installed and touched, diverges under the very same copy-on-write rules. UFFD removes the multi-GB local download; kernel CoW is what makes every local restore and fork O(metadata) in the first place.

In short: this post is the mechanism that makes memory sharing and lazy divergence work at all, on a single host; userfaultfd extends the same lazy-by-default philosophy across the network. Both leave untouched memory untouched, and both defer real work to the precise moment a page is actually needed.

The honest caveats

• Divergence is not free forever. The clone is O(metadata); a write-heavy, long-lived guest eventually takes a CoW fault for most of its pages and ends up roughly as large as a full copy. The win is largest for short-lived clones and slowly-diverging trees — measure your own write patterns before assuming the sharing holds.
• A CoW fault is cheap but not zero. Each first-write is a trap plus a 4 KiB copy plus a TLB shootdown's worth of bookkeeping. A tight loop that writes across a huge freshly-shared region pays a burst of minor faults up front. It's microseconds each, but they're real, and they show up as a spike in min_flt right after resume.
• Sharing pages is a density optimization, not the isolation model. Read-only frame sharing across guests improves memory density; it is the KVM hardware boundary and a per-microVM guest kernel that isolate one sandbox from another. Don't confuse shared unwritten pages with a shared trust domain.
• Major faults are the enemy of the fast path. Everything here stays quick because the faults are minor — resolved from RAM. If the working set spills to swap or the memory image isn't resident (cross-host), faults turn major and latency jumps. Keeping the parent's RAM resident on the fork's host is what keeps same-host forks in the 400–750ms range.

The mental model that holds up: memory is shared by reference and copied by exception. A restored or forked guest's RAM is the snapshot's memory image mapped read-only and copy-on-write; every access that only reads rides a shared frame, and only a write trips the fault that mints a private page — one 4 KiB page, one minor fault, at the exact moment of divergence. That is what turns "clone gigabytes of RAM" into a few hundred milliseconds of metadata. PandaStack's core is open source under Apache-2.0, so you can run the agent on your own KVM hosts, fork a warm guest, dirty a handful of pages in the child, and watch min_flt tick up by exactly that many — the page-fault lifecycle, made visible. For the product view of forking see /blog/copy-on-write-memory-fork-explained; for the disk twin, /blog/copy-on-write-rootfs; for the streaming variant, /blog/userfaultfd-explained.