Firecracker Dirty Page Tracking, Explained

A full Firecracker memory snapshot is conceptually simple and operationally heavy: freeze the guest, serialize its entire RAM to a file, done. For a 4 GiB guest that's a 4 GiB `vm.mem` no matter how little of that memory the guest actually changed. A diff snapshot is the clever version — it writes only the pages that changed since the previous snapshot — and the whole feature rests on one kernel capability doing the bookkeeping: KVM's dirty-page logging. This post is about that machinery: how KVM knows which guest pages got written, how Firecracker reads that knowledge, and why it's the quiet foundation under fast, small snapshots.

I'm Ajay; I built PandaStack, where every sandbox create restores a baked snapshot on demand (restore step ~49ms, p50 179ms end to end). Diff snapshots keep those snapshot artifacts small, which is exactly why understanding dirty-page tracking is worth eight minutes even if you never call the ioctl yourself.

Guest memory is a host mapping

Start from the substrate. A Firecracker guest's physical RAM is a contiguous region of host virtual memory — an anonymous mmap the VMM created and registered with KVM as a memory slot. When the guest CPU accesses a guest-physical address, the hardware's second-level address translation (the extended/nested page tables, EPT on Intel, NPT on AMD) maps it to a host-physical page inside that region. There is no guest RAM that isn't, ultimately, host memory the VMM owns. (The full layout of that region — the low-memory hole, the MMIO gap, where the kernel loads — is its own topic; see /blog/firecracker-guest-memory-layout-explained.)

Because the mapping goes through hardware page tables, the CPU can do something for free that would be expensive in software: every time the guest writes to a page, the hardware sets a dirty bit in the second-level page-table entry for that page. Reads don't set it; only writes. That per-page dirty bit is the raw signal. All dirty-page tracking is really about harvesting those bits and clearing them so you can measure the next interval.

The dirty-log bitmap and KVM_GET_DIRTY_LOG

To make dirty bits usable, the VMM asks KVM to enable dirty logging on a memory slot. Once enabled, KVM maintains a bitmap — one bit per guest page — where a set bit means "this page was written since you last checked." When Firecracker wants to know what changed, it calls the `KVM_GET_DIRTY_LOG` ioctl for the slot, and KVM copies out the current bitmap. Reading the log also resets it (KVM clears the tracked bits, or you clear them explicitly with the newer clear ioctl), so the next `KVM_GET_DIRTY_LOG` reports writes since this read, not since the beginning of time. That reset is what makes the interval well-defined.

The mechanism KVM uses to catch writes is worth naming: when dirty logging is on, KVM write-protects the guest's pages in the second-level tables. The first write to a protected page traps into KVM, KVM records the page as dirty in the bitmap and removes the write-protection so subsequent writes to that page are fast, and the guest continues. So there's a one-time trap per page per interval — cheap for a mostly-idle guest, more costly for a write-heavy one, which is the fundamental tradeoff of this style of tracking.

// Conceptual harvest loop (not real Firecracker code -- the shape, not the API).
// 1. Turn on dirty logging for the guest memory slot.
ioctl(kvm_vm_fd, KVM_SET_USER_MEMORY_REGION, &slot_with_LOG_DIRTY_PAGES)

// ... guest runs, writing some pages ...

// 2. Ask KVM which pages were written since last time.
bitmap := make([]uint64, pagesInSlot/64)      // one bit per guest page
ioctl(kvm_vm_fd, KVM_GET_DIRTY_LOG, &{ slot, &bitmap })  // resets tracked bits

// 3. Walk the bitmap; each set bit is a page to serialize into the diff.
for i, word := range bitmap {
    for b := 0; b < 64; b++ {
        if word&(1<<b) != 0 {
            page := (i*64 + b)
            writePageToDiffSnapshot(page)   // ONLY changed pages hit the file
        }
    }
}

From bitmap to diff snapshot

Now the payoff. A diff snapshot is taken relative to a base — usually a full snapshot you already have. To create it, Firecracker pauses the guest, reads the dirty-log bitmap for the memory slots, and writes out only the pages whose bits are set, along with the updated device and vCPU state. The result is a small `.mem` diff plus a state file: it captures what changed, and it's meaningless without the base it diffs against.

1. Take a full snapshot as the base — this serializes all guest RAM once and establishes the reference point.
2. Enable dirty logging so KVM begins tracking writes from here forward.
3. Let the guest run and do work — it dirties some subset of its pages.
4. Pause and call KVM_GET_DIRTY_LOG to harvest the bitmap of changed pages (and reset it for the next interval).
5. Write only the dirtied pages into the diff .mem, plus the current device/vCPU state file.
6. To reconstruct: layer the diff over the base — apply the changed pages on top of the base's memory image, then restore state.

This is why diff snapshots are small for the workloads that matter. A sandbox that boots, waits, and services one request touches a small fraction of its RAM; the diff is a small fraction of `vm.mem`, not the whole thing. The tradeoff is honest: a diff is worthless without its base, so you're trading snapshot size for a dependency chain you have to keep intact. The full mechanics of that tradeoff — chains, bases, when to re-baseline — are covered in /blog/firecracker-diff-snapshots-explained; this post is specifically about the dirty-tracking that makes it possible.

The dirty-ring evolution

The classic bitmap has a scaling wrinkle: it's proportional to guest RAM, and harvesting it means the VMM scans the whole bitmap even when only a few pages changed. For large guests and write-heavy workloads (the extreme case being live migration, where you harvest continuously), that's a lot of scanning. KVM's answer is the dirty ring (`KVM_CAP_DIRTY_LOG_RING`): instead of a full bitmap, each vCPU gets a ring buffer into which KVM pushes an entry per newly-dirtied page. The VMM consumes entries from the ring — work proportional to the number of dirtied pages, not to total RAM — and resets them to hand the slots back.

For a sandbox platform, the relevant fact is not which ioctl fires but what dirty-page tracking buys you: memory snapshots that cost proportional to change, not to size. That's what keeps diff artifacts small enough to store cheaply and distribute quickly, and small artifacts are part of why restore-on-create stays fast. On PandaStack every create restores a baked snapshot (restore step ~49ms, p50 179ms), and streaming can page memory in on demand rather than downloading it whole — both downstream of the same principle the dirty log establishes: track what changed, move only that.

If you want the layers around this: the base mmap-and-copy-on-write story is in /blog/firecracker-memory-file-mmap-explained, the diff-snapshot chain mechanics are in /blog/firecracker-diff-snapshots-explained, and the copy-on-write side of restore is in /blog/copy-on-write-memory-fork-explained.