Firecracker Diff Snapshots Explained

Firecracker can take two kinds of memory snapshot, and the distinction is worth getting exactly right because it decides how much you write, how fast, and what you need on hand to restore. A full snapshot writes the guest's entire physical RAM — every page, touched or not. A diff snapshot writes only the pages that were dirtied since the last snapshot, tracked by asking KVM which pages the guest has written to. The mental model that survives contact with the details: a full snapshot is a photograph of everything; a diff snapshot is a sticky note that says 'only the third drawer moved.' The note is tiny and quick to write, but it's useless without the photograph it refers back to. This post covers how diff snapshots actually work — enabling dirty-page tracking, the KVM dirty log, capturing a sparse memory file, and the layering you have to do on restore — plus the honest trade-offs and where diffs earn their keep. It's a Firecracker internals post; PandaStack shows up at the end because our create path deliberately uses the other primitive.

Full snapshots vs diff snapshots

Every Firecracker snapshot is fundamentally two things on disk: a state file (vCPU registers, the in-kernel interrupt controller, the clock, every virtio device's configuration and queue pointers) and a memory file (the guest's physical RAM). The state file is small and always written in full. The snapshot type only governs the memory file — specifically, which of the guest's pages get written into it.

• Full snapshot — writes the guest's complete physical RAM, every page. Self-contained: you can restore it standalone, on any host, with no dependency on a prior snapshot. This is what you bake a template or a golden image from.
• Diff snapshot — writes only the guest pages dirtied since the previous snapshot, as a sparse memory file. Much smaller and faster to write, but not restorable on its own — you reconstruct full memory state by layering the diff on top of its base.

That's the entire conceptual split. A diff isn't a smaller full snapshot; it's an incomplete one by design — it deliberately omits every page that hasn't changed, on the assumption you still have the base that holds those pages. Whether that assumption is a feature or a foot-gun depends on your workflow, which is the whole point of the trade-offs section below.

How dirty-page tracking works

For a diff snapshot to write 'only the changed pages,' Firecracker has to know which pages changed. It doesn't scan RAM comparing bytes — that would cost as much as writing a full snapshot. Instead it leans on hardware and KVM. When you enable dirty-page tracking, KVM sets up write-protection bookkeeping so that the first write to any guest page is recorded in a dirty bitmap: one bit per guest page, flipped to 1 the moment that page is written.

Mechanically, KVM exposes this through the KVM_GET_DIRTY_LOG ioctl. When Firecracker is about to take a diff snapshot, it asks KVM for the dirty log — the bitmap of which pages have been written since tracking started (or since the last time the log was retrieved) — and writes exactly those pages into the sparse diff memory file, at their correct guest-physical offsets. Pages whose dirty bit is still 0 are simply not written; on restore they'll come from the base. The runtime cost is the write-protection fault handling: the first write to a clean page traps into KVM so it can mark the page dirty and let the write proceed. It's a small, per-page-first-touch overhead, not a per-write one — but it isn't free, which is why you enable it deliberately rather than always-on.

Enabling tracking is a machine-configuration flag set before the guest starts running (and it is also implied whenever a snapshot is loaded so the resumed VM can itself be diff-snapshotted). You turn it on with track_dirty_pages in the machine config; from that point KVM maintains the dirty bitmap for you.

The API calls: base then diff

Firecracker drives all of this over its HTTP API socket. The sequence is: enable dirty tracking in the machine config, boot and run the workload, pause and take a full base snapshot, let the workload run some more, then pause again and take a diff snapshot that captures only what changed since the base. You must pause the vCPUs before each snapshot — you can't serialize a moving target.

# --- Before boot: enable dirty-page tracking in the machine config ---
curl --unix-socket /run/fc.sock -X PUT 'http://localhost/machine-config' \
  -d '{
        "vcpu_count": 2,
        "mem_size_mib": 2048,
        "track_dirty_pages": true
      }'

# ... configure boot source + drives, then start the guest, let it run ...

# --- 1. Full base snapshot: pause, then write the ENTIRE memory image ---
curl --unix-socket /run/fc.sock -X PATCH 'http://localhost/vm' \
  -d '{"state": "Paused"}'

curl --unix-socket /run/fc.sock -X PUT 'http://localhost/snapshot/create' \
  -d '{
        "snapshot_type": "Full",
        "snapshot_path": "/snap/base.state",
        "mem_file_path": "/snap/base.mem"
      }'

# Resume and keep working. The guest dirties some pages.
curl --unix-socket /run/fc.sock -X PATCH 'http://localhost/vm' \
  -d '{"state": "Resumed"}'

# ... time passes, workload mutates a fraction of RAM ...

# --- 2. Diff snapshot: pause, then write ONLY the dirtied pages ---
curl --unix-socket /run/fc.sock -X PATCH 'http://localhost/vm' \
  -d '{"state": "Paused"}'

curl --unix-socket /run/fc.sock -X PUT 'http://localhost/snapshot/create' \
  -d '{
        "snapshot_type": "Diff",
        "snapshot_path": "/snap/diff-01.state",
        "mem_file_path": "/snap/diff-01.mem"
      }'
# diff-01.mem is sparse: it holds only the pages the guest wrote since base.mem.

Take that diff repeatedly and you build a chain: base, diff-01, diff-02, and so on, each capturing what changed since the one before it. That's incremental checkpointing — you re-serialize only the delta each time instead of the whole multi-gigabyte image, so periodic saves stay cheap even for a long-running guest.

Restoring a diff: layering base + diff

Here's where the sticky-note analogy pays off. A diff memory file is sparse — it has real bytes only at the offsets of pages that changed, and holes everywhere else. Restoring it means presenting Firecracker with a complete memory image where every page resolves: dirtied pages come from the diff, and every untouched page comes from the base. You layer the diff on top of the base so that reads of unchanged offsets fall through to the base image and reads of changed offsets hit the diff.

There are two common ways to do that layering, and they're both just 'make the base bytes visible everywhere the diff has a hole':

1. Merge into a full image. Copy the base memory file, then overlay each diff in order — write the diff's populated pages over the base at their offsets. If you have a chain (base + diff-01 + diff-02), apply the diffs oldest-to-newest so later writes win. The result is a self-contained full memory file you can restore like any other.
2. Layer at restore time with a fault handler. Instead of pre-merging, back the guest's memory with a userfaultfd handler that, on a page fault, serves the page from the newest diff that has it and falls back to the base otherwise. No merge step, but you carry the base and every diff in the chain for the life of the restore.

Either way, the load-time API call is the ordinary /snapshot/load — you point it at the reconstructed state and memory. The catch that never goes away is the dependency: a diff is only as good as the exact base it was taken against. Lose the base, or point at the wrong generation, and the diff is unrecoverable — the holes have nothing to fall through to. A full snapshot has no such dependency, which is exactly why it's the right primitive for anything you need to restore anywhere, anytime.

The trade-offs, side by side

Neither type is 'better' — they optimize for different things. The comparison, on the axes that actually matter:

• Snapshot size — Full: the whole guest RAM image, so a 2 GiB guest writes ~2 GiB. Diff: only dirtied pages, so it scales with how much the workload changed, not with total RAM — often dramatically smaller for a short interval.
• Write speed — Full: has to serialize the entire memory image every time. Diff: writes far fewer pages, so a periodic save is much cheaper — the main reason diffs exist.
• Restore — Full: self-contained, load and go, no dependencies. Diff: must reconstruct by layering onto its base (and any intermediate diffs), so restore is more work and needs the whole chain present.
• Runtime cost — Full: no dirty-tracking overhead required. Diff: needs track_dirty_pages on, which adds a small write-protection cost to the running guest.
• Portability — Full: restore on any host with just the two files. Diff: not portable alone; you must ship the base and the full chain alongside it.
• Best use case — Full: baked templates, golden images, anything restored constantly and independently. Diff: incremental checkpointing, periodic saves, snapshot chains, and reducing storage for many near-identical checkpoints.

When diff snapshots earn their keep

Diff snapshots are a specialist tool. They pay off precisely when you're taking many snapshots of the same long-lived guest and the delta between them is small relative to total RAM.

Incremental checkpointing of long-running VMs

A guest that runs for hours and needs periodic save points is the textbook case. A full snapshot every few minutes would re-serialize gigabytes of mostly-unchanged RAM each time — wasteful in both time and storage. With a full base plus periodic diffs, each checkpoint captures only what moved since the last one. You get frequent, cheap save points, and you can fold the chain down to a fresh full base whenever the accumulated diffs get long enough that restore layering is no longer worth it.

Reducing snapshot storage and migration bandwidth

If you're storing many checkpoints of a slowly-evolving guest, diffs collapse the storage bill: one full base plus a series of small deltas instead of many redundant full images. The same math applies to live migration and pre-copy — you can send a full base while the guest keeps running, then send a small diff of the pages that changed during the transfer, minimizing the final stop-the-world window. In all of these, the win is the same: never re-ship a page that didn't change.

How this relates to copy-on-write and UFFD

Diff snapshots are about what you write to storage. Copy-on-write memory and userfaultfd streaming are about what you read on restore. They attack the same underlying quantity from opposite ends — both are ways to avoid touching pages that haven't changed or aren't needed — which is why they compose cleanly rather than competing.

When Firecracker restores from a local memory file, it maps that file with MAP_PRIVATE: the mapping is lazy and copy-on-write at page granularity, so a page is faulted in only when the guest touches it and a write makes a private copy of just that 4 KiB page. That's copy-on-write on the read side — you pay only for the working set the guest actually touches, not the whole image. Diff snapshots are copy-on-write's mirror image on the write side: dirty-page tracking is literally the record of which pages diverged from the base, and the diff serializes exactly that divergence. Same concept — 'track what changed, share what didn't' — one applied to restore, one to capture.

UFFD memory streaming extends the read side across the network: instead of a local file, the agent backs guest memory with a userfaultfd and fetches faulted pages on demand from object storage, so a restore can begin before a multi-gigabyte image has arrived. You could imagine composing all three — a diff-based checkpoint chain, restored through a fault handler that streams base pages from storage and overlays diff pages on top. In practice they're complementary techniques for the same goal: keep the bytes you move, store, and materialize as close as possible to just the working set.

Where PandaStack lands: full snapshots on the create path

PandaStack's model makes a deliberate choice to use the other primitive on the hot path. Every sandbox create restores a baked template snapshot on demand — there's no warm pool of idle VMs — and that restore lands the create at a 179ms p50 (roughly 203ms p99), with the /snapshot/load step itself around 49ms. For that path, a self-contained full snapshot is exactly right: every create restores the same clean baked baseline independently, on whatever agent the scheduler picks, with no base-plus-chain to assemble. A diff snapshot would add a dependency and a layering step to the single operation we most want to keep dependency-free.

The place PandaStack leans hardest on 'only move what changed' is the read side — MAP_PRIVATE copy-on-write for local restore, UFFD streaming for cross-host restore, and reflink copy-on-write for the rootfs, where a same-host fork runs in roughly 400–750ms by sharing the parent's memory pages and disk extents until the child writes. Diff snapshots and those copy-on-write techniques aren't in tension; they're the write-side and read-side halves of the same idea. If your workload is long-running guests that need cheap periodic checkpoints, diffs are the tool. If it's stamping out fresh sandboxes from a clean baseline as fast as possible, a full baked snapshot restored copy-on-write is.

The summary

A full Firecracker snapshot writes the guest's entire RAM and restores standalone; a diff snapshot writes only the pages dirtied since its base — tracked via KVM's dirty-page log, enabled with track_dirty_pages — and restores by layering onto that base. Diffs are much smaller and faster to write, which makes them the right primitive for incremental checkpointing, snapshot chains, reduced storage, and migration; the cost is a small dirty-tracking overhead at runtime and a hard dependency on the base (and every intermediate diff) at restore. Copy-on-write memory and UFFD streaming are the same 'only touch what changed' idea applied to the read side, so they compose with diffs rather than replacing them. PandaStack's create path deliberately uses self-contained full snapshots — restored copy-on-write for a 179ms p50 create with no dependency chain to assemble — because a create should never need to hunt down a base.

The core of PandaStack is open source under Apache-2.0, so you can run the control-plane API and per-host agent on your own Linux KVM hosts and watch the restore timings yourself. For how a full snapshot's memory is mapped and paged in, start with /blog/firecracker-memory-snapshots; for the create pipeline end to end, /blog/snapshot-restore-boot-path; for the streaming side, /blog/userfaultfd-explained.