How Firecracker Restores Guest Memory: mmap & MAP_PRIVATE

When Firecracker takes a snapshot, it writes the guest's entire physical RAM to a memory file — conventionally vm.mem — byte for byte. A 2 GiB guest produces a 2 GiB file. The obvious way to bring that machine back would be to read the whole file into memory and then run. Firecracker does not do that. On restore it memory-maps vm.mem with mmap and the MAP_PRIVATE flag, loads the small device-state file, and unpauses the vCPUs. No bulk read happens. The mapping just tells the kernel that the guest's physical RAM is backed by this file — and then every page faults in lazily, the first time the guest actually touches it. The restore, in effect, lies to the guest: all of its RAM is 'there,' none of it is loaded yet. This post is the kernel mechanics of that lie — what a memory file is, why MAP_PRIVATE gives you copy-on-write and page-cache sharing, why the map is O(1) instead of O(bytes), what the first touch of each page actually costs, and how this file-backed path relates to the userfaultfd path that streams pages from remote storage.

The snapshot memory file: guest RAM on disk

A microVM's guest RAM is, at the host level, just a big anonymous memory region that Firecracker hands to KVM as the guest's physical address space. When you snapshot a paused guest, Firecracker serializes two artifacts: a tiny state file (vCPU registers, the in-kernel interrupt controller, the clock, virtio device and queue state) and the memory file — a flat dump of that entire guest-physical RAM region. There is no compression or structure to it: offset 0 in the file is guest-physical address 0, offset N is address N. It is the guest's memory, laid out linearly on disk.

That flat layout is what makes the restore trick possible. Because the file is a byte-for-byte image of the address space, the host kernel can map file offset X directly to guest-physical page X with no translation table and no parsing. The state file says 'the guest was here, registers like this, devices like that'; the memory file says 'and this is every byte of RAM it was holding.' Restore is: map the second, load the first, resume.

Restore maps the file, it doesn't read it

The naive mental model — 'load a 2 GiB snapshot' meaning 'read 2 GiB off disk, then run' — is wrong, and it's worth being precise about why. If restore read the whole memory file up front, restore latency would scale linearly with guest RAM: a 4 GiB guest would take twice as long to restore as a 2 GiB guest, purely in I/O, before executing a single guest instruction. That is not what the measurements show, because that is not what happens.

Instead, Firecracker calls mmap() on the memory file. mmap establishes a virtual mapping — it edits page tables and virtual-memory bookkeeping — and returns. It does not touch the file's contents. After the call, the guest's physical RAM region is 'backed by' vm.mem, meaning the kernel knows where each page's bytes live on disk, but not a single byte has been read into RAM yet. The mapping is O(1) in the size of the region: mapping a 2 GiB file and a 200 GiB file cost about the same, because in both cases you're setting up metadata, not moving data.

/* Restore: map the whole guest RAM image. This does NOT read the file.
 * It edits page tables and returns. O(1) in the size of the region. */
void *guest_ram = mmap(
    NULL,
    guest_ram_size,              /* e.g. 2 GiB — no bytes are read here */
    PROT_READ | PROT_WRITE,
    MAP_PRIVATE,                 /* copy-on-write: our writes never hit the file */
    fd_vm_mem,                   /* the snapshot memory file */
    0);
/* mmap returned. Zero pages are resident. The guest "has" 2 GiB of RAM,
 * but every page is a promise the kernel will keep on first touch. */

resume_vcpus();                  /* guest runs against a memory image it hasn't loaded */

/* Later, deep inside guest execution: */
unsigned char b = guest_ram[0x1a3f000];  /* FIRST touch of this page ->
    minor page fault: kernel reads the 4 KiB page from vm.mem (or the page
    cache if it's already resident), wires it into our page tables, and the
    load retries transparently. The guest never knows it stalled. */

So the guest resumes against a memory image that is, physically, almost entirely not in RAM. It runs anyway. The moment it dereferences a pointer into a page that isn't resident, the CPU's MMU raises a page fault, traps into the kernel, the kernel resolves it from the mapped file, and the faulting instruction retries. From the guest's point of view nothing happened — it's just accessing its own RAM. From the host's point of view, the memory materialized on demand.

Lazy page-in: what the first touch of a page costs

This on-demand materialization is called lazy page-in or demand paging, and it's the whole reason restore is fast. You pay only for the pages the guest actually touches between resume and 'ready' — its working set — not for the full image. A guest with 2 GiB of RAM that reaches a usable state by touching a few hundred megabytes pays for a few hundred megabytes. The rest of vm.mem is never read from disk.

The cost is not free, though, and it's important to be honest about where it lands: on the first access to each page. That first touch is a page fault. There are two flavors, and the distinction matters:

• Minor page fault — the page's contents are already sitting in the host's page cache (because another VM restored from the same file, or this VM touched a neighboring page and the kernel read ahead). The kernel just wires the already-resident page into this mapping's page tables. Cost: a trap and some bookkeeping, no disk I/O. Sub-microsecond.
• Major page fault — the page is not in the page cache and has to be read from disk (or, on the streaming path, from the network). Cost: a real storage round trip on the faulting vCPU before it can continue. This is the expensive case, and it's the one lazy restore trades a bulk up-front read for.

The trade is deliberate. Bulk-reading the whole file up front turns every page into a guaranteed disk read whether the guest needs it or not, and blocks the guest from running until all of them finish. Lazy page-in defers each read to the exact moment it's needed and skips the reads that are never needed — at the cost of scattering those reads across the guest's early execution as minor and major faults instead of one big sequential slurp. For a workload that touches a fraction of its RAM before becoming useful, that's a large net win. The pages the guest never touches cost nothing at all.

MAP_PRIVATE vs MAP_SHARED: why private matters

The flag passed to mmap decides what happens when the guest writes to a mapped page, and it's the single most consequential choice in the restore path. Firecracker uses MAP_PRIVATE. Contrast the two:

• MAP_PRIVATE — writes are private and copy-on-write. Reads share the file's pages directly. The first time the guest writes to a page, the kernel makes a fresh private 4 KiB copy of just that page for this mapping, redirects the mapping to the copy, and the write lands there. The backing file, vm.mem, is never modified. Every other reader of the same file still sees the pristine original page.
• MAP_SHARED — writes go straight back to the file. There is no private copy; the mapping and the file are the same bytes. A guest write would mutate vm.mem on disk, and every other VM mapping that file would see the change. Restores would corrupt each other and the template.

MAP_SHARED is obviously wrong for a snapshot you want to restore many times: the first guest to write anything would poison the memory image for everyone else and for every future restore. MAP_PRIVATE is what makes the memory file a read-only baseline that any number of guests can restore from independently. Each guest gets a private copy-on-write view; the shared file stays clean. This is the exact same mechanism that makes Unix fork() cheap — copy-on-write of an address space — applied to a whole virtual machine's RAM.

Page-cache sharing: why many VMs from one file is a density win

Here's where MAP_PRIVATE pays a second dividend that's easy to miss. When multiple guests on the same host restore from the same memory file, their clean, unwritten pages are not just logically identical — they are physically the same pages in the host's page cache. The kernel reads a given 4 KiB page of vm.mem from disk once, and every mapping of that file that faults on that offset gets wired to the same resident copy. Ten VMs restored from one template that all read the same kernel text, the same libc, the same warmed caches share one physical copy of every page none of them has written.

The consequences compound:

• First restore warms the cache — it takes the major faults, pulling its working set off disk into the page cache once.
• Second, third, tenth restore ride minor faults — their working set is largely the same pages, already resident, so most of their faults resolve without touching disk. Later restores of a familiar template are measurably faster than the first.
• Density: unwritten memory is shared, not duplicated — a host running many VMs from one template holds one physical copy of the shared baseline plus each VM's small set of privately-written pages, rather than one full RAM image per VM. RAM footprint tracks divergence, not VM count.

You can watch this directly. The page cache is observable, and a tool like vmtouch (or /proc/PID/smaps) will show you how much of the memory file is resident and shared versus how much each VM has copied private:

# How much of the snapshot memory file is currently resident in the page cache?
# After the first restore warms it, this climbs; later restores hit these pages.
vmtouch -v /var/lib/pandastack/seeds/base/vm.mem
#   [OOo   o    O  oOOo ] 12345/524288  resident pages
#           2.4% of the 2 GiB file is in cache — the shared working set

# Per-process: how much of the guest RAM mapping is shared vs privately copied.
# "Shared_Clean" = pages ridden from the file cache (free, shared across VMs).
# "Private_Dirty" = pages this guest wrote and now owns a copy-on-write copy of.
grep -E 'Shared_Clean|Private_Dirty' /proc/$(pgrep -n firecracker)/smaps \
  | awk '{s[$1]+=$2} END{for (k in s) print k, s[k]" kB"}'
#   Shared_Clean:  180224 kB   <- shared with other restores, in page cache
#   Private_Dirty:  41216 kB   <- this guest's own diverged pages

The deeper treatment of this density effect — how far it scales and where it breaks — is in /blog/page-cache-sharing-microvm-density. The point here is that it falls straight out of MAP_PRIVATE plus the shared page cache: it isn't a separate feature you enable, it's what file-backed copy-on-write mmap does by default.

Nudging the kernel: MADV_ hints

The kernel's default page-in behavior is good but generic, and mmap regions can be tuned with madvise() hints about expected access patterns. A couple are relevant to memory restore. MADV_WILLNEED asks the kernel to read a range ahead of time — useful for prefetching a known hot set into the page cache before the guest faults on it, turning would-be major faults into minor ones. MADV_RANDOM tells the kernel not to bother with aggressive readahead for a region whose access pattern is scattered, so it doesn't waste I/O pulling in neighbors that won't be used; MADV_SEQUENTIAL is the opposite hint. These are advisory — the kernel is free to ignore them — but they're the standard levers for shaping when a mapped file's pages become resident, and prefetching a snapshot's hot working set is exactly the kind of thing MADV_WILLNEED exists for.

File-backed mmap vs userfaultfd streaming

The mmap path described so far assumes vm.mem is a local file. That assumption is exactly what breaks in a multi-host fleet, where snapshots are published to object storage so any host can restore any template, and the memory file may not be local yet. You have two ways to serve the guest's page faults, and they differ in where a faulted page comes from:

• File-backed mmap (this post) — the memory file must be a local file. The kernel resolves faults from disk (or the page cache). Simple, fast when the file is already local, and it's the default restore path. But it requires the whole vm.mem to be present locally before you can map it — if it lives in object storage, you must download the full multi-gigabyte file first, and most of those bytes are pages this restore will never touch.
• userfaultfd (UFFD) streaming — instead of a local file, Firecracker is handed a userfaultfd, and a user-space handler answers page faults. When the guest faults on a page, the handler fetches the surrounding chunk from object storage over an HTTP Range GET and installs it. The guest pulls only the pages it touches, over the network, and can start before the full image has arrived — no up-front download.

The key insight is that both paths implement the same idea — lazy, on-demand page-in of a memory image — and the guest sees identical copy-on-write semantics either way. mmap+MAP_PRIVATE serves the fault from a local file; UFFD serves it from a handler that can reach across the network. UFFD is what lets a host restore a template it has never held locally without paying for the whole memory file first. The full mechanics of that path — the fault flow, zero-page elision, prefetch, and the shared chunk cache — are in /blog/userfaultfd-explained. One scoping note that trips people up: UFFD streams memory, not the disk. The rootfs still has to be a local file, because copy-on-write disk cloning (reflink or dm-snapshot) needs a local block device. Streaming removes the big vm.mem download specifically.

The summary

Firecracker restores guest memory by mmap-ing the snapshot's memory file with MAP_PRIVATE and resuming — a mapping operation that is O(1) in the size of the RAM image, not a bulk read. Nothing is loaded up front; each page faults in lazily the first time the guest touches it, as a cheap minor fault if the page is already in the host's page cache or a major fault if it has to come off disk. MAP_PRIVATE makes those touches copy-on-write, so the memory file stays a pristine, shareable baseline: a guest's writes go to private per-page copies, and every clean page is physically shared across all VMs restored from that file in the page cache — the density win. The same lazy-page-in idea, served by a userfaultfd handler instead of a local file, becomes the streaming path that pulls pages from object storage on demand.

On PandaStack every sandbox, managed database, and git-driven app boots by restoring a baked snapshot through exactly this path — the memory-load step lands around 49ms because it maps the file rather than reading it, and clean pages are page-cache-shared across every VM restored from the same template on a host. The core is open source under Apache-2.0, so you can run the agent on your own Linux KVM hosts and watch the resident/shared page counts move with vmtouch and /proc/PID/smaps yourself. For the snapshot artifact model start with /blog/firecracker-memory-snapshots; for the copy-on-write fork story, /blog/copy-on-write-memory-fork-explained; for the streaming variant, /blog/userfaultfd-explained.