Firecracker Guest Memory Layout, Explained

When a Firecracker guest boots and reads its own `/proc/meminfo`, it sees a physical address space that looks like a small, tidy PC. It isn't a PC — it's a carefully shaped region of the host's virtual memory that the VMM allocated and handed to KVM. Understanding the shape of that region is more than trivia: the guest's physical memory layout is exactly what a snapshot serializes, it's what copy-on-write restore maps back in, and it's what userfaultfd pages in on demand. Every fast-boot and fast-fork trick a sandbox platform plays is played on top of this layout. This post walks it.

I'm Ajay; I built PandaStack, where `vm.mem` — the serialized guest RAM — is the artifact every create restores (restore step ~49ms, p50 179ms). Knowing what's inside that file, and why it has the holes it has, is what makes the rest of the snapshot story make sense.

Guest-physical is host-virtual

Start with the core mapping. Firecracker asks the host for a big anonymous region with `mmap` and registers it with KVM as a memory slot: "guest physical addresses in this range live at this host virtual address." When the guest CPU touches a guest-physical address (GPA), the hardware's second-level page tables (EPT on Intel, NPT on AMD) translate it to a host-physical page backing that mmap. KVM maintains the GPA-to-host-virtual-address (HVA) correspondence; the CPU does the fast per-access translation. The practical upshot: guest RAM is host memory the VMM owns, and "the guest's physical memory" is just a window into the host's virtual address space.

That single fact is what makes everything downstream possible. Because guest RAM is a host mapping, the VMM can serialize it to a file (that's a snapshot's `vm.mem`), map a file back in with `MAP_PRIVATE` so the kernel does copy-on-write on the first write to each page (that's restore and fork), or register the region with `userfaultfd` so a page fault becomes an event the VMM can service by fetching the page from somewhere else (that's streaming). None of these tricks are exotic — they're ordinary things you can do to a memory mapping. The guest just happens to live inside one.

The shape of the guest address space (x86_64)

The region isn't a featureless slab. On x86_64, the guest physical address space inherits some shape from PC history and some from the need to keep device windows out of RAM. Roughly, low to high:

1. Below 1 MiB — legacy low memory. The bottom of the address space carries the ghosts of the PC's real-mode past: the boot region, and reserved sub-1MiB areas. Firecracker keeps this region for boot compatibility even though a microVM has no BIOS in the traditional sense.
2. Low RAM, from ~1 MiB up — the main block of usable guest RAM for guests that fit under the 4 GiB line. The kernel image, the boot command line, and any initrd get loaded into known offsets in this region during setup, so the guest kernel knows where to find itself.
3. The MMIO gap, near 3-4 GiB — a deliberate hole in RAM. Memory-mapped device windows (the virtio-MMIO device register blocks, the APIC, and so on) need physical addresses, and they can't overlap RAM. So the layout leaves a gap below the 4 GiB boundary where those windows live. RAM stops before the gap.
4. High RAM, above 4 GiB — if the guest has more memory than fits below the MMIO gap, the remainder is mapped above the 4 GiB line, past the device windows. A guest with a few GiB of RAM has a low chunk and, if it's big enough, a high chunk, with the MMIO gap between them.

The MMIO gap is the part people trip on. "Why does my guest with 4 GiB configured see slightly less usable RAM, split into two ranges?" Because a contiguous 0-to-4GiB of RAM would have nowhere to put device windows. The gap is the compromise: reserve a band of physical address space near 4 GiB for MMIO, push the RAM that would have been there up above 4 GiB. It's the same reason a physical machine with 4 GiB installed historically showed ~3.2 GiB usable under 32-bit addressing — the address space near the top is spoken for by devices.

Why this layout is exactly what you snapshot

Now the payoff for a sandbox platform. When Firecracker takes a memory snapshot, `vm.mem` is this guest-physical region serialized — the low RAM, the high RAM, the whole populated address space, as bytes. The device state and vCPU registers go in a separate state file, but the memory file is a direct image of the guest's physical RAM. Restore is the reverse: map `vm.mem` back into a fresh mmap at the right layout, hand it to KVM as the memory slot, and load the state file. The guest resumes believing not a nanosecond passed.

The layout being a plain memory image is why restore can be lazy. On PandaStack, restore maps the memory file `MAP_PRIVATE`, so the kernel doesn't copy the whole image up front — pages fault in as the guest touches them, and a write triggers copy-on-write so the shared base image stays pristine. That's why the restore step is ~49ms rather than however long copying gigabytes would take: most of the image isn't touched in the first moments of the guest's life, so most of it never gets paged in on the critical path.

from pandastack import Sandbox

# Every create restores a baked snapshot. Under the hood, vm.mem -- the
# serialized guest-physical region described above -- is mapped MAP_PRIVATE,
# so pages fault in lazily and writes copy-on-write. That's the ~49ms restore.
sbx = Sandbox.create(template="code-interpreter", ttl_seconds=120)  # ~179ms p50

# Fork clones the DISK copy-on-write and restores the same memory image, so a
# forked guest inherits the baseline's populated address space cheaply.
snap = sbx.snapshot()
child = snap.fork(ttl_seconds=120)     # ~400-750ms same-host
child.delete()
sbx.delete()

And it's why streaming works. Because guest RAM is a mapping the VMM can intercept faults on, PandaStack can register the region with userfaultfd and serve those faults from a memory file living in object storage — paging the guest's RAM in over the network on demand instead of downloading the whole `vm.mem` before the guest can start. The layout doesn't change; only where the pages come from does. The guest's view of its own physical memory is identical whether the bytes arrived from a local file or a range-GET against a bucket.

The takeaway

• Guest RAM is a host mmap — a contiguous region the VMM owns, mapped through second-level page tables; KVM keeps the GPA-to-HVA correspondence.
• The layout has deliberate holes — a legacy region below 1MiB and an MMIO gap near 4GiB so device windows don't collide with RAM; big guests get a low chunk and a high chunk above 4GiB.
• vm.mem is that region serialized — a memory snapshot is a direct image of the guest's physical RAM; state and registers live in a separate file.
• The image nature is what makes restore fast — MAP_PRIVATE lets pages fault in lazily with copy-on-write, so restore is ~49ms instead of a full copy, and userfaultfd lets pages stream from object storage on demand.

You rarely need to reason about the exact offset the kernel loads at. But knowing that the guest's memory is one shaped, serializable host mapping demystifies the whole snapshot-and-fork story: it's not magic, it's a mmap with a known layout, and every fast-path trick is just something you're allowed to do to a mmap. For the adjacent mechanics, /blog/firecracker-memory-file-mmap-explained covers the mmap-and-CoW path in depth, /blog/userfaultfd-explained is the on-demand paging story, and /blog/firecracker-memory-snapshots ties the full snapshot artifact together.