Every VM tutorial says the guest "has 512 MiB of RAM." It doesn't, really. It has a promise. The guest believes it owns a contiguous range of physical addresses starting near zero; the host has handed it nothing more than a range of ordinary virtual memory inside a userspace process, plus a lookup table telling the kernel how to get from one to the other. That table is the memslot list, and it is the reason microVM snapshots, copy-on-write restore, lazy memory streaming and dirty tracking are all cheap tricks instead of hypervisor features.
This post is about that table: what a memslot is, the KVM_SET_USER_MEMORY_REGION ioctl that creates one, how second-stage page tables get filled from it on demand, and how Firecracker's snapshot restore exploits the fact that guest RAM is just a host mapping that someone else is allowed to fault in. The broader KVM ioctl tour lives in How Firecracker uses the Linux KVM API: an internals explainer; this one stays on memory.
Documentation/virt/kvm/api.rst and have moved over the years. Where a claim depends on kernel version or architecture, it says so. Verify against your kernel's headers before you build on any of it.Guest physical vs host virtual: the two address spaces
A guest kernel manages its own page tables. They translate guest virtual addresses (GVA) to guest physical addresses (GPA). The guest has full control of that layer and has no idea anything sits beneath it. Meanwhile the VMM, Firecracker in our case, is a normal Linux process. The memory it reserves for the guest lives at host virtual addresses (HVA) in its own address space, which the host kernel translates to host physical addresses (HPA) like any other process memory.
So there are four address types and two owners. The guest owns GVA to GPA. The host owns HVA to HPA. The missing link is GPA to HVA, and that is exactly what a memslot provides: a guest-physical base, a length, and the host virtual address that backs it. Translation is arithmetic, not magic: hva = userspace_addr + (gpa - guest_phys_addr).
| Translation | Owner | Mechanism |
|---|---|---|
| GVA to GPA | Guest kernel | Guest page tables (CR3 on x86, TTBR on arm64) |
| GPA to HVA | KVM | Memslot list, set by the VMM |
| HVA to HPA | Host kernel | The VMM process's ordinary page tables |
| GPA to HPA (cached) | KVM + hardware | Second-stage tables: EPT, NPT, or arm64 stage-2 |
The last row is the one hardware actually walks. KVM collapses the middle two hops into second-stage page tables so the CPU can go straight from GPA to HPA without asking anyone. The memslot is the source of truth; the stage-2 tables are a cache of it, built lazily.
KVM_SET_USER_MEMORY_REGION, field by field
A memslot is created, changed or deleted with one ioctl on the VM file descriptor, KVM_SET_USER_MEMORY_REGION, which takes a struct kvm_userspace_memory_region. It has five fields and each one earns its place.
slot— the slot id. On x86 the high 16 bits select an address space (KVM keeps a separate one for System Management Mode), so a VMM that does not emulate SMM just uses small integers. How many slots you get is reported byKVM_CAP_NR_MEMSLOTS; the limit has been raised over the years, so query it rather than hard-coding it.flags—KVM_MEM_LOG_DIRTY_PAGESturns on dirty tracking for the slot;KVM_MEM_READONLYmakes guest writes exit to userspace instead of landing (handy for emulating ROM or flash). Flags are the one property you can change on a live slot by calling the ioctl again with the same slot, base and size.guest_phys_addr— where the guest sees this RAM begin. Page-aligned.memory_size— length in bytes, a multiple of the page size. Setting it to zero deletes the slot. You cannot resize or move an existing slot in place; you delete it and create a new one, which is why VMMs pick their layout once.userspace_addr— the HVA of the backing memory in the VMM's own address space. KVM does not care how you got it: anonymous mmap, a file mapping, memfd, hugetlbfs. That indifference is the whole trick.
// Sketch: back guest RAM with a host mmap and register it as memslot 0.
// Linux only; error handling trimmed. Not Firecracker source.
package main
import (
"unsafe"
"golang.org/x/sys/unix"
)
// Mirrors struct kvm_userspace_memory_region in <linux/kvm.h> (32 bytes).
type kvmUserspaceMemoryRegion struct {
Slot uint32 // low 16 bits: slot id; high 16 bits: address space id
Flags uint32 // KVM_MEM_LOG_DIRTY_PAGES (1<<0), KVM_MEM_READONLY (1<<1)
GuestPhysAddr uint64 // where the guest sees this RAM start (GPA)
MemorySize uint64 // bytes; 0 deletes the slot
UserspaceAddr uint64 // host virtual address of the backing (HVA)
}
const (
kvmCreateVM = 0xAE01 // _IO(KVMIO, 0x01)
kvmSetUserMemoryRegion = 0x4020AE46 // _IOW(KVMIO, 0x46, 32-byte struct)
kvmMemLogDirtyPages = 1 << 0
)
func main() {
kvm, _ := unix.Open("/dev/kvm", unix.O_RDWR|unix.O_CLOEXEC, 0)
vm, _, _ := unix.Syscall(unix.SYS_IOCTL, uintptr(kvm), kvmCreateVM, 0)
const ramSize = 512 << 20 // 512 MiB
// Ordinary anonymous memory in OUR address space. Nothing is allocated
// yet: pages appear when somebody (the guest, via KVM) touches them.
ram, _ := unix.Mmap(-1, 0, ramSize,
unix.PROT_READ|unix.PROT_WRITE,
unix.MAP_PRIVATE|unix.MAP_ANONYMOUS|unix.MAP_NORESERVE)
region := kvmUserspaceMemoryRegion{
Slot: 0,
Flags: kvmMemLogDirtyPages, // opt in to dirty tracking
GuestPhysAddr: 0,
MemorySize: ramSize,
UserspaceAddr: uint64(uintptr(unsafe.Pointer(&ram[0]))),
}
unix.Syscall(unix.SYS_IOCTL, vm, kvmSetUserMemoryRegion,
uintptr(unsafe.Pointer(®ion)))
// The guest now has 512 MiB of "physical" RAM at GPA 0. KVM has built
// zero stage-2 entries for it. They get filled in on first touch.
}
Notice what did not happen. KVM did not copy anything, allocate anything or pin anything. It recorded a range. The 512 MiB exists only as a virtual reservation; with MAP_NORESERVE it is not even charged against overcommit up front. The first physical page appears when the guest touches one.
KVM_SET_USER_MEMORY_REGION2 and guest_memfd
Newer kernels (it landed around Linux 6.8, alongside guest_memfd) add KVM_SET_USER_MEMORY_REGION2 and struct kvm_userspace_memory_region2. It keeps the original five fields and adds guest_memfd and guest_memfd_offset, used with the KVM_MEM_GUEST_MEMFD flag. The point is confidential computing: memory from a guest_memfd file is not mapped into the VMM's address space at all, so the host process cannot read guest private memory even if it wanted to. That breaks the property the rest of this post relies on ("guest RAM is a host mapping you can mmap, CoW and fault"), so classic snapshot restore does not carry over unchanged. Exactly which architectures and VM types support it, and what the struct padding looks like, has been shifting between releases; check api.rst for your kernel. Firecracker's standard restore path, as described below, uses ordinary userspace mappings.
Second-stage translation is filled in on fault
Right after registration, the second-stage tables for that slot are empty. Intel calls them EPT (Extended Page Tables), AMD calls them NPT (Nested Page Tables), arm64 calls them stage-2. The first time the guest touches a GPA, the hardware walk finds no entry and raises an EPT violation, an NPT fault, or a stage-2 abort. That is a VM exit, but one KVM usually handles in the kernel without ever bothering the VMM. The exit side of this is covered in VM exits: the actual currency of virtualization overhead.
- Find the memslot containing the faulting GPA. No slot at all means the address is not RAM, so KVM hands the access to userspace as
KVM_EXIT_MMIO. That is how virtio-mmio device registers work: they are simply holes in the memslot map. - Compute the HVA from the slot's base and offset.
- Resolve the HVA to a host physical page, the way any kernel code resolves user memory. If the host page is not present, this triggers an ordinary host page fault on the VMM's mapping: allocate a zero page for anonymous memory, read from page cache for a file mapping, break CoW for a private write, or, if the range is registered with userfaultfd, put the faulting thread to sleep and notify a handler.
- Install a stage-2 entry GPA to HPA, writable only if the host mapping allows it, and re-enter the guest. Next access to that page costs nothing.
Step 3 is the hinge. KVM does not have its own memory semantics; it borrows whatever the host mapping has. And KVM registers MMU notifiers on the VMM's address space, so when the host changes a mapping (swaps a page out, migrates it, breaks CoW, punches a hole) KVM zaps the matching stage-2 entry and the next guest access faults it back in. The stage-2 table stays a coherent cache of the host mapping, not a separate copy that can drift.
KVM_MEM_LOG_DIRTY_PAGES and KVM_GET_DIRTY_LOG
Set KVM_MEM_LOG_DIRTY_PAGES on a slot and KVM starts keeping a bitmap, one bit per guest page in the slot. To catch writes it write-protects the slot's stage-2 entries (or, on Intel hardware that has it, uses PML, Page Modification Logging, to batch the records); the first write to a page traps, KVM sets its bit and lets the write through. KVM_GET_DIRTY_LOG, with a struct kvm_dirty_log naming the slot and a user buffer, copies the bitmap out. Classically that read also resets tracking; with KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2 enabled, the reset becomes an explicit KVM_CLEAR_DIRTY_LOG so you can re-protect in smaller chunks. There is also a per-vCPU dirty ring (KVM_CAP_DIRTY_LOG_RING) that scales with pages dirtied instead of RAM size.
Two consumers matter. Live migration harvests the log in rounds and resends dirtied pages until the set is small enough to stop the VM. Firecracker's diff snapshots enable tracking (track_dirty_pages in its machine configuration) and write only the pages that changed since the last snapshot. The cost is real: write-protection turns the first write to every page into an exit, which is why tracking is opt-in. Both mechanisms get full treatment in Firecracker Dirty Page Tracking, Explained and Firecracker Diff Snapshots Explained.
How Firecracker snapshot restore uses the memslot
A Firecracker memory snapshot, vm.mem, is guest RAM written out byte for byte, region by region. Restore has to recreate the guest's memory before the vCPUs resume, and the cheap way to do that is to not recreate it at all. Firecracker picks a backing for the HVA range, registers that range as the memslot, and lets stage-2 faults do the rest. Its /snapshot/load API offers two memory backends, and they differ only in what sits behind userspace_addr.
File backend: mmap vm.mem MAP_PRIVATE
With the file backend, Firecracker mmaps vm.mem with MAP_PRIVATE and registers that mapping as guest RAM. A guest read of an untouched page faults through stage-2 into a host fault that reads the page from the page cache, and the stage-2 entry can map it read-only. A guest write breaks copy-on-write in the host: the kernel copies the page into a private anonymous page for this process, the MMU notifier invalidates the stale stage-2 entry, and the guest gets a writable mapping of its own copy. The file on disk is never modified.
Restore cost becomes proportional to the pages the guest touches, not the size of RAM. And clean pages stay shared: ten VMs restored from the same vm.mem on one host read the same page-cache pages until each one writes. This is the memory copy-on-write that PandaStack's snapshot-restore path is built on; the flag-level details are in MAP_PRIVATE vs MAP_SHARED: The mmap Flag That Decides Everything and the restore walkthrough in How Firecracker Restores Guest Memory: mmap & MAP_PRIVATE.
UFFD backend: a handler serves the faults
With the UFFD backend, Firecracker maps anonymous memory for guest RAM, registers it as the memslot as usual, and also registers the same range with userfaultfd. It then connects to a Unix socket the operator provides and sends two things: the userfaultfd file descriptor (as SCM_RIGHTS ancillary data) and a JSON array describing each region, with base_host_virt_addr, size, offset into the snapshot file, and page_size. From then on, a host fault in that range does not allocate anything. The faulting thread sleeps, the handler reads a UFFD_EVENT_PAGEFAULT event, fetches the bytes from wherever it likes, and installs them with UFFDIO_COPY (or UFFDIO_ZEROPAGE for a page it knows is zero), which wakes the thread.
Look at what that buys. The memslot still says GPA to HVA. The handler's region list says HVA to file offset. Chain them and a guest touching physical address X becomes a request for byte range Y of a file that does not have to be on this machine.
// The two lookups a streamed restore performs on one guest page touch.
// Hop 1 is KVM's (memslot); hop 2 is the UFFD handler's (region mapping).
type memslot struct{ gpa, size, hva uint64 }
// KVM, on a stage-2 fault: which host virtual address backs this GPA?
func gpaToHVA(slots []memslot, gpa uint64) (uint64, bool) {
for _, s := range slots {
if gpa >= s.gpa && gpa < s.gpa+s.size {
return s.hva + (gpa - s.gpa), true
}
}
return 0, false // no slot: not RAM -> KVM_EXIT_MMIO to the VMM
}
// The UFFD handler, on the resulting host fault: which byte of vm.mem?
// Same shape as GuestRegionUffdMapping in Firecracker's handoff JSON.
type uffdRegion struct{ baseHostVirtAddr, size, offset, pageSize uint64 }
func hvaToFileOffset(r uffdRegion, hva uint64) uint64 {
page := hva &^ (r.pageSize - 1) // 4 KiB or 2 MiB, per region
return r.offset + (page - r.baseHostVirtAddr)
}
That is how PandaStack streams memory. When PANDASTACK_STREAM_RESTORE is on and the template's seed is a thin one, the agent points /snapshot/load at a socket served by its own handler (agent/internal/uffd). The handler maps each fault to a file offset exactly as above, and the resolver behind it (agent/internal/memstream) answers from a baked zero-chunk index, a shared per-host chunk cache, or an HTTP Range GET of vm.mem in GCS. Zero pages on read faults go in with UFFDIO_ZEROPAGE so they cost no private RAM; write faults go straight to UFFDIO_COPY. A transient storage error is retried rather than killing the guest, and a watchdog fails the sandbox loudly if the handler dies or stalls, instead of leaving a vCPU asleep in a fault forever. Note the limit: this streams memory only. The rootfs is always a local file, because disk copy-on-write (reflink or dm-snapshot) needs a local block device.
- File backend — Backing:
vm.memmappedMAP_PRIVATE. Fault served by: the host kernel, from page cache. Needs: the whole file local. Sharing: clean pages shared across VMs via page cache. - UFFD backend — Backing: anonymous memory plus userfaultfd. Fault served by: your handler process. Needs: a handler and a source for the bytes, local or remote. Sharing: whatever your handler arranges; installed pages are private to the VM, except zero pages mapped with
UFFDIO_ZEROPAGE.
The handler protocol, and the many ways one can wedge a guest, are covered in userfaultfd: Lazy Memory for Instant VM Restore, How to Write a userfaultfd Handler for Firecracker and userfaultfd in Production: How the Pager Fails.
Hugepages: fewer faults, fewer TLB misses, sharper edges
Everything above happens per page, and a page is 4 KiB by default. If the memslot is backed by 2 MiB hugetlbfs pages, one host fault covers 512 times as much memory, and KVM can install a 2 MiB block entry in the stage-2 table when the guest physical and host virtual addresses are both suitably aligned and the host backing is itself a 2 MiB mapping. That matters twice: fewer faults on restore, and fewer TLB misses afterwards, because a two-dimensional page walk (guest tables times stage-2 tables) is expensive, and big leaf entries cut both its depth and how often you pay it.
The edges are sharp. Hugepage-ness is a property of the snapshot, and Firecracker restores hugepage snapshots only through the UFFD backend, so the handler must serve whole 2 MiB pages (the page_size in the handoff tells it which). UFFDIO_ZEROPAGE does not work on hugetlb ranges, so zero pages cost real memory there. And hugetlbfs pages come from a pool reserved at boot that is never returned to page cache, which is why PandaStack keeps that pool small and runs its first-party templates on 4 KiB pages. The full trade-off is in Firecracker Hugepages for Guest Memory, Explained.
What to take away
- A memslot is a linear GPA-to-HVA mapping. KVM's memory model is mostly "whatever the host mapping does."
- Stage-2 tables are a lazily filled, MMU-notifier-coherent cache of the memslots. Memory is committed on touch, not on registration.
- Holes in the memslot map are MMIO. Device emulation is literally the absence of RAM.
- Dirty tracking is a per-slot flag plus a bitmap (or ring) that write-protection fills; it costs exits, so it is opt-in.
- Snapshot restore picks the backing:
MAP_PRIVATEfile for page-cache CoW, or userfaultfd for a handler that can fetch pages from anywhere. Same memslot either way. - If you are going to read one file, read
Documentation/virt/kvm/api.rstfor your kernel. It is less dramatic than this post and more correct.
On PandaStack you never touch any of this directly. Every Sandbox.create() restores a baked template snapshot through this path as part of a 179 ms p50 create, and the guest pays for memory only as it touches it. But when a restored sandbox's memory graph looks odd, flat RSS that climbs on first request, or a stall on a cold page, this is the machinery underneath it.