all postsblog / firecracker & microvms

blog

KVM Memslots Explained: How a microVM's RAM Is Actually Mapped

written by
Ajay Kumar
published
reading time
10 min read
topics

Every VM tutorial says the guest "has 512 MiB of RAM." It doesn't, really. It has a promise. The guest believes it owns a contiguous range of physical addresses starting near zero; the host has handed it nothing more than a range of ordinary virtual memory inside a userspace process, plus a lookup table telling the kernel how to get from one to the other. That table is the memslot list, and it is the reason microVM snapshots, copy-on-write restore, lazy memory streaming and dirty tracking are all cheap tricks instead of hypervisor features.

This post is about that table: what a memslot is, the KVM_SET_USER_MEMORY_REGION ioctl that creates one, how second-stage page tables get filled from it on demand, and how Firecracker's snapshot restore exploits the fact that guest RAM is just a host mapping that someone else is allowed to fault in. The broader KVM ioctl tour lives in How Firecracker uses the Linux KVM API: an internals explainer; this one stays on memory.

Kernel details here follow Documentation/virt/kvm/api.rst and have moved over the years. Where a claim depends on kernel version or architecture, it says so. Verify against your kernel's headers before you build on any of it.

Guest physical vs host virtual: the two address spaces

A guest kernel manages its own page tables. They translate guest virtual addresses (GVA) to guest physical addresses (GPA). The guest has full control of that layer and has no idea anything sits beneath it. Meanwhile the VMM, Firecracker in our case, is a normal Linux process. The memory it reserves for the guest lives at host virtual addresses (HVA) in its own address space, which the host kernel translates to host physical addresses (HPA) like any other process memory.

So there are four address types and two owners. The guest owns GVA to GPA. The host owns HVA to HPA. The missing link is GPA to HVA, and that is exactly what a memslot provides: a guest-physical base, a length, and the host virtual address that backs it. Translation is arithmetic, not magic: hva = userspace_addr + (gpa - guest_phys_addr).

Who translates what
TranslationOwnerMechanism
GVA to GPAGuest kernelGuest page tables (CR3 on x86, TTBR on arm64)
GPA to HVAKVMMemslot list, set by the VMM
HVA to HPAHost kernelThe VMM process's ordinary page tables
GPA to HPA (cached)KVM + hardwareSecond-stage tables: EPT, NPT, or arm64 stage-2

The last row is the one hardware actually walks. KVM collapses the middle two hops into second-stage page tables so the CPU can go straight from GPA to HPA without asking anyone. The memslot is the source of truth; the stage-2 tables are a cache of it, built lazily.

KVM_SET_USER_MEMORY_REGION, field by field

A memslot is created, changed or deleted with one ioctl on the VM file descriptor, KVM_SET_USER_MEMORY_REGION, which takes a struct kvm_userspace_memory_region. It has five fields and each one earns its place.

  • slot — the slot id. On x86 the high 16 bits select an address space (KVM keeps a separate one for System Management Mode), so a VMM that does not emulate SMM just uses small integers. How many slots you get is reported by KVM_CAP_NR_MEMSLOTS; the limit has been raised over the years, so query it rather than hard-coding it.
  • flags — KVM_MEM_LOG_DIRTY_PAGES turns on dirty tracking for the slot; KVM_MEM_READONLY makes guest writes exit to userspace instead of landing (handy for emulating ROM or flash). Flags are the one property you can change on a live slot by calling the ioctl again with the same slot, base and size.
  • guest_phys_addr — where the guest sees this RAM begin. Page-aligned.
  • memory_size — length in bytes, a multiple of the page size. Setting it to zero deletes the slot. You cannot resize or move an existing slot in place; you delete it and create a new one, which is why VMMs pick their layout once.
  • userspace_addr — the HVA of the backing memory in the VMM's own address space. KVM does not care how you got it: anonymous mmap, a file mapping, memfd, hugetlbfs. That indifference is the whole trick.
go
// Sketch: back guest RAM with a host mmap and register it as memslot 0.
// Linux only; error handling trimmed. Not Firecracker source.
package main

import (
	"unsafe"

	"golang.org/x/sys/unix"
)

// Mirrors struct kvm_userspace_memory_region in <linux/kvm.h> (32 bytes).
type kvmUserspaceMemoryRegion struct {
	Slot          uint32 // low 16 bits: slot id; high 16 bits: address space id
	Flags         uint32 // KVM_MEM_LOG_DIRTY_PAGES (1<<0), KVM_MEM_READONLY (1<<1)
	GuestPhysAddr uint64 // where the guest sees this RAM start (GPA)
	MemorySize    uint64 // bytes; 0 deletes the slot
	UserspaceAddr uint64 // host virtual address of the backing (HVA)
}

const (
	kvmCreateVM            = 0xAE01     // _IO(KVMIO, 0x01)
	kvmSetUserMemoryRegion = 0x4020AE46 // _IOW(KVMIO, 0x46, 32-byte struct)
	kvmMemLogDirtyPages    = 1 << 0
)

func main() {
	kvm, _ := unix.Open("/dev/kvm", unix.O_RDWR|unix.O_CLOEXEC, 0)
	vm, _, _ := unix.Syscall(unix.SYS_IOCTL, uintptr(kvm), kvmCreateVM, 0)

	const ramSize = 512 << 20 // 512 MiB
	// Ordinary anonymous memory in OUR address space. Nothing is allocated
	// yet: pages appear when somebody (the guest, via KVM) touches them.
	ram, _ := unix.Mmap(-1, 0, ramSize,
		unix.PROT_READ|unix.PROT_WRITE,
		unix.MAP_PRIVATE|unix.MAP_ANONYMOUS|unix.MAP_NORESERVE)

	region := kvmUserspaceMemoryRegion{
		Slot:          0,
		Flags:         kvmMemLogDirtyPages, // opt in to dirty tracking
		GuestPhysAddr: 0,
		MemorySize:    ramSize,
		UserspaceAddr: uint64(uintptr(unsafe.Pointer(&ram[0]))),
	}
	unix.Syscall(unix.SYS_IOCTL, vm, kvmSetUserMemoryRegion,
		uintptr(unsafe.Pointer(&region)))
	// The guest now has 512 MiB of "physical" RAM at GPA 0. KVM has built
	// zero stage-2 entries for it. They get filled in on first touch.
}

Notice what did not happen. KVM did not copy anything, allocate anything or pin anything. It recorded a range. The 512 MiB exists only as a virtual reservation; with MAP_NORESERVE it is not even charged against overcommit up front. The first physical page appears when the guest touches one.

KVM_SET_USER_MEMORY_REGION2 and guest_memfd

Newer kernels (it landed around Linux 6.8, alongside guest_memfd) add KVM_SET_USER_MEMORY_REGION2 and struct kvm_userspace_memory_region2. It keeps the original five fields and adds guest_memfd and guest_memfd_offset, used with the KVM_MEM_GUEST_MEMFD flag. The point is confidential computing: memory from a guest_memfd file is not mapped into the VMM's address space at all, so the host process cannot read guest private memory even if it wanted to. That breaks the property the rest of this post relies on ("guest RAM is a host mapping you can mmap, CoW and fault"), so classic snapshot restore does not carry over unchanged. Exactly which architectures and VM types support it, and what the struct padding looks like, has been shifting between releases; check api.rst for your kernel. Firecracker's standard restore path, as described below, uses ordinary userspace mappings.

Second-stage translation is filled in on fault

Right after registration, the second-stage tables for that slot are empty. Intel calls them EPT (Extended Page Tables), AMD calls them NPT (Nested Page Tables), arm64 calls them stage-2. The first time the guest touches a GPA, the hardware walk finds no entry and raises an EPT violation, an NPT fault, or a stage-2 abort. That is a VM exit, but one KVM usually handles in the kernel without ever bothering the VMM. The exit side of this is covered in VM exits: the actual currency of virtualization overhead.

  1. Find the memslot containing the faulting GPA. No slot at all means the address is not RAM, so KVM hands the access to userspace as KVM_EXIT_MMIO. That is how virtio-mmio device registers work: they are simply holes in the memslot map.
  2. Compute the HVA from the slot's base and offset.
  3. Resolve the HVA to a host physical page, the way any kernel code resolves user memory. If the host page is not present, this triggers an ordinary host page fault on the VMM's mapping: allocate a zero page for anonymous memory, read from page cache for a file mapping, break CoW for a private write, or, if the range is registered with userfaultfd, put the faulting thread to sleep and notify a handler.
  4. Install a stage-2 entry GPA to HPA, writable only if the host mapping allows it, and re-enter the guest. Next access to that page costs nothing.

Step 3 is the hinge. KVM does not have its own memory semantics; it borrows whatever the host mapping has. And KVM registers MMU notifiers on the VMM's address space, so when the host changes a mapping (swaps a page out, migrates it, breaks CoW, punches a hole) KVM zaps the matching stage-2 entry and the next guest access faults it back in. The stage-2 table stays a coherent cache of the host mapping, not a separate copy that can drift.

This is why "the VM has 4 GiB" and "the VM is using 4 GiB" are different statements. A restored guest's resident set is the pages it has actually touched since restore, plus whatever got prefetched. Everything else is still a promise.

KVM_MEM_LOG_DIRTY_PAGES and KVM_GET_DIRTY_LOG

Set KVM_MEM_LOG_DIRTY_PAGES on a slot and KVM starts keeping a bitmap, one bit per guest page in the slot. To catch writes it write-protects the slot's stage-2 entries (or, on Intel hardware that has it, uses PML, Page Modification Logging, to batch the records); the first write to a page traps, KVM sets its bit and lets the write through. KVM_GET_DIRTY_LOG, with a struct kvm_dirty_log naming the slot and a user buffer, copies the bitmap out. Classically that read also resets tracking; with KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2 enabled, the reset becomes an explicit KVM_CLEAR_DIRTY_LOG so you can re-protect in smaller chunks. There is also a per-vCPU dirty ring (KVM_CAP_DIRTY_LOG_RING) that scales with pages dirtied instead of RAM size.

Two consumers matter. Live migration harvests the log in rounds and resends dirtied pages until the set is small enough to stop the VM. Firecracker's diff snapshots enable tracking (track_dirty_pages in its machine configuration) and write only the pages that changed since the last snapshot. The cost is real: write-protection turns the first write to every page into an exit, which is why tracking is opt-in. Both mechanisms get full treatment in Firecracker Dirty Page Tracking, Explained and Firecracker Diff Snapshots Explained.

How Firecracker snapshot restore uses the memslot

A Firecracker memory snapshot, vm.mem, is guest RAM written out byte for byte, region by region. Restore has to recreate the guest's memory before the vCPUs resume, and the cheap way to do that is to not recreate it at all. Firecracker picks a backing for the HVA range, registers that range as the memslot, and lets stage-2 faults do the rest. Its /snapshot/load API offers two memory backends, and they differ only in what sits behind userspace_addr.

File backend: mmap vm.mem MAP_PRIVATE

With the file backend, Firecracker mmaps vm.mem with MAP_PRIVATE and registers that mapping as guest RAM. A guest read of an untouched page faults through stage-2 into a host fault that reads the page from the page cache, and the stage-2 entry can map it read-only. A guest write breaks copy-on-write in the host: the kernel copies the page into a private anonymous page for this process, the MMU notifier invalidates the stale stage-2 entry, and the guest gets a writable mapping of its own copy. The file on disk is never modified.

Restore cost becomes proportional to the pages the guest touches, not the size of RAM. And clean pages stay shared: ten VMs restored from the same vm.mem on one host read the same page-cache pages until each one writes. This is the memory copy-on-write that PandaStack's snapshot-restore path is built on; the flag-level details are in MAP_PRIVATE vs MAP_SHARED: The mmap Flag That Decides Everything and the restore walkthrough in How Firecracker Restores Guest Memory: mmap & MAP_PRIVATE.

UFFD backend: a handler serves the faults

With the UFFD backend, Firecracker maps anonymous memory for guest RAM, registers it as the memslot as usual, and also registers the same range with userfaultfd. It then connects to a Unix socket the operator provides and sends two things: the userfaultfd file descriptor (as SCM_RIGHTS ancillary data) and a JSON array describing each region, with base_host_virt_addr, size, offset into the snapshot file, and page_size. From then on, a host fault in that range does not allocate anything. The faulting thread sleeps, the handler reads a UFFD_EVENT_PAGEFAULT event, fetches the bytes from wherever it likes, and installs them with UFFDIO_COPY (or UFFDIO_ZEROPAGE for a page it knows is zero), which wakes the thread.

Look at what that buys. The memslot still says GPA to HVA. The handler's region list says HVA to file offset. Chain them and a guest touching physical address X becomes a request for byte range Y of a file that does not have to be on this machine.

go
// The two lookups a streamed restore performs on one guest page touch.
// Hop 1 is KVM's (memslot); hop 2 is the UFFD handler's (region mapping).

type memslot struct{ gpa, size, hva uint64 }

// KVM, on a stage-2 fault: which host virtual address backs this GPA?
func gpaToHVA(slots []memslot, gpa uint64) (uint64, bool) {
	for _, s := range slots {
		if gpa >= s.gpa && gpa < s.gpa+s.size {
			return s.hva + (gpa - s.gpa), true
		}
	}
	return 0, false // no slot: not RAM -> KVM_EXIT_MMIO to the VMM
}

// The UFFD handler, on the resulting host fault: which byte of vm.mem?
// Same shape as GuestRegionUffdMapping in Firecracker's handoff JSON.
type uffdRegion struct{ baseHostVirtAddr, size, offset, pageSize uint64 }

func hvaToFileOffset(r uffdRegion, hva uint64) uint64 {
	page := hva &^ (r.pageSize - 1) // 4 KiB or 2 MiB, per region
	return r.offset + (page - r.baseHostVirtAddr)
}

That is how PandaStack streams memory. When PANDASTACK_STREAM_RESTORE is on and the template's seed is a thin one, the agent points /snapshot/load at a socket served by its own handler (agent/internal/uffd). The handler maps each fault to a file offset exactly as above, and the resolver behind it (agent/internal/memstream) answers from a baked zero-chunk index, a shared per-host chunk cache, or an HTTP Range GET of vm.mem in GCS. Zero pages on read faults go in with UFFDIO_ZEROPAGE so they cost no private RAM; write faults go straight to UFFDIO_COPY. A transient storage error is retried rather than killing the guest, and a watchdog fails the sandbox loudly if the handler dies or stalls, instead of leaving a vCPU asleep in a fault forever. Note the limit: this streams memory only. The rootfs is always a local file, because disk copy-on-write (reflink or dm-snapshot) needs a local block device.

  • File backend — Backing: vm.mem mapped MAP_PRIVATE. Fault served by: the host kernel, from page cache. Needs: the whole file local. Sharing: clean pages shared across VMs via page cache.
  • UFFD backend — Backing: anonymous memory plus userfaultfd. Fault served by: your handler process. Needs: a handler and a source for the bytes, local or remote. Sharing: whatever your handler arranges; installed pages are private to the VM, except zero pages mapped with UFFDIO_ZEROPAGE.

The handler protocol, and the many ways one can wedge a guest, are covered in userfaultfd: Lazy Memory for Instant VM Restore, How to Write a userfaultfd Handler for Firecracker and userfaultfd in Production: How the Pager Fails.

Hugepages: fewer faults, fewer TLB misses, sharper edges

Everything above happens per page, and a page is 4 KiB by default. If the memslot is backed by 2 MiB hugetlbfs pages, one host fault covers 512 times as much memory, and KVM can install a 2 MiB block entry in the stage-2 table when the guest physical and host virtual addresses are both suitably aligned and the host backing is itself a 2 MiB mapping. That matters twice: fewer faults on restore, and fewer TLB misses afterwards, because a two-dimensional page walk (guest tables times stage-2 tables) is expensive, and big leaf entries cut both its depth and how often you pay it.

The edges are sharp. Hugepage-ness is a property of the snapshot, and Firecracker restores hugepage snapshots only through the UFFD backend, so the handler must serve whole 2 MiB pages (the page_size in the handoff tells it which). UFFDIO_ZEROPAGE does not work on hugetlb ranges, so zero pages cost real memory there. And hugetlbfs pages come from a pool reserved at boot that is never returned to page cache, which is why PandaStack keeps that pool small and runs its first-party templates on 4 KiB pages. The full trade-off is in Firecracker Hugepages for Guest Memory, Explained.

What to take away

  • A memslot is a linear GPA-to-HVA mapping. KVM's memory model is mostly "whatever the host mapping does."
  • Stage-2 tables are a lazily filled, MMU-notifier-coherent cache of the memslots. Memory is committed on touch, not on registration.
  • Holes in the memslot map are MMIO. Device emulation is literally the absence of RAM.
  • Dirty tracking is a per-slot flag plus a bitmap (or ring) that write-protection fills; it costs exits, so it is opt-in.
  • Snapshot restore picks the backing: MAP_PRIVATE file for page-cache CoW, or userfaultfd for a handler that can fetch pages from anywhere. Same memslot either way.
  • If you are going to read one file, read Documentation/virt/kvm/api.rst for your kernel. It is less dramatic than this post and more correct.

On PandaStack you never touch any of this directly. Every Sandbox.create() restores a baked template snapshot through this path as part of a 179 ms p50 create, and the guest pays for memory only as it touches it. But when a restored sandbox's memory graph looks odd, flat RSS that climbs on first request, or a stall on a cold page, this is the machinery underneath it.

faq

Frequently asked questions

What is a KVM memslot?

A memslot is an entry in KVM's per-VM table that maps a range of guest physical addresses to a range of host virtual addresses in the VMM process. The VMM creates it with the KVM_SET_USER_MEMORY_REGION ioctl, giving a slot id, flags, the guest physical base, the size and the host virtual address of the backing memory. KVM uses it to resolve guest memory accesses and fills second-stage page tables from it on demand.

Does KVM_SET_USER_MEMORY_REGION allocate or pin guest memory?

No. It records the mapping. The backing is whatever the VMM mapped at userspace_addr, and physical pages are only allocated when the guest touches them and KVM resolves the resulting second-stage fault through the host's normal page-fault path. KVM tracks host changes through MMU notifiers instead of long-term pinning, though specific features such as device passthrough can pin memory.

What is the difference between KVM_SET_USER_MEMORY_REGION and KVM_SET_USER_MEMORY_REGION2?

The second version, added in newer kernels (around Linux 6.8) together with guest_memfd, extends the struct with a guest_memfd file descriptor and offset. With the KVM_MEM_GUEST_MEMFD flag, guest private memory comes from that file rather than from a mapping in the VMM's address space, which is meant for confidential VMs where the host should not be able to read guest memory. Support varies by architecture and kernel version, so check your kernel's api.rst.

How does Firecracker restore guest memory from a snapshot?

Either by mapping the vm.mem snapshot file with MAP_PRIVATE and registering that mapping as guest RAM, so pages are read from page cache on first touch and copied only when the guest writes, or by registering anonymous memory with userfaultfd and handing the descriptor to an external handler that supplies each page on fault. In both cases the memslot is the same; only the backing behind the host virtual address changes.

Why do hugepages make microVM restore faster?

With 2 MiB pages, one fault populates 512 times as much memory as a 4 KiB fault, and KVM can use large second-stage entries, which reduces TLB misses during the two-level page walk. The costs are that Firecracker restores hugepage snapshots only through the userfaultfd backend, zero pages cannot use UFFDIO_ZEROPAGE, and hugetlbfs memory comes from a reserved pool the host cannot reuse for page cache.

links

Keep reading

more posts

Related posts

  • Firecracker Guest Memory Layout, Explained

    A Firecracker guest's RAM is one contiguous host mmap with a specific shape — a legacy hole below 1MiB, an MMIO gap near 4GiB, the kernel loaded at a known offset. That layout is what snapshots serialize and CoW restore rides on.

  • Firecracker Diff Snapshots Explained

    A full snapshot is a photograph of the entire guest; a diff snapshot is a note that says 'only the third drawer moved.' Firecracker tracks dirtied pages with KVM's dirty-page log and writes just those — tiny and fast, but restore has to layer the diff back onto its base.

  • UFFDIO_ZEROPAGE vs UFFDIO_COPY: Stop Paying RAM for Zeros

    A demand-paged microVM restore answers thousands of page faults. If your handler answers all of them with UFFDIO_COPY, you are allocating anonymous pages to store nothing — the single biggest hidden cost in a restore fleet.

  • VM exits: the actual currency of virtualization overhead

    "Virtualization overhead is about N percent" is a meaningless sentence. The unit that actually costs you something is the VM exit — and how many you burn depends entirely on what your guest is doing.

  • Snapshot-Restore vs Live Migration: Not the Same Problem

    Live migration moves ONE running VM to a new host without dropping it. Snapshot-restore freezes ONE VM once and restores it as MANY new ones. They both involve freezing a machine and picking it up somewhere else, and that's exactly where the confusion starts.

More in Firecracker & microVMs · See Firecracker microVM sandboxes

pandastack

Run code in a microVM in one API call.

50–100 ms cold start. Fork, snapshot, and scale to zero.

measured on the host; your network round trip is extra