Firecracker Hugepages for Guest Memory, Explained

I build PandaStack, a Firecracker microVM platform where every sandbox create is a snapshot restore rather than a boot, so the cost of populating guest memory is not an abstract concern — it is most of the latency budget. Hugepages are the one knob that changes the arithmetic of that budget rather than shaving a few percent off it: with 2 MiB pages, one page fault covers 2 MiB of guest RAM instead of 4 KiB. That is 512x fewer faults to populate the same memory. It is also, by a distance, the most under-documented feature in the Firecracker surface, because turning it on quietly changes what your snapshots are and which restore paths can read them. This post is what I wish had existed before I turned it on: the mechanics, the restore-path payoff, the snapshot gotcha that will bite you, the UFFD handler consequence, the host prerequisites, and the honest tradeoffs.

What 2 MiB hugepages actually are

On x86-64 the MMU translates virtual addresses to physical ones by walking a four-level page table, with the standard leaf granularity being a 4 KiB page. The hardware also supports terminating that walk early: a page-table entry at the second level can map a 2 MiB region directly, and at the third level a 1 GiB region. Those larger mappings are hugepages. A 2 MiB hugepage is not 512 pages glued together in software — it is a single page-table entry describing 2 MiB of physically contiguous memory.

Two things follow. First, page tables get dramatically smaller: 2 GiB of guest RAM needs 524,288 leaf PTEs at 4 KiB granularity, or 1,024 at 2 MiB. Second, and more importantly at runtime, TLB coverage explodes. The TLB is a small hardware cache of address translations — a few thousand entries. With 4 KiB pages, a few thousand entries cover a few megabytes of working set, so a memory-heavy guest walks the page tables constantly. With 2 MiB pages, the same entries cover gigabytes. Fewer TLB misses means fewer page-table walks means less time spent on address translation and nothing else.

Under virtualization this argument compounds, because there are two levels of translation: the guest's own page tables plus the host's nested/extended page tables (EPT on Intel, NPT on AMD). A TLB miss that has to walk both is genuinely expensive. Backing guest RAM with host hugepages shortens the host-side half of that walk for every guest access.

For microVMs, the metric that matters is fault count

The TLB story is the textbook argument for hugepages, and it is real, but for a platform whose unit of work is "restore a microVM and get it serving in under 200ms" the more interesting property is a second-order one: the size of a page fault's payload.

A Firecracker snapshot restore does not read guest RAM up front. It maps the memory image and resumes, and each page materializes on first touch as a page fault — the mechanics are in /blog/firecracker-memory-file-mmap-explained. The guest's path from resume to ready touches a working set: some hundreds of megabytes of kernel text, libc, page cache, and whatever your template pre-warmed. Every 4 KiB page of that working set is one trap, one resolution, one retry.

Make the pages 2 MiB and the same working set is populated by 512x fewer faults. That figure is arithmetic — 2 MiB divided by 4 KiB — not a benchmark, and it is worth being pedantic about the difference. Fewer faults does not automatically mean proportionally less time, because each fault now moves 512 times as much data, and if that data comes off a local disk or the page cache you have mostly traded a large number of cheap operations for a small number of expensive ones. Where the trade becomes lopsided is when the per-fault fixed cost dominates the per-byte cost.

The restore-path payoff: when each fault is a round trip

That condition — per-fault fixed cost dominating — is exactly what happens when guest memory is demand-paged from somewhere that isn't local RAM. On PandaStack, snapshots are published to object storage so any host in the fleet can restore any template, and memory is streamed on demand: Firecracker is handed a userfaultfd, and our handler answers each guest page fault by fetching the relevant chunk over an HTTP Range GET. The full mechanics are in /blog/userfaultfd-explained.

In that architecture a fault is not a trap and a memcpy. It is: kernel posts an event, a user-space handler thread wakes, translates the guest address to a file offset, checks a local chunk cache, possibly issues an HTTPS request to object storage, waits, then installs the page with an ioctl and wakes the parked vCPU. The fixed cost of that sequence is large and mostly independent of how many bytes it delivers. Serving 2 MiB costs barely more than serving 4 KiB once the round trip is paid for.

So the streaming case is where hugepages stop being a micro-optimization. Every mitigation we already run on that path — 4 MiB fetch chunks so one Range GET amortizes across many pages, a bake-time prefetch trace that races ahead of the guest, a persistent per-host chunk cache — exists to reduce the number of faults that turn into round trips. Hugepages attack the same quantity from the other direction, at the hardware level: the guest simply generates far fewer faults in the first place, because each one covers 512x more address space.

The gotcha: hugepage-ness is a property of the snapshot

Here is the thing that is not in anybody's blog post and that I learned the way you usually learn these things. You do not enable hugepages on a running fleet. Hugepages are not a host-side runtime flag that a VM opts into at start time and forgets about. They are baked into the snapshot.

When Firecracker takes a snapshot of a guest whose memory is backed by hugetlbfs, the resulting memory image is a hugepage-backed image, and the snapshot metadata records that. On restore, Firecracker refuses to bring that image back through the ordinary file-backed path — pointing it at a plain mem_file_path is rejected. A hugepage snapshot can only be restored through the userfaultfd backend, because only there can the memory be re-established with the right page size under the VMM's control. That is not a configuration preference. It is a hard constraint in the restore path, and it produces a genuinely confusing failure: a snapshot you took successfully, on a host that is configured correctly, that simply will not load.

Three practical consequences fall out of that, and if you are building a platform on Firecracker you need all three:

1. Every snapshot taken from a hugepage-backed VM must be marked as such, and that marker must travel with the snapshot everywhere it goes — into your seed tarballs, into object storage alongside the memory image and state file, into whatever manifest your agents sync. A snapshot that has lost its marker is a snapshot that will fail to restore on the wrong path with an unhelpful error.
2. Every restore path must check the marker and force the UFFD backend when it is set — regardless of whether memory streaming is otherwise enabled for that host or template. In our agent, streaming is normally gated behind a flag; the hugepage marker overrides the flag, because there is no other legal way to load the image.
3. Enabling hugepages is a template re-bake, not a toggle. Existing snapshots were captured with 4 KiB backing and stay 4 KiB forever; they restore fine, they just don't get the benefit. A fleet mid-migration is running two classes of snapshot simultaneously, and both restore paths have to work. Plan the rollout as a re-bake campaign with a marker-aware restore path shipped first.

The UFFD handler consequence: serve whole 2 MiB pages

Because hugepage snapshots restore only through userfaultfd, the correctness of your UFFD handler becomes load-bearing in a new way. A handler written against 4 KiB assumptions will not work, and it will not fail cleanly.

When Firecracker hands off to the UFFD backend it sends, over the Unix socket, the userfaultfd file descriptor plus a description of the guest memory regions. That description includes each region's page size. A handler that hardcodes 4096 will compute the wrong faulting-page base address, install 4 KiB of data into a fault the kernel expects to be resolved with a full 2 MiB page, and then either have the ioctl rejected, or — worse — leave the faulting thread parked because the fault was never actually satisfied. The guest appears to hang mid-restore with no error anywhere. Debugging that from the outside is miserable.

The fix is mechanical: read the page size per region from the handoff, align the faulting address down to that size, and always install a full page's worth of bytes.

// Per-region metadata from Firecracker's UFFD handoff. page_size is in BYTES:
// 4096 for a normal snapshot, 2097152 (2 MiB) for a hugepage-backed one.
// NEVER hardcode this — the same handler binary must serve both.
type region struct {
	base     uint64 // guest-physical base address of this region
	size     uint64
	offset   uint64 // corresponding offset in the snapshot memory image
	pageSize uint64 // 4096 or 2 << 20 — from the handoff, not from a constant
}

func (h *handler) onFault(addr uint64) error {
	r := h.regionFor(addr)
	if r == nil {
		return fmt.Errorf("fault at %#x outside any registered region", addr)
	}

	// Align DOWN to this region's page size. With 2 MiB pages the kernel
	// expects the whole 2 MiB to be installed in one UFFDIO_COPY; a partial
	// copy leaves the faulting vCPU parked forever with no error surfaced.
	pageStart := addr &^ (r.pageSize - 1)
	srcOff := r.offset + (pageStart - r.base)

	buf, err := h.source.ReadAt(srcOff, r.pageSize) // local cache, else Range GET
	if err != nil {
		return err
	}
	if uint64(len(buf)) != r.pageSize {
		return fmt.Errorf("short read: got %d want %d", len(buf), r.pageSize)
	}

	// One ioctl covers 2 MiB of guest RAM instead of 4 KiB: 512x fewer of these.
	return h.uffdCopy(pageStart, buf)
}

One knock-on effect worth planning for: if your streamer fetches from object storage in chunks, the chunk size must be a multiple of the page size and chunk boundaries must be page-aligned, or a single 2 MiB fault straddles two chunks and needs two fetches. Our 4 MiB chunks divide evenly into 2 MiB pages, which is not a coincidence. Zero-page elision needs the same treatment: a 2 MiB page is only elidable if all 2 MiB of it is zero, so a hugepage build sees a somewhat lower elision rate than a 4 KiB build over the same image.

Host prerequisites: hugetlbfs, and reservation vs overcommit

Hugetlbfs pages come from a dedicated pool that the kernel manages separately from normal memory, and you have two ways to fill it. This choice matters more than it looks.

• vm.nr_hugepages — static reservation. The kernel carves that many hugepages out of physical RAM immediately and holds them. Reserved pages are guaranteed available and permanently unavailable to everything else, whether or not any VM is using them. On a host that also runs an agent, a container runtime, and a page cache you actually want warm, statically fencing off half your RAM is a real cost paid all the time for a benefit paid only sometimes.
• vm.nr_overcommit_hugepages — on-demand assembly. This sets a ceiling rather than a reservation: hugepages above the reserved count get assembled from free memory when something asks for them, and are returned to the general pool when freed. Nothing is carved away up front. This is what you want for a fleet with variable VM density, and it is what our cloud-init sets — to roughly half the host's RAM, as a ceiling, not a reservation.

# 1) Ceiling for on-demand hugepage assembly (NOT a static reservation).
#    Sized here as ~half of RAM expressed in 2 MiB pages.
HUGE_KB=$(awk '/Hugepagesize/{print $2}' /proc/meminfo)          # 2048
TOTAL_KB=$(awk '/MemTotal/{print $2}' /proc/meminfo)
sudo sysctl -w vm.nr_overcommit_hugepages=$(( TOTAL_KB / 2 / HUGE_KB ))
echo "vm.nr_overcommit_hugepages = $(( TOTAL_KB / 2 / HUGE_KB ))" \
  | sudo tee /etc/sysctl.d/60-pandastack-hugepages.conf

# 2) Inspect the pool. Read these in the right order:
#      HugePages_Total   - currently in the pool (reserved + assembled)
#      HugePages_Free    - in the pool, unused
#      HugePages_Rsvd    - promised to a mapping but not yet faulted in
#      HugePages_Surp    - "surplus": assembled on demand above nr_hugepages
#                          (this is the number that moves with overcommit)
#      Hugepagesize      - 2048 kB on x86-64 / arm64 with 4 KiB base pages
grep -E 'HugePages_|Hugepagesize|Hugetlb' /proc/meminfo

# 3) hugetlbfs mount that Firecracker's guest memory is allocated from.
sudo mkdir -p /dev/hugepages
mountpoint -q /dev/hugepages \
  || sudo mount -t hugetlbfs -o pagesize=2M nodev /dev/hugepages

# 4) Fragmentation check. Assembling a 2 MiB page needs 2 MiB of physically
#    CONTIGUOUS free memory. On a long-uptime host the low-order buddy
#    columns fill up and the order-9 column (2 MiB) empties out - that is
#    when on-demand allocation starts failing or stalling in compaction.
cat /proc/buddyinfo
#   Node 0, zone   Normal  4123  2011   980   412   190   88  41  17  6  2  0
#                                                             order-9 ^  ^ order-10
grep -E 'compact_stall|compact_fail' /proc/vmstat

That last point deserves emphasis because it is the failure mode that shows up in production rather than in testing. A 2 MiB hugepage requires 2 MiB of physically contiguous free memory. On a freshly booted host that is trivially available. On a host that has been up for weeks, churning through thousands of microVM creates and deletes, physical memory is fragmented, and the kernel may have to run memory compaction — moving pages around to manufacture a contiguous 2 MiB block — before it can satisfy the request. Compaction is not free and it is not bounded in any way you control. An on-demand hugepage allocation on a fragmented host can be slow, or can simply fail and fall back. If you adopt overcommit hugepages, watch compact_stall and compact_fail, and treat allocation failure as an expected state your code handles rather than an exception.

Configuring Firecracker

On the VMM side the setting lives on the machine configuration, alongside vCPU count and memory size — which is itself a hint that it is a property of the machine, and therefore of any snapshot taken from it, rather than of a particular run.

# Cold boot with 2 MiB-backed guest RAM. huge_pages sits on /machine-config
# next to vcpu_count and mem_size_mib -- it describes the MACHINE, which is
# why it survives into every snapshot taken from this VM.
curl --unix-socket /run/firecracker.sock -i \
  -X PUT 'http://localhost/machine-config' \
  -H 'Content-Type: application/json' \
  -d '{
        "vcpu_count": 2,
        "mem_size_mib": 4096,
        "huge_pages": "2M",
        "track_dirty_pages": false
      }'

# Snapshot it as usual...
curl --unix-socket /run/firecracker.sock -X PUT 'http://localhost/snapshot/create' \
  -H 'Content-Type: application/json' \
  -d '{"snapshot_type":"Full","snapshot_path":"/seed/vm.state","mem_file_path":"/seed/vm.mem"}'

# ...but you can NOT restore it like this. A hugepage-backed memory image
# through mem_file_path is rejected outright:
curl --unix-socket /run/firecracker.sock -X PUT 'http://localhost/snapshot/load' \
  -H 'Content-Type: application/json' \
  -d '{"snapshot_path":"/seed/vm.state","mem_backend":{"backend_type":"File","backend_path":"/seed/vm.mem"}}'
#   400 Bad Request -- hugepage snapshots require the Uffd backend

# The only legal restore: hand it a UFFD socket and back the memory yourself.
curl --unix-socket /run/firecracker.sock -X PUT 'http://localhost/snapshot/load' \
  -H 'Content-Type: application/json' \
  -d '{"snapshot_path":"/seed/vm.state","mem_backend":{"backend_type":"Uffd","backend_path":"/run/uffd.sock"},"resume_vm":true}'

A version note, since this is the kind of detail that eats an afternoon: if you drive Firecracker through an older SDK, the machine-config model may predate the huge_pages field and silently drop it from the serialized request. We hit exactly that and had to inject a raw PUT to /machine-config as an init hook rather than going through the typed model. If your hugepage VMs mysteriously come up with 4 KiB backing, check what your client actually put on the wire before you go blaming the kernel.

Honest tradeoffs

Hugepages are a real trade, and the costs land in places that are easy to miss until they're expensive.

• Granularity and waste — 4 KiB pages: a guest that touches one byte of a region makes 4 KiB resident. 2 MiB hugepages: the same byte makes 2 MiB resident. Sparse access patterns inflate real memory use, and on a dense host you may fit fewer VMs even though nothing changed about the guests.
• Page-cache sharing — 4 KiB pages: clean file-backed pages are shared across every VM restored from the same image, so density tracks divergence rather than VM count. 2 MiB hugepages: hugetlbfs memory does not participate in that sharing the same way, so the cross-VM sharing win from MAP_PRIVATE file-backed restore is reduced or gone.
• Dirty-page tracking — 4 KiB pages: dirty granularity is 4 KiB, so diff snapshots capture only genuinely changed pages. 2 MiB hugepages: dirty granularity becomes 2 MiB, so one modified byte marks 2 MiB dirty and diff snapshots inflate substantially.
• Allocation behaviour — 4 KiB pages: always available, no contiguity requirement, no compaction. 2 MiB hugepages: needs 2 MiB physically contiguous, can stall in compaction or fail on a fragmented long-uptime host.
• Ballooning — 4 KiB pages: virtio-balloon reclaims at 4 KiB granularity and works normally. 2 MiB hugepages: balloon reclaim interacts poorly with hugetlbfs backing, since freeing a 4 KiB sub-page of a hugepage returns nothing to the host. Assume ballooning and hugepages do not compose.
• Fault count on restore — 4 KiB pages: one fault per 4 KiB of working set. 2 MiB hugepages: one fault per 2 MiB, i.e. 512x fewer faults for the same working set — the reason to do any of this.
• Operational surface — 4 KiB pages: nothing to configure, every restore path works. 2 MiB hugepages: a sysctl, a hugetlbfs mount, a snapshot marker, a UFFD-forcing restore path, and a full template re-bake.

The dirty-tracking row is the one I would flag hardest for anyone doing incremental snapshots. If your architecture leans on diff snapshots — see /blog/firecracker-diff-snapshots-explained — moving dirty granularity from 4 KiB to 2 MiB can inflate your diffs by orders of magnitude for a guest that makes small scattered writes. Hugepages and diff snapshots pull in opposite directions and you should pick one deliberately.

Why 1 GiB pages are usually wrong for microVMs

The x86-64 MMU supports 1 GiB pages too, and the fault-count argument scales: one fault would cover a gigabyte. For microVMs it is nonetheless almost always the wrong call. A typical microVM has 1–4 GiB of RAM, so 1 GiB granularity means your guest memory is two to four pages total — any touch anywhere makes an entire gigabyte resident, which destroys density and makes memory usage a step function. Worse, 1 GiB pages generally cannot be assembled on demand at all: they need a gigabyte of physically contiguous memory, which in practice means boot-time reservation via the hugepagesz/hugepages kernel command line, permanently carving RAM away. And the marginal TLB benefit over 2 MiB is small, because 2 MiB entries already cover gigabytes of working set. 2 MiB is the sweet spot for this workload; 1 GiB belongs to single-tenant hosts running one enormous database, not to fleets of small VMs.

Where this sits in a real restore path

For context on what hugepages are optimizing against: on PandaStack there is no warm pool. Every sandbox create restores a baked snapshot on demand, landing at a p50 of about 179ms and a p99 around 203ms, of which the memory-load step is roughly 49ms. A first cold boot of a template that has no snapshot yet is about 3 seconds. Same-host forks land in 400–750ms because the memory and rootfs are already local; cross-host forks are 1.2–3.5s because the artifacts have to come over the network. That last gap is precisely the regime where fault count dominates and where cutting it by 512x is worth an operational rollout.

From the outside none of this is visible, which is the point — you ask for a sandbox and it exists:

from pandastack import Sandbox

# Restores a baked snapshot on demand -- whether that snapshot's memory is
# 4 KiB- or 2 MiB-backed, streamed from object storage or read from local
# disk, is a property of the seed and the host, not of this call.
sbx = Sandbox.create(template="base", ttl_seconds=900)

# The guest can see how ITS OWN kernel maps memory; host-side hugetlbfs
# backing is invisible to it, which is exactly the contract you want.
print(sbx.exec("grep -E 'AnonHugePages|Hugepagesize' /proc/meminfo").stdout)
print(sbx.exec("free -m").stdout)

sbx.destroy()

The summary: 2 MiB hugepages replace 512 page-table entries with one, which cuts TLB pressure and — the part that matters for snapshot restore — cuts the number of page faults needed to populate a working set by 512x. That is worth the most when each fault is expensive, which is to say when memory is being demand-paged over a network rather than read from a warm local page cache. The cost of admission is that hugepage-ness is a property of the snapshot: such snapshots restore only through the userfaultfd backend, so they must be marked, every restore path must honour the marker, your UFFD handler must serve whole pages at the region's declared page size, and enabling it at all means re-baking your templates. PandaStack's core is open source under Apache-2.0, so the marker plumbing, the UFFD handler, and the machine-config injection are all readable. If you want the surrounding mechanics first, start with /blog/firecracker-memory-file-mmap-explained for the file-backed path and /blog/userfaultfd-explained for the streaming one.