userfaultfd in Production: How the Pager Fails

There is a specific kind of outage that only happens to people who have written a userfaultfd handler. A sandbox reports healthy. The Firecracker process is running, its log is clean, the host has free CPU and free memory, and the guest has not moved in eleven minutes. Nothing crashed. Nothing errored. A vCPU thread went to fetch a page of its own RAM and simply never came back. If you have built a demand-paging layer under a VM and have not yet had this page you, you will. This post is the catalogue: what actually goes wrong when a userfaultfd handler runs in production, how each failure presents from the outside, and what control you get to put on it. /blog/userfaultfd-explained is the concept and /blog/write-a-userfaultfd-handler-for-firecracker is the build guide; this is the part where it is 3am.

The contract you signed

The setup is short and its consequences are not. Firecracker registers the guest's memory regions with a userfaultfd, then connects to a Unix socket you are listening on and hands you that file descriptor as ancillary data over SCM_RIGHTS, alongside a JSON description of the regions: base host virtual address, size, offset into the snapshot memory file, and the page size backing each one. From that moment, your process is the pager for that guest's physical RAM.

The steady state: a vCPU touches a page that has never been populated, the CPU traps, the kernel notices the address is registered with a userfaultfd, and instead of resolving the fault it queues a UFFD_EVENT_PAGEFAULT for you and puts the faulting thread to sleep. The thread wakes when — and only when — you install that page with UFFDIO_COPY or UFFDIO_ZEROPAGE. Read that last sentence again with an operator's eye. There is no timeout. There is no fallback pager. There is no error you can hand back that means "I could not produce this page, please fail the access." The kernel asked a question and it will block that thread until it gets an answer, and its patience is infinite. Everything in this post is a corollary of that one fact.

Why every bug here is the same bug

Normal software fails by producing wrong output, throwing, or dying. Those are all loud in their own way: a stack trace, a non-zero exit, a spike in a counter. A pager that stops answering produces none of those. The VM stays in the process table. The vCPU threads stay scheduled — technically they are asleep in a kernel wait, not spinning — so CPU usage goes to zero rather than to a hundred, which reads as "idle" on every dashboard you own. Firecracker itself is fine; from its perspective the guest is just not executing instructions, which is a thing guests are allowed to do. So the useful mental exercise, before you ship any of this, is to enumerate the ways you can stop answering and decide the control for each one in advance. Here is that table, in the order I have actually hit them.

• Handler death — Symptom: guest freezes mid-instruction, no OOPS, no Firecracker log line, VM still "running" with zero CPU. Control: supervise the handler and make handler death mean VM death; never let a VM outlive its pager.
• Backing-store stall — Symptom: a subset of VMs go unresponsive during an object-storage incident and recover, or don't, with no error anywhere. Control: retries with backoff inside the fault path, a hard per-fault deadline that escalates to killing the VM, and a local chunk cache so one blip isn't re-paid per page.
• Page-size mismatch — Symptom: restore of a re-baked template hangs immediately, or UFFDIO_COPY returns EINVAL on the very first fault. Control: read page_size per region from the handshake; never assume a global 4 KiB.
• Teardown race — Symptom: errors that look catastrophic (ENOENT, ESRCH, EAGAIN, EBADF) but are just a VM that already left. Control: treat them as expected at shutdown; distinguish "the mapping is gone" from "I am broken".
• Partial fill — Symptom: no symptom for hours, then an inexplicable segfault or filesystem corruption inside a guest, on a host you have already forgotten about. Control: length-check every read; never call UFFDIO_COPY on a buffer you did not completely fill.
• Fault storm — Symptom: latency, not failure — restores get slower under concurrency and no single component looks busy. Control: a worker pool, per-page single-flight, and prefetch of the known-hot working set.
• Handoff failure — Symptom: snapshot load fails with a confusing memory-backend error, or succeeds and then hangs on the first fault. Control: listen before you load, consume the region message and its control message in one recvmsg, validate you got exactly one fd.

Failure 1: the handler dies

This is the fundamental one and everything else is a variation of it. Your handler gets OOM-killed, panics on a nil map, deadlocks on a mutex someone held across an HTTP call, or is restarted by a well-meaning deploy that treats it as a stateless daemon. The guest does not notice immediately, which is the cruel part: it only freezes at the next fault on an unpopulated page. A guest that has already touched its working set can look perfectly healthy for minutes after its pager is gone, then wedge the instant some process allocates fresh memory. From outside, the fingerprint is consistent: the Firecracker process exists, its threads are in an uninterruptible-ish sleep, host CPU for that VM is flat zero, the guest console has no new output, and there is nothing in firecracker.log because from the VMM's point of view nothing happened. Network probes to the guest time out rather than refuse — the stack is loaded, it just isn't running. Here is the triage I actually run, and it answers the question in under a minute.

# A sandbox reports "running" and is doing nothing. Decide, fast, whether the
# pager stopped answering or the guest is genuinely idle.

VM=8f3c9a12-...                                  # sandbox id
FC=$(pgrep -f "firecracker .*${VM}")
UH=$(pgrep -f "uffd-handler .*${VM}")            # the pager process

# 1. Is the pager alive at all? Most of the time this IS the answer.
if [ -z "$UH" ]; then
  echo "HANDLER GONE for ${VM} -- this guest will never take another fault"
fi

# 2. Where are the vCPU threads parked? Every vCPU asleep in the same place,
#    not moving between samples, is the fingerprint of an unanswered fault.
for t in /proc/${FC}/task/*; do
  printf '%s %-16s %s\n' "${t##*/}" "$(cat ${t}/comm)" "$(cat ${t}/wchan 2>/dev/null)"
done
# Kernel stacks need root and kptr_restrict=0; exact wchan/stack symbols vary
# by kernel version, so read them as "stuck here", not as a stable API:
#   sudo cat /proc/${FC}/task/*/stack

# 3. Is the handoff socket still connected, or did one end walk away?
sudo ss -xp | grep -E "uffd-${VM}|firecracker" || echo "no peer on the uffd socket"

# 4. Has this handler served a fault recently? "Never" and "not since 04:12"
#    are completely different incidents.
curl -s localhost:9100/metrics | grep -E 'uffd_(faults_total|bytes_installed_total|source_errors_total|last_fault_age_seconds)'

The policy that follows is not subtle: the handler's lifetime must strictly enclose the VM's. If the handler exits for any reason, the VM must be reaped, and the orchestrator must recreate the sandbox somewhere else. Not "restart the handler" — a fresh handler cannot adopt a userfaultfd it never received, and the fd died with the process that held it. There is no reattach. A VM whose pager is gone is already dead; the only question is whether your control plane knows it yet.

Two practical corollaries. First, never let the fault path allocate unboundedly, because your handler being the process the OOM killer picks is a self-inflicted hang. Second, never take a lock in the fault path that any other part of your process can hold across I/O; a lock-order inversion in a pager is indistinguishable from a dead pager, and it happens under load, which is to say in front of customers.

Failure 2: the backing store is having a day

Once the fault source is object storage rather than a local file, you have put an HTTP request on the critical path of a page fault. That is a defensible trade — it is exactly what lets a host restore a template whose multi-gigabyte memory image it has never held — but it means every failure mode of HTTP is now a failure mode of memory. A 503, a connection reset, a DNS blip, a token that expired mid-restore: each one arrives as an error in a code path that has no way to report an error.

I want to be very explicit here, because it is the thing people get wrong on the first try: returning an error from your fault handler does not fail the guest's memory access. There is no mechanism for that. Your options are to install a page, or to not install a page, and "not install a page" means "this vCPU sleeps until the heat death of the universe or the next reboot, whichever you schedule first." You cannot fail a fault. You can only be slow, or escalate to killing the VM. So the fault path needs three things layered together: retry with backoff inside the fault, a bounded total deadline whose expiry escalates rather than returns, and a cache so a transient blip is paid once instead of once per page.

// There is no "this fault failed" signal. The only two outcomes are
// "installed" and "the vCPU is still asleep", so the escalation for a
// blown budget is not an error return -- it is tearing down the VM.
const (
	attemptTimeout = 2 * time.Second  // one try against the object store
	faultDeadline  = 20 * time.Second // total, across every retry
	maxBackoff     = 2 * time.Second
)

func (h *Handler) serveFault(uffd int, addr uint64) error {
	r, ok := h.regionFor(addr)
	if !ok {
		return fmt.Errorf("fault %#x outside every registered region", addr)
	}
	pageSize := r.PageSizeKiB << 10
	base := addr &^ (pageSize - 1) // kernel reports the faulting BYTE
	off := r.Offset + (base - r.BaseHostVirtAddr)

	start := time.Now()
	backoff := 25 * time.Millisecond

	for attempt := 1; ; attempt++ {
		ctx, cancel := context.WithTimeout(context.Background(), attemptTimeout)
		page, err := h.source.PageAt(ctx, off, pageSize) // cache, then HTTP range
		cancel()

		if err == nil {
			h.metrics.FaultSeconds.Observe(time.Since(start).Seconds())
			if page == nil { // known-zero chunk: no bytes ever move
				return zeroPage(uffd, base, pageSize)
			}
			return copyPage(uffd, base, page)
		}

		h.metrics.SourceErrors.WithLabelValues(classify(err)).Inc()

		if time.Since(start) >= faultDeadline {
			// Escalate. A dead VM is strictly better than a hung one,
			// because a dead one is visible to the control plane and gets
			// recreated. Do NOT "give up and return" -- that is the hang.
			h.fatalf("fault %#x at off %#x: source unavailable after %d attempts in %v: %v",
				addr, off, attempt, time.Since(start), err)
		}

		h.metrics.Retries.Inc()
		time.Sleep(jitter(backoff))
		if backoff < maxBackoff {
			backoff *= 2
		}
	}
}

// fatalf kills the VM and exits the handler. The supervisor treats handler
// exit as VM death; the control plane recreates the sandbox elsewhere.
func (h *Handler) fatalf(format string, args ...any) {
	h.log.Error(fmt.Sprintf(format, args...))
	h.killVM() // SIGKILL the firecracker process, then leave
	os.Exit(1)
}

The deadline is the part that requires a decision rather than a technique. Twenty seconds is not a magic number; it is an assertion that a guest stalled longer than that is more usefully dead. Pick it against your own recovery path — if your control plane can recreate a sandbox in a couple of seconds from a snapshot restore, a long deadline is pure downtime, and if recreation means losing state a user cares about, you want to wait longer and alert. What you must not do is leave the deadline as infinity by omission, which is the default if you write the obvious loop. The cache is the other half. Faults arrive at 4 KiB granularity but a sane implementation fetches a much larger chunk and keeps it, so a network blip during one chunk's fetch does not repeat for the thousand-odd pages inside it. On PandaStack that is a persistent per-host chunk cache keyed by snapshot generation: the first restore of a template on a host pays object-storage latency for its working set once, every later restore reads local disk, and a re-bake produces a new generation so stale chunks self-invalidate instead of quietly serving memory from a previous build.

Failure 3: page-size mismatch

Guest memory is not always backed by 4 KiB pages. If a snapshot was taken from a hugepage-backed guest, its regions are 2 MiB hugetlbfs pages, and UFFDIO_COPY must cover a whole aligned 2 MiB page. Serving 4 KiB into a 2 MiB region does not partially work; the ioctl rejects it, and if your error handling papers over that, the fault is never satisfied and you are back to a hang. What makes this one nasty operationally is the delay between cause and effect. A handler that hardcodes 4 KiB is correct for as long as nobody enables hugepages. Then someone re-bakes a template with hugepage backing — a reasonable thing to do, since one fault then covers 2 MiB instead of 4 KiB — and every restore of that one template hangs while every other template is fine. The page size is a property of the snapshot, carried in the handshake, and your only correct move is to read it per region.

// PageSizeKiB comes from the handshake, PER REGION: 4 for normal pages,
// 2048 for 2 MiB hugetlbfs. Older Firecracker builds omit it -- absent
// means 4 KiB. Never derive it from os.Getpagesize(); the host's page size
// says nothing about how this snapshot was baked.
func (h *Handler) install(uffd int, r *Region, addr uint64) error {
	pageSize := r.PageSizeKiB << 10
	if pageSize == 0 || pageSize&(pageSize-1) != 0 {
		return fmt.Errorf("region page size %d is not a power of two", pageSize)
	}

	// Align DOWN to the page THIS REGION is backed by. For hugetlb both the
	// destination and the length must be 2 MiB-aligned or UFFDIO_COPY fails
	// with EINVAL -- and an EINVAL you swallow is a vCPU you never wake.
	base := addr &^ (pageSize - 1)
	end := r.BaseHostVirtAddr + r.Size
	if base < r.BaseHostVirtAddr || base+pageSize > end {
		return fmt.Errorf("page %#x+%#x straddles region [%#x,%#x)",
			base, pageSize, r.BaseHostVirtAddr, end)
	}

	off := r.Offset + (base - r.BaseHostVirtAddr)
	buf, err := h.source.PageAt(context.Background(), off, pageSize)
	if err != nil {
		return err
	}
	if buf == nil {
		// UFFDIO_ZEROPAGE is not supported on hugetlb regions on every
		// kernel -- check your own before relying on it, and fall back to
		// a COPY from a shared pre-zeroed 2 MiB buffer.
		if pageSize != 4<<10 {
			return copyPage(uffd, base, h.zeroBuf[:pageSize])
		}
		return zeroPage(uffd, base, pageSize)
	}
	if uint64(len(buf)) != pageSize {
		// Never install a partly filled page. See failure 5.
		return fmt.Errorf("short page at off %#x: got %d want %d", off, len(buf), pageSize)
	}
	return copyPage(uffd, base, buf)
}

Two operational notes that go with this. Hugepage-ness travels with the snapshot, so every restore path in your system — not just the streaming one — has to know: Firecracker will only restore a hugepage snapshot through the UFFD backend, and passing a plain memory-file path is rejected. We carry a marker file next to the snapshot artifacts so the restore code can tell before it commits to a path. And exact ioctl behaviour around hugetlb and ZEROPAGE has moved between kernel versions, so verify against the kernel and Firecracker docs for the versions you actually run rather than trusting a blog post, including this one.

Failure 4: races at teardown

VMs go away. They go away while faults are in flight, while a prefetch worker is mid-fetch, and while a chunk is being installed. The kernel is not going to coordinate this for you, so your handler will observe events for regions that no longer exist and ioctls that fail for reasons that are, in context, entirely normal. The specific returns to expect: ENOENT when the destination mapping has been torn down under you; ESRCH when the process that owned the address space is gone; EAGAIN when the copy was interrupted mid-range, with the byte count of what actually landed sitting in the ioctl's own struct; EBADF if you race your own close; and a POLLHUP on the descriptor when Firecracker exits, which is the clean, expected end of a handler's life. There are also UFFD_EVENT_REMOVE and UFFD_EVENT_UNMAP events telling you a range was discarded — those must invalidate cached content for the range rather than be ignored, or you will re-serve freed memory into a running kernel.

None of these should be fatal by default, and none of them should be silent either. The rule I use: at shutdown, mapping-gone errors are expected and get counted, not logged as errors; before shutdown, the exact same errno is a genuine bug and should be loud. That means your handler needs to know that a teardown has started — a flag set when you initiate the kill, or when you observe the VMM exit — so it can classify. Handlers that treat every ENOENT as catastrophic produce alert fatigue, and handlers that treat every ENOENT as fine will one day swallow the one that meant something.

Failure 5: partial reads, and the weekend they cost

This is the only failure in this post that is not a hang, and it is worse. Suppose your source does a read that returns fewer bytes than requested — a truncated HTTP response, a short read on a file, a range request that got a partial body because a connection died and you did not check the length. If you install that buffer with UFFDIO_COPY anyway, the guest gets a page whose tail is whatever was in your buffer. Not an error. Not a fault. Just wrong memory, installed atomically and permanently, in a machine that is now running. What that looks like operationally is nothing, for hours. Then a process in the guest segfaults for no reason. Or a JIT emits garbage. Or a filesystem check finds corruption on a disk that is fine. The symptom appears arbitrarily far from the cause, in a different subsystem, on a host that has since restarted, and it looks so much like a bug in the customer's code that you will spend a long time believing it is. I have burned a weekend on exactly this class of bug and the fix, once found, was four lines.

The controls are boring and absolute. Use full reads — io.ReadFull, or a loop, never a bare Read. Assert the returned length equals the requested page size before the ioctl, and treat a mismatch as a hard error that goes through the same escalation path as a source failure. Check HTTP status and Content-Length on range requests and reject a 200 where you asked for a 206, because a 200 means you got the whole object where you expected a slice. Verify chunk integrity when it is cheap; a checksum recorded at bake time turns silent corruption into a loud fetch error. And never reuse a buffer across faults without fully overwriting it — a recycled buffer with stale contents is the same bug with better performance characteristics.

Failure 6: fault storms and head-of-line blocking

The naive handler is one loop: read an event, resolve it, install it, read the next event. With a local memory file, that is genuinely fine — the resolve step is a page-cache hit. With a network source it is a design flaw, because every fault in the VM now queues behind the slowest fetch in the VM. One vCPU's cache miss stalls the other vCPUs' faults, including ones that would have been instant cache hits. This does not present as a failure; it presents as restores that get mysteriously slower under concurrency while no component looks busy, which is a much more expensive thing to debug than a crash. Three things fix it, and they compose. A worker pool: the reader thread does nothing but drain the descriptor and dispatch, while workers resolve and install concurrently. Per-page single-flight: several vCPUs faulting into the same chunk should produce one fetch, not N — and this is precisely why EEXIST from UFFDIO_COPY has to be treated as success rather than as an error, because concurrent installs of the same page are a normal outcome of the design. And prefetch: a template touches roughly the same hot set on every restore, so record it at bake time and replay it in the background the instant restore begins, racing ahead of the guest so most faults land on chunks that are already local.

Worth stating the limit plainly: prefetch and caching hide latency, they do not remove it. The first restore of an unfamiliar template on a cold host is still bounded by object-storage latency for its working set, and a cache miss in the fault path is a network round trip where a local mmap would have done a memory read. For reference, the local path on our fleet — mmap of a memory file that is already on disk — lands the memory-load step around 49ms inside a create that is p50 179ms end to end. Streaming is what you reach for when the alternative is downloading gigabytes first, not because it beats a warm local file.

Failure 7: the handoff itself

Before any of the above can go wrong, the handshake has to work, and it has an ordering requirement people trip over exactly once. Your handler must be listening on the Unix socket before you issue the snapshot load. Firecracker is the client here, not the server. If the socket is not there when it connects, the load fails with an error about the memory backend that reads like a snapshot problem and is actually a startup-sequencing problem, and you will look in the wrong place for twenty minutes.

The second trap is the SCM_RIGHTS mechanics. The region JSON and the file descriptor arrive together in a single message; the fd travels as a control message, not as bytes. If you read the body with a plain read and then look for the descriptor, it is gone — the kernel had one chance to install it in your process and you declined. Use recvmsg with a control buffer sized for exactly one fd, parse the control message, and validate that you got exactly one. Then fully consume and parse the region message: a handler that reads a partial JSON body and proceeds with a truncated region list will happily serve faults for the regions it knows about and hang on the first fault into the one it dropped, which on x86 is usually the region above the PCI hole and therefore not the one your smoke test touches.

One more environmental precondition that belongs in your startup checks rather than in an incident: unprivileged userfaultfd is gated by a sysctl on most distributions. Discovering that a host has it disabled during a customer's restore is worse in every way than asserting it when the agent starts and refusing to advertise streaming capability if it fails.

What to export, and the one alert that matters

Because the failure mode is silence, instrumentation is not optional here in the way it might be for an ordinary service. You are not trying to measure performance; you are trying to make an absence visible. The set I would not run without:

• Faults served, as a counter, labelled by kind — copy versus zero-page. Zero-page ratio is also a decent canary for a broken non-zero index.
• Bytes installed, as a counter. Divergence between bytes installed and bytes fetched tells you the cache is doing its job — or isn't.
• Fault latency, as a histogram, not an average. The tail is the whole story: p50 will look wonderful while a small fraction of faults are eating twenty seconds.
• Source errors, as a counter labelled by class — HTTP status, timeout, connection reset, short read. This is your early warning that an object-storage incident is about to become a VM incident.
• Retry count, so you can see the fault path absorbing a degraded backend before customers do.
• Cache hit rate, per host. A host whose hit rate collapsed is a host that just came up cold, or a host whose cache generation changed under it after a re-bake.
• Handler liveness, as a per-VM gauge. Not "is the process up" — is this specific handler still associated with this specific VM.

And then the one alert that actually catches the outage this post is about: time since the last fault served, on a VM that should be making progress. Not error rate — there are no errors. Not CPU — it is zero, which is what idle looks like. Not fault latency — the fault that hangs never completes, so it never lands in your histogram at all, and a hung pager makes your latency graphs look better, which is the most sinister property of the whole system.

Export a per-VM gauge of seconds since the last fault was served, and pair it with a signal that the guest is supposed to be doing something — an active session, a running deploy, a health probe that used to pass and now times out. A guest that has genuinely finished paging in its working set legitimately serves no faults for long stretches, so the metric alone is noisy. It is the conjunction that is diagnostic: nothing paged, nothing progressing, everything nominally alive.

The posture this asks for

None of this is an argument against userfaultfd. Streaming memory on demand is what lets a host restore a template it has never held without a multi-gigabyte download standing between a user and their sandbox, and it is the only supported restore path for hugepage-backed guests. It earns its place. But it moves a piece of the kernel's job into a process you wrote and deploy on a Tuesday, and the kernel's contract with that process has no error channel and no timeout. So operate it accordingly. Couple lifetimes: handler death is VM death, enforced by the supervisor. Bound everything: per-attempt timeouts, a total fault deadline, and an escalation that kills rather than waits. Assume the source will fail and make the failure cost one fetch instead of a thousand. Read the page size from the handshake, check every length before you install, and classify teardown errno instead of panicking on it. Instrument for absence, and alert on silence.

You cannot fail a page fault. You can only be slow, or decide that this VM is over. Choosing not to decide is choosing to hang.

The ioctls in a userfaultfd handler are the easy part — a few hundred lines, and /blog/write-a-userfaultfd-handler-for-firecracker walks through them. The engineering is entirely in what happens when the bytes are late, wrong, or short. PandaStack's core is open source under Apache-2.0, so the real handler, the chunk cache, and the prefetch trace are readable rather than reconstructed from a post, including the parts where the comments are visibly written by someone who had just been paged.