How to Write a userfaultfd Handler for Firecracker

Firecracker will let you back a guest's entire physical RAM with your own code. You create a Unix socket, start the VMM in UFFD mode, and during PUT /snapshot/load Firecracker connects to that socket and hands you a userfaultfd file descriptor plus a description of the guest's memory regions. From that moment you own every page fault the guest takes: the kernel parks the faulting vCPU and asks your process what belongs at that address. This post is the build guide for that handler — the handshake, the event loop, the ioctls, the page-size traps that will bite you, and how the whole thing turns into a streaming restore once the fault source can be a network object instead of a local file. If you want the conceptual version first, /blog/userfaultfd-explained covers what userfaultfd is and why it exists; this one is the part where you write code that a vCPU is blocked on.

Why bother: restore cost should track pages touched, not file size

A Firecracker snapshot is a tiny state file (vCPU registers, interrupt controller, virtio queues) plus vm.mem — a flat, byte-for-byte dump of the guest's physical RAM. A 4 GiB guest produces a 4 GiB memory file. If that file is already sitting on local NVMe, the default restore path is genuinely excellent: Firecracker mmaps it MAP_PRIVATE and resumes, the mapping is O(1) in the size of the region, and pages fault in lazily from the page cache. On PandaStack that memory-load step lands around 49ms inside a create that is p50 179ms end to end. There is nothing to fix there. (/blog/firecracker-memory-file-mmap-explained has the full mechanics.)

The problem appears the moment the memory file is not local. In a multi-host fleet you publish snapshots to object storage so any host can restore any template. The naive path is: download four gigabytes, then map it, then resume. That download is dead time on the critical path, and most of those bytes are pages this particular boot will never touch. A guest that reaches ready by touching a few hundred megabytes of working set just paid for 4 GiB of transfer to use maybe a tenth of it.

userfaultfd deletes the wait. Instead of pointing Firecracker at a file, you point it at a socket. The guest starts running against memory that does not exist yet, and each page materializes when — and only when — the guest reaches for it. Restore cost stops being a function of image size and becomes a function of working set.

The handshake: receiving a file descriptor over SCM_RIGHTS

The ordering matters and it is the first thing people get wrong: your handler must be listening before Firecracker tries to load the snapshot. Create the Unix socket, accept in a goroutine, then issue the snapshot/load request. If the socket isn't there, Firecracker fails the load and you get a confusing error about the memory backend rather than about your startup ordering.

Firecracker is the client. It connects and sends exactly one message: a JSON body describing the guest memory regions, with the userfaultfd itself attached as ancillary data via SCM_RIGHTS. File descriptors cannot be sent as bytes — they are kernel objects, so they travel in the control-message channel of a Unix socket and the kernel installs a new descriptor number in the receiving process. Each region tells you three things you need and one you must not ignore: the base host virtual address (an address in Firecracker's address space, which is also where faults will be reported), the size, the offset of those bytes in the snapshot memory file, and the page size backing that region.

package uffd

import (
	"encoding/json"
	"fmt"
	"net"

	"golang.org/x/sys/unix"
)

// Region is one guest memory region exactly as Firecracker describes it on
// the handshake. BaseHostVirtAddr is an address in FIRECRACKER's address
// space -- fault addresses arrive in the same space. Offset is where those
// bytes live in the snapshot memory file (vm.mem).
type Region struct {
	BaseHostVirtAddr uint64 `json:"base_host_virt_addr"`
	Size             uint64 `json:"size"`
	Offset           uint64 `json:"offset"`
	// 4 for normal pages, 2048 for 2 MiB hugepages. Older Firecracker
	// builds omit the field entirely -- treat absent as 4 KiB.
	PageSizeKiB uint64 `json:"page_size_kib"`
}

// Accept must be called BEFORE PUT /snapshot/load. Firecracker is the client:
// it connects, sends the region JSON, and attaches the uffd via SCM_RIGHTS.
func Accept(sockPath string) (uffd int, regions []Region, err error) {
	l, err := net.Listen("unix", sockPath)
	if err != nil {
		return -1, nil, err
	}
	defer l.Close()

	c, err := l.Accept()
	if err != nil {
		return -1, nil, err
	}
	conn := c.(*net.UnixConn)
	defer conn.Close()

	body := make([]byte, 64<<10)           // region JSON
	oob := make([]byte, unix.CmsgSpace(4)) // room for exactly one fd
	n, oobn, _, _, err := conn.ReadMsgUnix(body, oob)
	if err != nil {
		return -1, nil, fmt.Errorf("handshake read: %w", err)
	}

	scms, err := unix.ParseSocketControlMessage(oob[:oobn])
	if err != nil || len(scms) == 0 {
		return -1, nil, fmt.Errorf("no SCM_RIGHTS control message: %v", err)
	}
	fds, err := unix.ParseUnixRights(&scms[0])
	if err != nil || len(fds) != 1 {
		return -1, nil, fmt.Errorf("expected exactly one fd: %v", err)
	}

	if err := json.Unmarshal(body[:n], &regions); err != nil {
		unix.Close(fds[0])
		return -1, nil, fmt.Errorf("region json: %w", err)
	}
	for i := range regions {
		if regions[i].PageSizeKiB == 0 {
			regions[i].PageSizeKiB = 4
		}
	}

	// From here on the uffd is ours. Nothing else in the system will
	// service these faults. If we stop, the guest stops.
	return fds[0], regions, nil
}

Two details worth internalizing. First, you must read the body and the control message in the same recvmsg call — the fd rides along with the bytes, and if you read the JSON with a plain Read you will have thrown the descriptor away. Second, the regions arrive sorted and contiguous-ish but you should not assume anything: build a lookup structure (a sorted slice plus binary search is plenty; there are usually two or three regions, because of the PCI hole) and validate that every fault address lands inside exactly one of them.

The event loop: poll, read uffd_msg, UFFDIO_COPY

Now the steady state. The userfaultfd is a readable descriptor: you poll it, and each readable event is a fixed-size struct uffd_msg (32 bytes on every architecture you care about). Byte 0 is the event type. For UFFD_EVENT_PAGEFAULT the faulting address sits at offset 16 and the flags at offset 8. You translate the address to a snapshot-file offset, get the bytes, and install them with the UFFDIO_COPY ioctl, which atomically places the page and wakes the parked vCPU.

The ioctl numbers are the standard _IOWR encodings with magic 0xAA: UFFDIO_COPY is 0xc028aa03 (a 40-byte struct) and UFFDIO_ZEROPAGE is 0xc020aa04 (32 bytes). Hardcoding them is normal practice in Go handlers; just leave a comment so the next person doesn't think you generated them with a random number generator.

const (
	uffdMsgSize        = 32 // sizeof(struct uffd_msg)
	uffdEventPagefault = 0x12
	uffdEventRemove    = 0x15
	uffdEventUnmap     = 0x16

	// _IOWR(UFFDIO=0xAA, _UFFDIO_COPY=0x03, struct uffdio_copy /*40B*/)
	ioctlUFFDIOCopy = 0xc028aa03
)

type uffdioCopy struct {
	Dst, Src, Len, Mode uint64
	Copy                int64 // bytes copied, or -errno
}

// Serve owns the uffd until the guest goes away.
func (h *Handler) Serve(uffd int) error {
	defer unix.Close(uffd)
	pfd := []unix.PollFd{{Fd: int32(uffd), Events: unix.POLLIN}}
	var msg [uffdMsgSize]byte

	for {
		if _, err := unix.Poll(pfd, -1); err != nil {
			if err == unix.EINTR {
				continue
			}
			return err
		}
		if pfd[0].Revents&unix.POLLHUP != 0 {
			return nil // Firecracker exited; the guest is gone. Clean stop.
		}

		n, err := unix.Read(uffd, msg[:])
		if err == unix.EAGAIN {
			continue
		}
		if err != nil {
			return err
		}
		if n != uffdMsgSize {
			return fmt.Errorf("short uffd_msg: %d bytes", n)
		}

		switch msg[0] {
		case uffdEventPagefault:
			addr := binary.LittleEndian.Uint64(msg[16:24])
			if err := h.fault(uffd, addr); err != nil {
				// A fault we never answer is a vCPU that never runs again.
				return fmt.Errorf("fault %#x: %w", addr, err)
			}
		case uffdEventRemove, uffdEventUnmap:
			start := binary.LittleEndian.Uint64(msg[8:16])
			end := binary.LittleEndian.Uint64(msg[16:24])
			h.forget(start, end) // balloon/madvise: this range is now stale
		}
	}
}

// fault resolves one faulting address into one whole page and installs it.
func (h *Handler) fault(uffd int, addr uint64) error {
	r, ok := h.regionFor(addr)
	if !ok {
		return fmt.Errorf("address outside every registered region")
	}
	pageSize := r.PageSizeKiB << 10        // 4096, or 2 MiB for hugepages
	base := addr &^ (pageSize - 1)         // align DOWN; the kernel reports
	                                       // the exact faulting byte, not the page
	off := r.Offset + (base - r.BaseHostVirtAddr)

	page, err := h.source.PageAt(off, pageSize) // local file OR the network
	if err != nil {
		return err
	}
	if page == nil { // known-zero: nothing to transfer
		return zeroPage(uffd, base, pageSize)
	}
	return copyPage(uffd, base, page)
}

func copyPage(uffd int, dst uint64, page []byte) error {
	c := uffdioCopy{
		Dst: dst,
		Src: uint64(uintptr(unsafe.Pointer(&page[0]))),
		Len: uint64(len(page)),
	}
	defer runtime.KeepAlive(page) // do not let the GC move this out from under the kernel

	for {
		_, _, errno := unix.Syscall(unix.SYS_IOCTL,
			uintptr(uffd), ioctlUFFDIOCopy, uintptr(unsafe.Pointer(&c)))
		switch errno {
		case 0:
			return nil
		case unix.EAGAIN:
			// Interrupted mid-copy. c.Copy holds how much actually landed;
			// advance past it and retry the remainder.
			if c.Copy > 0 {
				done := uint64(c.Copy)
				c.Dst += done
				c.Src += done
				c.Len -= done
			}
			c.Copy = 0
			continue
		case unix.EEXIST:
			return nil // someone already filled it; benign race, not an error
		default:
			return errno
		}
	}
}

Zero pages: UFFDIO_ZEROPAGE

A large fraction of a snapshotted guest's RAM is zeros — pages the guest OS zeroed and never used. Copying zeros through your handler is a waste of a memcpy at best and a network round trip at worst. UFFDIO_ZEROPAGE installs an all-zero page with no source buffer at all. It is the single cheapest thing your handler can do, and knowing which pages are zero up front is where a lot of the real-world win lives.

// _IOWR(UFFDIO=0xAA, _UFFDIO_ZEROPAGE=0x04, struct uffdio_zeropage /*32B*/)
const ioctlUFFDIOZeropage = 0xc020aa04

type uffdioZeropage struct {
	Start, Len, Mode uint64
	Zeropage         int64
}

func zeroPage(uffd int, dst, size uint64) error {
	z := uffdioZeropage{Start: dst, Len: size}
	for {
		_, _, errno := unix.Syscall(unix.SYS_IOCTL,
			uintptr(uffd), ioctlUFFDIOZeropage, uintptr(unsafe.Pointer(&z)))
		switch errno {
		case 0, unix.EEXIST:
			return nil
		case unix.EAGAIN:
			continue
		default:
			// Note: on hugepage-backed regions ZEROPAGE is not supported on
			// all kernels -- fall back to COPY from a zeroed 2 MiB buffer.
			return errno
		}
	}
}

REMOVE and UNMAP: don't serve stale data

If you register with the right features, the kernel also tells you when a registered range is madvise(MADV_DONTNEED)'d or unmapped. This is not decoration. A guest with a virtio balloon driver will hand memory back to the host, the host will discard those pages, and the guest will later fault on them again — expecting zeros, because that memory was freed. If your handler cheerfully re-serves the original snapshot contents for that range, you have just resurrected freed memory inside a running kernel. That bug does not look like a memory bug; it looks like the guest going insane an hour later. Handle UFFD_EVENT_REMOVE and UFFD_EVENT_UNMAP by invalidating any cached content for the range and marking it zero-on-next-fault.

Getting it right: the four things that will bite you

Page size and alignment

The kernel reports the exact faulting address, not the page base. Every install must be aligned down to the region's page size, and the length must be exactly one page (or a multiple of it). This is trivial for 4 KiB and a genuine trap for hugepages: if a region is backed by 2 MiB hugetlbfs pages, you must serve a whole aligned 2 MiB page. A 4 KiB UFFDIO_COPY into a hugepage region fails, and if you paper over the failure you get a hang. Read page_size per region and never assume a global constant — the hugepage-ness of a snapshot is a property of how it was baked, not of your handler's preferences.

Partial copies and EAGAIN

UFFDIO_COPY can return EAGAIN having copied part of the range, with the byte count in the struct's copy field. Handle it as a loop, not an error. EEXIST means the page is already present — another thread beat you to it, or the guest's own activity resolved it — and should be treated as success. Treating EEXIST as fatal is the kind of bug that only shows up under concurrency, which is to say in production.

Don't block the loop on slow I/O

A single-threaded loop that reads an event, does a synchronous network fetch, then reads the next event will serialize every fault in the VM behind the slowest one. With a local file that's fine. With object storage it is not: one vCPU's cache miss stalls the other vCPUs' faults behind it. The fix is the ordinary one — a small worker pool that resolves and installs faults concurrently while the reader thread keeps draining the descriptor. Faults for the same page can arrive from multiple threads at once, which is exactly why EEXIST is not an error. Keep a per-page single-flight so N concurrent faults on one chunk produce one fetch, not N.

What happens when your handler dies

This is the part to take seriously. If your handler crashes, deadlocks, or gets OOM-killed, nothing services the next fault. The guest does not crash. It does not panic. The faulting vCPU simply never returns from its memory access, and the VM freezes mid-instruction — often after appearing perfectly healthy for a while, because it hadn't touched a missing page yet. A page-fault handler that deadlocks is the quietest outage you will ever run: no error, no exit code, no log line, just a machine that stopped having opinions.

So design for it. Never take a lock in the fault path that anything else in your process can hold across I/O. Put a hard timeout on every fetch and decide, in advance and in writing, what you do when it expires — most systems should fail loudly (kill the VM, surface an error, let the orchestrator recreate it) rather than serve wrong bytes or hang forever. Log the faulting address and region on every error path, because that's the only breadcrumb you'll get. And if your handler is a separate process, make its lifetime strictly enclose the VM's: if the handler exits, the VM must be reaped too, or you'll accumulate frozen ghosts that look like running instances in every dashboard you own.

mmap file-backed restore vs a UFFD handler

Before writing any of this, be honest about which path your bytes actually need:

• Fault source — mmap file-backed restore: the kernel resolves faults from the local file and page cache. UFFD handler: your user-space code resolves them from anywhere — a file, a cache, an HTTP range request.
• Prerequisite — mmap file-backed restore: the entire vm.mem must already be a local file before you can map it. UFFD handler: nothing needs to be local; the guest can start before any of the image has arrived.
• Fault-path cost — mmap file-backed restore: a trap plus a page-cache lookup or one disk read, no context switch out of the kernel. UFFD handler: a trap, a wake of your handler thread, your fetch, and an ioctl — strictly more overhead per fault.
• Failure mode — mmap file-backed restore: an I/O error surfaces as a guest-visible fault or a killed process. UFFD handler: if the handler stops answering, the guest hangs silently, which is much harder to diagnose.
• Cross-host reuse — mmap file-backed restore: every host must hold a full copy of every memory image it might restore. UFFD handler: hosts hold only the chunks they actually touched, so a host can restore a template it has never held.
• Complexity — mmap file-backed restore: one mmap call. UFFD handler: a socket protocol, an event loop, a concurrency model, a cache, and a new class of hang to debug.
• Hugepages — mmap file-backed restore: not supported for hugepage snapshots. UFFD handler: the only supported restore path for them.

Going remote: when the fault source is object storage

Notice that the fault path above already hides everything interesting behind one call: h.source.PageAt(offset, size). Once the handler asks an interface for bytes at a snapshot-file offset, the implementation can be a local file — or a range request against an object in cloud storage. That substitution is the entire streaming restore, and everything after it is engineering to make the remote case not feel remote.

// The whole abstraction: "give me the page at this offset in vm.mem."
// Returning (nil, nil) means "this page is known to be all zeros" so the
// handler can use UFFDIO_ZEROPAGE and skip the transfer entirely.
type Source interface {
	PageAt(off, size uint64) ([]byte, error)
}

// Local: a plain pread against a file on disk.
type fileSource struct{ f *os.File }

func (s *fileSource) PageAt(off, size uint64) ([]byte, error) {
	buf := make([]byte, size)
	if _, err := s.f.ReadAt(buf, int64(off)); err != nil {
		return nil, err
	}
	return buf, nil
}

// Remote: fetch the enclosing 4 MiB chunk, cache it, slice out the page.
type gcsSource struct {
	zeroes *ChunkBitmap // baked at snapshot time: which chunks are non-zero
	cache  *SharedCache // per-host, per-snapshot-generation, on disk
	obj    ObjectReader // HTTP Range GET against the snapshot object
}

const chunkSize = 4 << 20

func (s *gcsSource) PageAt(off, size uint64) ([]byte, error) {
	idx := off / chunkSize
	if s.zeroes.IsZero(idx) {
		return nil, nil // no fetch, no bytes: UFFDIO_ZEROPAGE
	}
	chunk, err := s.cache.GetOrFetch(idx, func() ([]byte, error) {
		start := idx * chunkSize
		return s.obj.Range(start, start+chunkSize-1) // one round trip, 1024 pages
	})
	if err != nil {
		return nil, err
	}
	in := off - idx*chunkSize
	return chunk[in : in+size], nil
}

That sketch is the shape of what PandaStack ships. Four ideas do the work, and each one exists because the naive version was too slow:

1. Chunk, don't page. Faults arrive 4 KiB at a time; we fetch 4 MiB. Memory access has strong spatial locality, so one range request amortizes the round trip across the neighbors the guest is about to want anyway. Fetching per-page over a network is a non-starter.
2. Elide zeros. At bake time we write a header recording which chunks contain any non-zero byte. A fault in an all-zero region is answered with UFFDIO_ZEROPAGE and no fetch at all. Zeros are the cheapest bytes in the system precisely because they never move.
3. Cache per host, keyed by generation. Chunks land in a persistent sparse file shared by every restore of that snapshot on that host. The first restore of a template pays object-storage latency for its working set; later ones read local disk. The cache key includes the snapshot generation, so a re-bake self-invalidates instead of serving memory from a previous build — and the present-bitmap is only advanced after the data is fdatasync'd, so a crash can never leave a bit set over bytes that aren't there.
4. Prefetch the hot set. A given template touches roughly the same pages every restore; the path from resume to ready is close to deterministic. So we record that hot chunk list at bake time and replay it in the background the instant restore starts, racing ahead of the guest so faults land on chunks that are already local.

I want to be careful not to oversell this. Streaming does not make memory arrive faster than the network allows; it makes you stop transferring memory nobody asked for, and then hides most of the remaining latency behind prefetch and caching. The first restore of an unfamiliar template on a cold host is still bounded by object-storage latency for its working set, and a cache miss in the fault path is a network round trip that a local mmap would have served from RAM. Where it wins decisively is a fleet that restores the same handful of templates thousands of times across hosts that come and go — which is exactly the shape of a sandbox platform.

Testing and debugging

Start by confirming the host will even let you do this. Unprivileged userfaultfd is gated by a sysctl on most distros, and discovering that inside a failing restore is worse than checking it at startup.

# 1. Can an unprivileged process call userfaultfd(2) on this host?
cat /proc/sys/vm/unprivileged_userfaultfd
#   0 -> requires CAP_SYS_PTRACE (or UFFD_USER_MODE_ONLY at open time)
#   1 -> any process may call it
sudo sysctl -w vm.unprivileged_userfaultfd=1   # or just run the handler privileged

# 2. Kernel support at all (5.10+ is fine for COPY/ZEROPAGE + REMOVE events)
uname -r
grep -E '^CONFIG_USERFAULTFD=' "/boot/config-$(uname -r)" || true

# 3. If you bake hugepage snapshots, the host needs pool headroom.
#    On-demand assembly beats boot-time reservation for a mixed fleet.
sysctl vm.nr_overcommit_hugepages

# 4. Watch the handler while a guest boots: every restore should show a burst
#    of faults that tails off as the working set lands.
curl -s http://localhost:9100/metrics | grep -E 'uffd_(faults|zero|fetch)'

Then wire it into Firecracker. The UFFD backend replaces mem_file_path in the snapshot/load body — you pass a mem_backend of type Uffd pointing at your socket, and you cannot pass both. Load with resume_vm false, verify the machine is loaded, then resume explicitly; splitting the two makes it obvious whether a hang is in the load or in your first faults.

# Your handler must ALREADY be listening on /srv/run/uffd.sock at this point.
curl --unix-socket /run/firecracker/fc.sock -i \\
  -X PUT 'http://localhost/snapshot/load' \\
  -H 'Content-Type: application/json' \\
  -d '{
        "snapshot_path": "/srv/seeds/base/vm.state",
        "mem_backend": {
          "backend_type": "Uffd",
          "backend_path": "/srv/run/uffd.sock"
        },
        "enable_diff_snapshots": false,
        "resume_vm": false
      }'

# Resume separately, so a hang tells you WHICH step hung.
curl --unix-socket /run/firecracker/fc.sock -i \\
  -X PATCH 'http://localhost/vm' \\
  -H 'Content-Type: application/json' \\
  -d '{"state": "Resumed"}'

For debugging, three habits pay for themselves. Write a byte-for-byte differ: restore the same snapshot through mmap and through your handler, run an identical workload, and compare guest-visible memory — any divergence is a translation bug, and translation bugs otherwise present as impossible guest behavior. Count everything: faults served, zero-pages elided, chunks fetched, bytes transferred, and time spent inside PageAt. And when a guest hangs, check your handler before you check the guest — attach a profiler or send SIGQUIT for a goroutine dump, because a stack trace showing every worker parked on the same mutex answers the question in five seconds, while poking at a frozen VM answers nothing at all.

When not to bother

Don't write this handler if your snapshots are already local. A few hundred megabytes of vm.mem on NVMe, restored by mmap with MAP_PRIVATE, is fast, boring, and correct, and the page cache already shares clean pages across every VM restored from that file. Adding a user-space handler there buys nothing and costs you a failure mode where hangs replace errors. Don't write it for a dev laptop, or a workload where one cold boot of roughly 3s is acceptable, or if you can't afford the operational maturity — metrics, timeouts, lifetime coupling — that a component holding vCPUs hostage demands.

Do write it when memory images live somewhere other than the host that needs them, when the same templates are restored constantly across a changing fleet, when full-image downloads dominate your restore latency, or when you want hugepage-backed guests at all. On PandaStack that's the everyday case: every sandbox create restores a baked snapshot rather than cold-booting, agents come and go under autoscaling, and a new host should be able to serve a template it has never held without a multi-gigabyte download standing between a user and their VM. The handler is a few hundred lines of Go. The interesting part isn't the ioctls — it's deciding what happens when the bytes are late.

The core is open source under Apache-2.0, so you can read the real handler, chunk cache, and prefetch trace rather than reconstruct them from a blog post. For the conceptual grounding, /blog/userfaultfd-explained; for the local path this one replaces, /blog/firecracker-memory-file-mmap-explained.