NUMA Locality and Firecracker Snapshot Restore

There's a failure mode that only shows up once you've succeeded a little: you outgrow single-socket hosts, buy a fat two-socket box to get more microVMs per rack unit, and some of your guests get slower. Not all of them, not reliably, and not in a way that shows up in any single metric — just a fatter tail and a vague sense that the new hardware isn't paying for itself. We bought a bigger box and it got slower is one of the oldest complaints in systems engineering, and on a multi-socket machine the usual culprit is NUMA: where a guest's memory physically lives relative to the CPU that's reading it.

Snapshot restore makes this more interesting than it is for a normal long-lived VM, because restore is the moment when a guest's memory placement gets decided — and the decision is made implicitly, by whichever thread happens to touch each page first. With a plain mmap'd restore that's the vCPU thread. With a userfaultfd handler streaming pages in, it's the handler thread, which may not be on the same socket as the vCPU at all. I'm Ajay; I build PandaStack, a Firecracker platform where every sandbox create is a snapshot restore, so 'where do the pages land' is a question I've had to answer with tooling rather than vibes. This post is the primer, the exact interaction with both restore paths, the tools, the mitigations, and — importantly — the case for measuring before you build any of the machinery.

A quick, accurate NUMA primer

On a single-socket server, all cores reach all DRAM through the same memory controller, and access cost is uniform. That's UMA. On a multi-socket server, each socket has its own integrated memory controller and its own directly-attached DIMMs. A socket plus its local memory (plus, on modern CPUs, sometimes a sub-socket partition of it) is a NUMA node. Cores can still address every byte of every node's memory — it's one flat physical address space — but they don't all reach it the same way.

• Local access — a core reads memory attached to its own socket's memory controller. This is the fast path: address goes to the local controller, data comes back.
• Remote access — a core reads memory attached to a different socket's controller. The request crosses the inter-socket interconnect (UPI on Intel, Infinity Fabric on AMD), gets serviced by the remote controller, and the data comes back across the link. Higher latency, and the interconnect is finite shared bandwidth that every core on both sockets is competing for.
• The asymmetry is the whole point — 'non-uniform memory access' is the literal description. Remote access costs meaningfully more than local, and it's not a small rounding error, but the exact ratio depends entirely on your CPU generation, interconnect, memory speed, and how loaded the link already is. Measure yours (lat_mem_rd from lmbench, or Intel MLC) rather than trusting a number you read on a blog — including this one, which is why you won't find a ratio here.

Two second-order effects matter as much as raw latency. First, interconnect bandwidth is a shared resource: a handful of guests doing heavy remote reads can saturate the link and degrade every other cross-socket access on the box, which is a noisy-neighbor problem wearing a different hat. Second, remote latency is more variable than local latency — it depends on link contention — so cross-node traffic tends to show up as tail-latency jitter rather than a clean shift in the mean. That's exactly the kind of degradation that's hard to attribute.

First-touch allocation: the thread that faults decides

Here's the mechanism that makes NUMA a runtime concern rather than a static one. Linux's default memory policy is first-touch (MPOL_DEFAULT, sometimes called local allocation). When you allocate memory — mmap an anonymous region, say — the kernel does not pick a NUMA node at allocation time. It picks nothing at all; the mapping is just bookkeeping and no physical page exists yet. The node is chosen at the moment of the first actual access to each page: when a thread faults on a page, the kernel allocates a physical page on the NUMA node that thread is currently running on, if that node has free memory.

Read that again with the emphasis on the right word: the node the faulting thread is running on. Not the node of the process that allocated the region. Not the node of the thread that will do most of the reading later. Whichever thread got there first, and wherever the scheduler had placed it at that instant. Once the page is placed, it stays there — Linux will not spontaneously migrate it just because a different node's cores are the ones reading it (unless you've enabled automatic NUMA balancing, which does try, with its own tradeoffs and overheads).

The corollary for microVMs: your guest's RAM placement is an emergent property of thread scheduling during the first few milliseconds of its life. Nobody made a decision. The scheduler did, implicitly, on your behalf, and then it became permanent. Your memory is local until the scheduler has an opinion.

Restore path 1: mmap'd MAP_PRIVATE, faulted by the vCPU

The default Firecracker restore path maps the snapshot's memory file with mmap and MAP_PRIVATE and resumes the vCPUs; nothing is read up front and every page faults in lazily on first touch. (The full mechanics of that are in /blog/firecracker-memory-file-mmap-explained.) For NUMA purposes, what matters is who does the touching and what kind of page results.

There are two distinct populations of pages here, and they behave differently:

• Clean, unwritten pages — these are page-cache pages belonging to the snapshot memory file. The guest reads them; the mapping is wired to whatever physical page the page cache already holds. Their node was decided by whoever first read that file offset into cache — possibly a completely different VM's restore, possibly a prefetch, possibly your seed-sync writing the file. A later restore inherits that placement and has no say in it.
• Private copy-on-write pages — when the guest writes to a page, the kernel allocates a fresh anonymous page for the private copy. That allocation is a first-touch allocation, performed in the context of the faulting vCPU thread, so it lands on the node that vCPU is running on at that moment.

So on the mmap path, the guest's diverged (written) memory follows vCPU scheduling, which is at least the right instinct: the thread that will read that page most is the one that placed it. The catch is that it's only right if the vCPU stays put. If the host scheduler migrates a vCPU thread to the other socket after the guest has faulted in its working set — and by default nothing stops it — every one of those pages is now remote, and they will stay remote. The guest doesn't get slower at a moment you can point at; it gets slower as a consequence of a scheduling decision nobody logged.

The shared clean pages have the opposite character: they're a density win precisely because they're shared across every VM restored from the same template on that host, and shared pages can only live on one node. On a two-socket host, half your guests are reading a shared baseline that is, for them, remote — and there is no placement that fixes this, because the whole point is that one physical copy serves everyone. You can duplicate the page cache per node to fix it, at the cost of the density the sharing bought you. That is a real tradeoff, not a bug.

Restore path 2: userfaultfd, where the handler thread touches first

The streaming restore path replaces the local memory file with a userfaultfd: Firecracker registers the guest memory region with UFFD, and a user-space handler process answers page faults by fetching the page's contents (from a local cache, or over the network from object storage) and installing them with UFFDIO_COPY. The guest faults, the handler serves, the guest continues. The full fault-flow mechanics are in /blog/userfaultfd-explained.

Now look at that through the first-touch lens, because this is the non-obvious part. UFFDIO_COPY does two things: it allocates a physical page for the faulting address, and it copies your buffer's contents into it. That allocation happens in the context of the handler thread issuing the ioctl — not the vCPU thread that's blocked waiting on the fault. Under first-touch, the page is therefore placed on the handler's node.

Worse, it's a systematic bias rather than random noise. A single handler process serving many restores will tend to place all of their memory on its own node, which both concentrates memory pressure on one node (making it more likely to spill, at which point the kernel falls back to allocating on another node anyway) and leaves every guest scheduled on the other socket permanently remote. A per-VM handler thread inherits whatever node the scheduler put it on. Either way, the placement is accidental.

The fix is not complicated, it just has to be deliberate: make the handler and the vCPUs the same node. Pin the whole VMM process — vCPU threads and the UFFD handler serving it — to one node's CPUs and bind its allocations to that node's memory. Then it doesn't matter which of them faults first, because both answers are the same node.

Tooling: seeing your topology and where pages actually landed

None of this is worth reasoning about abstractly. The Linux NUMA tooling is mature and the answers are directly observable — you can see your topology, see per-node allocation counters, see which node a specific process's pages are on, and count remote-DRAM accesses in hardware.

# 1) What does this machine actually look like? Nodes, their cores, their RAM,
#    and the kernel's own distance matrix (10 = local, larger = further away).
numactl --hardware
#   available: 2 nodes (0-1)
#   node 0 cpus: 0-31,64-95
#   node 0 size: 257512 MB
#   node 1 cpus: 32-63,96-127
#   node 1 size: 257521 MB
#   node distances:
#   node   0   1
#     0:  10  21      <- relative cost, NOT nanoseconds. Do not treat as latency.
#     1:  21  10

lscpu | grep -i numa            # same topology, terser
ls /sys/devices/system/node/    # node0/ node1/ ... the raw kernel view
cat /sys/devices/system/node/node0/meminfo | head -4

# 2) Per-node allocation counters. numa_hit = allocation satisfied on the
#    intended node; numa_miss / numa_foreign = it had to go elsewhere.
#    A climbing miss count means first-touch is NOT getting what it asked for
#    (usually: that node is full).
numastat
numastat -p "$(pgrep -n firecracker)"   # per-node RSS for one VMM process
#   Node 0  Node 1  Total
#   ...     ...     ...   <- if a guest's memory is lopsided, you found it

# 3) Which node did THIS process's pages land on, per mapping?
#    (N0=... N1=... counts per VMA.)
grep -A4 -m3 'rw-' /proc/$(pgrep -n firecracker)/numa_maps

# 4) Is the guest actually paying for remote access? Hardware counters.
#    Event names are uarch-specific — list first, then count.
perf list | grep -iE 'remote_dram|remote_pmm|offcore.*remote'
perf stat -e mem_load_l3_miss_retired.remote_dram \
  -p "$(pgrep -n firecracker)" -- sleep 30
#   A high remote_dram share of L3 misses is the smoking gun. This is the
#   measurement that tells you whether any of this is worth fixing.

# 5) Last resort: move an already-running process's pages between nodes.
#    Real work (copy + TLB shootdowns), and it does not stop it happening again.
migratepages "$(pgrep -n firecracker)" 0 1   # move node 0 pages -> node 1

Two notes on reading this output. The distance matrix from numactl is a relative cost hint the firmware reports (SLIT), not a latency in nanoseconds — 21 vs 10 does not mean 'exactly 2.1x slower.' And numa_miss climbing is a different problem from remote access: miss means the first-touch allocation couldn't be satisfied on the desired node because it was full, so the kernel placed it elsewhere. Pinning memory to a node that's already exhausted doesn't get you locality, it gets you either a fallback allocation or, with a strict policy, an OOM. Capacity planning per node is part of this, not separate from it.

Mitigations, in the order you should try them

Pin the VMM and its UFFD handler to the same node

This is the high-leverage one, and it's a one-line change to how you launch. numactl --cpunodebind=N restricts the process (and its children, including threads) to node N's CPUs; --membind=N forces its allocations onto node N's memory. Applied to the whole VMM — vCPU threads, the API thread, and the UFFD handler if it's a child — it makes the first-touch question moot, because every candidate first-toucher is on the same node.

#!/usr/bin/env bash
# Launch one microVM entirely within NUMA node $NODE: CPUs and memory both.
set -euo pipefail
NODE="${NODE:-0}"
VM="${VM:-vm-abc123}"

# The UFFD handler must land on the SAME node as the vCPUs, because it is the
# thread that performs UFFDIO_COPY and therefore first-touches every guest page.
numactl --cpunodebind="$NODE" --membind="$NODE" \
  /usr/local/bin/pandastack-uffd-handler \
    --socket "/run/$VM/uffd.sock" \
    --mem-source "/var/lib/pandastack/seeds/base/vm.mem" &
HANDLER_PID=$!

# Same node for the VMM itself. --membind is strict: if node $NODE is out of
# memory the allocation FAILS rather than silently landing remote. Use
# --preferred=$NODE instead if you'd rather degrade than fail.
numactl --cpunodebind="$NODE" --membind="$NODE" \
  firecracker --api-sock "/run/$VM/fc.sock" &
FC_PID=$!

# Belt and braces: a cpuset cgroup pins the whole VM's thread tree, including
# threads spawned later that numactl's inherited affinity might not cover.
CG=/sys/fs/cgroup/pandastack/$VM
mkdir -p "$CG"
cat "/sys/devices/system/node/node$NODE/cpulist" > "$CG/cpuset.cpus"
echo "$NODE" > "$CG/cpuset.mems"
echo "$FC_PID"      > "$CG/cgroup.procs"
echo "$HANDLER_PID" > "$CG/cgroup.procs"

# Verify rather than assume — placement is emergent, so check it.
numastat -p "$FC_PID"

Note the cpuset.mems line: that's the cgroup-level equivalent of --membind, and it's the durable one. numactl sets a policy on the process it launches; a cpuset constrains the whole cgroup, including threads created later. If you're already running per-VM cgroups for CPU weight and quota — which you should be, see /blog/microvm-cpu-pinning-noisy-neighbor — adding cpuset.cpus and cpuset.mems is a small extension of machinery you have, not new machinery.

Size VMs to fit inside one node

A guest whose memory or vCPU count exceeds what one NUMA node can provide is guaranteed to straddle, and then you're not doing NUMA placement, you're doing NUMA damage control (and you'd need to expose a virtual NUMA topology to the guest so its own scheduler can make sane decisions — a much bigger project). For sandbox-shaped workloads this is a non-issue: guests sized in single-digit gigabytes fit inside a node with enormous room to spare, and the constraint costs you nothing. Keep it that way deliberately.

Keep the snapshot file's page cache node-aware

The shared clean pages of a memory file live wherever they were first read in, and every VM on the host shares them. If you prefetch or warm a template's memory file, do it from a thread pinned to a known node, so at least the placement is a decision instead of an accident. If you're running a fat host where cross-node access to the shared baseline actually shows up in your remote-DRAM counters, the option is to keep one warmed copy of the hot template per node — deliberately duplicating page cache to buy locality, and paying for it in memory. Only do this if measurement says the sharing is hurting you more than the duplication would.

Make node placement a scheduler decision, not a per-VM one

On a host running hundreds of microVMs, per-VM NUMA tuning is the wrong altitude. What you actually want is for whatever assigns VMs to hosts to also assign them to nodes — treat each NUMA node as a schedulable pool with its own free-CPU and free-memory accounting, and place a new VM into the node that has room, then pin it there for its lifetime. That's a bin-packing problem you already solve at the host level; NUMA just adds a second level to it. It also automatically handles the numa_miss failure mode, because you never place a VM on a node you know is full.

The tradeoff table

• Do nothing (default first-touch) — Locality: accidental; correct-ish on the mmap path, frequently wrong on the UFFD path. Cost: zero. Fit: single-socket hosts, and small short-lived guests where the penalty is genuinely noise.
• numactl --cpunodebind + --membind per VM — Locality: strong; vCPUs and handler share a node so first-touch lands right. Cost: one line at launch, plus per-node capacity planning. Fit: the default for multi-socket hosts. Highest leverage per unit of effort.
• cpuset cgroup (cpuset.cpus + cpuset.mems) — Locality: same as numactl but durable across later-spawned threads. Cost: needs per-VM cgroups, which dense fleets already have. Fit: production multi-socket, especially alongside cpu.weight/cpu.max.
• Per-node page-cache duplication of hot templates — Locality: fixes remote access to the shared baseline. Cost: gives back the page-cache sharing that density depends on. Fit: only when remote-DRAM counters prove the shared baseline is the bottleneck.
• migratepages after the fact — Locality: repairs a bad placement. Cost: real work (page copies + TLB shootdowns) and it doesn't prevent recurrence. Fit: firefighting and diagnosis, not a strategy.
• Automatic NUMA balancing (kernel) — Locality: self-correcting over time, no config. Cost: scanning and migration overhead, and it fights your explicit pinning. Fit: unpinned general-purpose workloads; usually disabled where you've pinned deliberately.
• Node-aware scheduling at the fleet level — Locality: correct by construction, and respects per-node capacity. Cost: scheduler complexity; a second bin-packing dimension. Fit: dense multi-socket fleets at scale — the right long-term home for this concern.

The honest counterpoint: measure before you build

Everything above is real, and for a lot of microVM fleets it is also irrelevant. Here's the case against building any of it.

• The penalty is per-access, and short-lived guests don't accumulate many. A sandbox that lives for ten seconds executing a bit of code may never do enough memory traffic for a remote-access penalty to be visible against the noise of everything else it's doing.
• Restore latency is mostly not memory-bandwidth-bound. The create path is dominated by process setup, snapshot state load, network configuration, and readiness probing — the memory work is deliberately lazy and O(1). Making the pages local doesn't obviously speed up the part you're actually measuring.
• Small working sets fit in cache. A guest whose hot set stays resident in L2/L3 pays the DRAM cost once. NUMA hurts workloads that stream through memory, not ones that thrash a small footprint.
• Pinning has its own costs. Binding VMs to nodes fragments capacity and reduces the scheduler's freedom to fill the box, which is the same density-versus-predictability tradeoff you face with CPU pinning — and density is usually why you bought the fat box in the first place.
• Most hosts are single-socket. If your fleet is one socket per host, there is exactly one NUMA node and none of this applies. Check before you architect.

The density angle: a scheduler concern, not a VM concern

If measurement does say it matters, the shape of the right answer changes with scale. For a handful of large latency-sensitive guests, per-VM numactl is fine and you're done. For a fat two-socket host running hundreds of small microVMs, per-VM tuning is both operationally annoying and insufficient — you'd be making hundreds of independent placement decisions with no global view, which is a recipe for lopsided node pressure and the numa_miss fallbacks that quietly undo the whole exercise.

At that density the correct framing is that a NUMA node is a schedulable unit. Your host-level scheduler already scores agents by free CPU and free memory to decide where a sandbox goes; the node-aware version scores (host, node) pairs by the same criteria and pins the VM to the winner. That gives you locality by construction, respects per-node capacity so first-touch actually gets the node it asks for, and keeps the decision in one place where you can reason about it — instead of scattered across launch scripts. It's more scheduler work, and it's the version that survives contact with a fleet.

It also composes with the neighboring concerns rather than fighting them. The cpuset you'd use for NUMA pinning is the same cpuset you'd use to keep tenant workloads off the cores reserved for the control plane and the restore path. The tiering logic is identical: most of the fleet runs unpinned and dense, and the workloads that pay for predictability get a node to themselves.

The summary

On a multi-socket host, a NUMA node is a socket plus its directly-attached memory; local access goes through the local memory controller and remote access crosses the interconnect at a meaningfully higher and more variable cost. Linux's default first-touch policy means a page's node is chosen when a thread first touches it, in that thread's context, and then it stays there. On the mmap'd MAP_PRIVATE restore path, the toucher is the vCPU, so private copy-on-write pages follow vCPU scheduling — right by default, wrong the moment a vCPU migrates sockets — while the shared clean page-cache pages sit wherever they were first read and are remote for half your guests by construction. On the userfaultfd path the toucher is the handler thread doing UFFDIO_COPY, which can be on a different node entirely from the vCPU waiting on the fault, so a guest's entire memory can be populated remotely without anything looking wrong.

The fix is to stop leaving it to the scheduler: pin the VMM and its UFFD handler to the same node with numactl --cpunodebind and --membind, make it durable with cpuset.cpus and cpuset.mems, size guests to fit inside a node, and at real density move the decision up into the scheduler so node placement respects per-node capacity. And the discipline that makes all of that worth doing is the measurement that comes first — numactl --hardware for topology, numastat and /proc/PID/numa_maps for where pages actually landed, and perf's remote-DRAM counters for whether any of it is costing you.

On PandaStack the restore path is the product: every sandbox create restores a baked snapshot rather than cold-booting, with the memory-load step landing around 49ms inside a create that's p50 179ms and p99 about 203ms (a true cold boot, only on first spawn, is around 3s; a same-host fork is 400–750ms and cross-host 1.2–3.5s). Because the memory work is lazy and O(1), NUMA doesn't show up in that create number — it shows up afterwards, in the steady-state performance of a guest whose pages landed on the wrong socket, which is exactly the kind of cost that hides from a create-latency dashboard. The SDK side of it is deliberately boring:

from pandastack import Sandbox

# Placement — which host, and on a multi-socket host which NUMA node — is a
# platform concern. The API surface is just "give me a sandbox"; the agent
# restores the baked snapshot and pins the VMM + its UFFD handler together.
with Sandbox(template="code-interpreter") as sbx:
    # Steady-state work is where NUMA locality shows up, not in the create.
    # If you want to know whether YOUR guest is paying a remote-access tax,
    # measure inside it over a real workload rather than at boot.
    res = sbx.exec("python -c \"import numpy as np; a=np.ones((4096,4096)); print(a.sum())\"")
    print(res.stdout, res.exit_code)

# Host-side, the corresponding check is per-VM and per-node:
#   numastat -p $(pgrep -n firecracker)
#   perf stat -e mem_load_l3_miss_retired.remote_dram -p <pid> -- sleep 30

The core is open source under Apache-2.0, so you can run the agent on your own two-socket KVM host and watch the placement yourself — which, given how much of NUMA folklore is unmeasured, is the only way to find out what it's worth on your hardware. For the restore mechanics that this all hangs off, start with /blog/firecracker-memory-file-mmap-explained; for the streaming handler that introduces the first-touch trap, /blog/userfaultfd-explained; and for the sibling problem of CPU-time isolation on the same dense hosts, /blog/microvm-cpu-pinning-noisy-neighbor.