Memory Prefetch: The Working Set Is the Real Unit of a Fast Restore

There are two honest ways to bring a snapshotted virtual machine back to life. The obvious one is to read the entire memory image, map it, and resume: a 2 GiB guest means 2 GiB of I/O before the first instruction executes, and that scales linearly with how generous you were about RAM at bake time. The other is to resume immediately with nothing behind the memory mapping at all, and let the guest tell you which pages it wants by faulting on them.

The second approach is what userfaultfd buys you. The VMM registers the guest's memory region with a userfaultfd, hands the file descriptor to a handler process, and the kernel routes every first-touch page fault to that handler instead of resolving it itself. The mechanism — the syscall, the region handoff, UFFDIO_COPY — is covered in /blog/userfaultfd-explained. This post is about the part that decides whether the result actually feels fast: which pages, and when.

I'm Ajay, I built PandaStack — this post is about the working set: why it's the real unit of a fast restore, how you record one at bake time and replay it at restore time, and what happens when you confidently prefetch the wrong one.

Restore stops being a download and becomes a payment plan

With a UFFD backend, the restore step no longer includes the memory. The VMM loads the state file (registers, device models, vCPU state), registers the memory regions as userfaultfd-backed, and resumes. The guest is running; its RAM is, physically speaking, not there yet. Every page it touches for the first time traps into your handler, which finds those bytes in the snapshot and installs them atomically at the faulting address.

This is a genuinely good trade and the numbers reflect it. On PandaStack every sandbox create restores a baked Firecracker snapshot — around 49ms for the restore step itself, p50 179ms end-to-end including network setup and the readiness probe, p99 around 203ms. A ~3s cold boot happens only once, the first time a template is baked, before a snapshot exists. Forks land at 400–750ms same-host and 1.2–3.5s cross-host, where the memory has to come from object storage. None of that would be possible if every create started with a multi-gigabyte read.

But notice what changed about the accounting. You didn't remove the cost of materializing memory — you moved it off the critical path of "restore" and onto the critical path of "the guest doing anything." Restore duration became a much less interesting metric the moment it stopped containing the work.

The catch: the bill now arrives as confetti

If the pages come from a local file, a fault is cheap — page cache or NVMe, microseconds, invisible. If they come over the network from object storage, every cold fault is a round trip. And a guest coming out of a snapshot is not shy about faulting: the scheduler runs, init wakes, a runtime touches its heap, and the first request drags a code path through a few thousand distinct pages nobody has touched since the bake.

Latency didn't vanish. It got shredded into confetti and sprinkled over the first few seconds of the guest's life, where your restore metric can't see it and your user definitely can.

This is the failure mode that makes demand paging look like a bad idea to anyone measuring from the outside. The dashboard says restore completed in milliseconds; the guest feels like it's running in syrup, because it stalls on a network fetch every few hundred microseconds of useful work. "Technically instant, practically janky" is worse than an honest slow path, because it's harder to attribute.

The insight: a guest has a working set, and it's boring

Here's the property that makes all of this tractable. A booted, warmed guest doesn't touch its memory uniformly. It has a working set — the pages it actually reads and writes to get from resume to serving — and that set is a small fraction of total RAM. More importantly, it is remarkably stable across restores of the same snapshot, because every restore of the same snapshot is the same machine: same guest kernel, same init sequence, same preloaded runtime, same libraries mapped at the same addresses, same first request handler.

Restoring a snapshot is not like starting a fresh process. There is no ASLR reroll, no different library versions, no "it depends what the OS decided to page out today." The guest resumes from a byte-identical memory image into a byte-identical execution state. If it faulted on chunk 4,119 last time, it will almost certainly fault on chunk 4,119 this time, at roughly the same point in the sequence.

• What makes it stable — a fixed memory image plus a fixed resume point means the code path from resume to ready is largely deterministic.
• What makes it small — most of a guest's RAM is never touched in its first seconds. A 4 GiB guest needing a few hundred MiB to answer a request is normal, not lucky.
• What makes it recordable — faults are already delivered to a user-space handler, so logging what it served during a warm run costs almost nothing and produces exactly the list you want.
• What makes it useful — a recorded set can be fetched ahead of the guest instead of behind it, converting a demand fault (network round trip) into a cache hit (memcpy).

Which reduces the whole thing to a scheduling question. The bytes move either way. Do they move while the guest is blocked waiting, or while it's busy doing something else?

Record at bake, replay at restore

The mechanic is straightforward once you accept the premise. At bake time — template booted, warmed, snapshotted — you run the guest through a representative warm-up and record the sequence of chunks the fault handler served into a trace file that travels with the snapshot. At restore, the moment the VM resumes, a background goroutine walks that trace and installs those chunks before the guest asks. The guest's faults then hit memory that's already resident, which means they aren't faults at all.

Why the trace keeps its order

You could store the working set as an unordered bitmap of "chunks that matter." Don't. Order buys two separate things. It front-loads correctly: the prefetcher races the guest along the same path the guest is about to walk, so the chunks needed earliest arrive earliest. An unordered prefetch can spend its first second on pages the guest won't touch for another three, while the guest stalls on something you deferred. And the order the guest touched memory in isn't random — it correlates with layout, so replaying it reads closer to sequentially against the backing store than a shuffle would. Object stores and disks both reward that.

Why you fetch 4 MiB to satisfy a 4 KiB fault

A page fault is 4 KiB. Fetching 4 KiB over HTTP to satisfy it is an act of self-harm: you pay a full round trip — connection reuse, request, first byte — for an amount of data that transfers in less time than the request header took to serialize. So the unit of transfer is a 4 MiB chunk, which amortizes that one round trip across roughly a thousand 4 KiB pages.

This works because spatial locality is real. A guest's memory layout is not a random scatter — slab allocations cluster, a runtime's heap grows contiguously, a mapped library's text section is a contiguous run of pages executed together. Chunking is a bet that a page the guest just faulted on has neighbours it wants shortly, and it's a bet the allocator has already rigged in your favour.

// prefetch replays the bake-time fault trace so the guest's first faults land on
// memory that is already resident. Started in the background the instant the VM
// resumes; the demand-fault path always has priority over this loop.
func (h *Handler) prefetch(ctx context.Context, trace []ChunkID) {
	for _, id := range trace {
		if ctx.Err() != nil {
			return // guest is gone, or the fault path asked us to get out of the way
		}

		// Never fetch a chunk the bake-time header says is entirely zeros.
		if h.header.IsZero(id) {
			h.uffd.ZeroPages(id.Addr(), ChunkSize) // UFFDIO_ZEROPAGE, no I/O at all
			continue
		}

		buf, ok := h.cache.Get(id) // shared per-host chunk cache, local disk
		if !ok {
			var err error
			buf, err = h.source.FetchChunk(ctx, id) // HTTP Range GET, 4 MiB
			if err != nil {
				continue // prefetch is best-effort; the demand path will retry on fault
			}
			h.cache.Put(id, buf) // fdatasync the data BEFORE flipping the present bit
		}

		// Install ahead of the guest. If the guest already faulted into this range
		// while we were fetching, the kernel tells us so and we move on.
		if err := h.uffd.CopyPages(id.Addr(), buf); err != nil && !errors.Is(err, unix.EEXIST) {
			h.log.Warn("prefetch install failed", "chunk", id, "err", err)
		}
	}
}

The EEXIST case is the whole design in one line. Prefetch and demand faulting race constantly, and the resolution is that whoever gets there first wins and the loser shrugs — no coordination protocol, no lock held across a network fetch. UFFDIO_COPY into an already-populated range fails cleanly, which makes "install it twice" a non-event rather than a corruption bug.

Two things that make the remaining fetches cheaper

Zero-page elision: the cheapest fetch is the one you skip

Now the free win. A freshly booted guest's memory image is mostly zeros: it used some RAM for kernel structures, page cache, and a runtime heap, and the rest is untouched — which in a snapshot serializes as a very long run of zero bytes. Shipping those zeros across a network to reconstruct nothing is pure waste.

So at bake time you scan the memory image and record, per chunk, whether it contains any non-zero byte, into a small header that travels with the snapshot. At restore, a fault into a chunk marked all-zero is satisfied with UFFDIO_ZEROPAGE — the kernel maps a zero page locally, no network, no disk, no bytes moved. That removes a large fraction of a typical image from the transfer set for both the prefetcher and the demand path, and the header is just a bitmap computed by one linear scan on a machine that isn't in a hurry.

The shared chunk cache: pay the network once per host

A host rarely does a single restore — it runs the same template over and over, same generation, same chunks, same working set. So chunks fetched from object storage land in a persistent, sparse, per-host cache on local disk. The first restore of a generation pays object-storage latency for its working set; every later one reads those chunks off local NVMe, where a fault costs microseconds and the confetti problem stops being a problem.

The cache is keyed by the snapshot's identity — a hash of the backing object's bucket and path — not by template name. Re-baking publishes a new generation with a new key, so old entries are never consulted again rather than silently serving stale memory into a guest expecting the new image. Self-invalidating by construction beats a cache invalidation you have to remember to trigger, because you will not remember to trigger it.

One line on crash safety, because this is the part that will hurt you: a cache claiming a chunk is present must have durably written it first. Write the data, fdatasync, then flip the present bit — never the other order. Get it backwards and a host that loses power comes back advertising garbage, which you will then install directly into a guest's address space at an address it believes holds its page tables. Debugging that from the symptoms is an experience you should not go looking for.

Hugepages: fewer faults, but decided at bake time

There's a blunter lever available. Back guest memory with 2 MiB hugepages instead of 4 KiB base pages and one fault covers 2 MiB — a 512x reduction in fault count for the same memory materialized. Fewer trips through the kernel, fewer handler wakeups, fewer chances to stall. For a fault-dominated restore that's a large structural win, and it stacks with prefetching rather than replacing it.

The constraint to internalize is that hugepage-ness is a property of the snapshot, not of the restore. A VM whose memory was backed by hugepages produces a snapshot that must be restored the same way — in Firecracker's case, only through the UFFD backend; pointing it at a plain memory file is rejected. So it's a bake-time decision that travels with the artifact. Turning it on doesn't speed up snapshots you already hold; it changes what future bakes produce. Marking it on the artifact itself, rather than inferring it from a host config flag, is the difference between a boring rollout and a confusing one.

Honest limits: prefetch only helps the predictable part

All of this works because the guest's early behaviour is determined by the snapshot. The moment it's determined by the input instead, the technique stops applying. A sandbox whose first act is genuinely data-dependent — running the code you just handed it, importing whatever that code needs — will take cold faults no trace could have anticipated, because at bake time that data did not exist.

The sharper failure is a trace that doesn't match production. A prefetch trace recorded from a warm-up that exercises the wrong path is a very efficient, very well-ordered mechanism for loading memory nobody wanted, while the guest stalls on the pages you didn't record. You've built a fast car and pointed it at the wrong city.

• Trace drift after a template change — the runtime got upgraded, the warm-up script changed, the app now imports a different library. The trace is still valid-looking and still replayed; it's just describing a machine that no longer exists. Re-record the trace as part of the bake, never as a separate manual step.
• Prefetch starving the demand path — both compete for the same connection pool and disk bandwidth. If the prefetcher hogs it you've made cold faults slower in order to avoid them. The demand path must always win.
• A transient object-storage error killing the stream — the demand path can't give up on a fault; the guest is blocked on that address forever. Retry with a real budget, not a single attempt, or the guest wedges in a way that looks like a hang rather than an error.
• Over-tuning chunk size — bigger chunks amortize round trips but waste bandwidth on unwanted pages and delay the fault currently blocking a vCPU. This is a curve with a middle, not a dial to max out.
• Benchmarking only warm hosts — the first restore of a generation is the one that pays the network bill and fills the cache. If you never measure it, you have not measured what a scale-out event feels like.

Measuring the number that actually matters

The discipline that follows: time the guest's first real unit of work, not the API call that created it. Create the sandbox, immediately run something that exercises the path you care about, and report both numbers separately so you can see which one moved.

import time
from pandastack import Sandbox

# The first request is where deferred page faults actually land, so measure it
# separately from create(). A restore that "completed" in milliseconds can still
# hand you a guest that spends its first second faulting over the network.

FIRST_REQUEST = """
import json, time
t = time.perf_counter()
import numpy as np, pandas as pd          # drags in a lot of previously-cold pages
df = pd.DataFrame({"x": np.arange(200_000)})
mean = float(df.x.mean())
print(json.dumps({"work_ms": round((time.perf_counter() - t) * 1000, 1), "mean": mean}))
"""

t0 = time.perf_counter()
sbx = Sandbox.create(template="code-interpreter", ttl_seconds=900)
create_ms = (time.perf_counter() - t0) * 1000

try:
    sbx.filesystem.write("/work/first_request.py", FIRST_REQUEST)

    t1 = time.perf_counter()
    res = sbx.exec("python3 /work/first_request.py", timeout_seconds=600)
    first_ms = (time.perf_counter() - t1) * 1000

    print(f"create        {create_ms:8.1f} ms")
    print(f"first request {first_ms:8.1f} ms   <- the number your users feel")
    print("guest-side   ", res.stdout.strip(), "exit", res.exit_code)
finally:
    sbx.kill()

Run that against a host that has never seen the template and against one that has restored it fifty times. The gap between the two first-request numbers is your cache and prefetch working — or not working. The create number will barely move, which is precisely the point.

# Same snapshot, restored three times on one host. The first restore pulls its
# working set over the network; the later ones read it from the local chunk cache.
for i in 1 2 3; do
  curl -s -X POST "$PANDASTACK_API/v1/sandboxes" \
    -H "Authorization: Bearer $PANDASTACK_API_KEY" \
    -H 'Content-Type: application/json' \
    -d '{"template":"code-interpreter","ttl_seconds":120}' \
    -o /dev/null -w "create %{time_total}s\n"
done

# Agent-side counters: boot durations plus fault/fetch/zero-elision totals.
curl -s http://localhost:9100/metrics | grep -E 'pandastack_(sandbox_boot|uffd)'

Strip away the syscalls and this is a scheduling result, not a memory result. Demand paging doesn't make a restore faster; it makes it later, which only helps if something useful happens in the gap. The layers stack in order: zero-elision removes most of the image from consideration, chunking amortizes the round trips that remain, the trace puts them in the order the guest will want them, the shared cache means a host pays that bill once per generation, and hugepages cut the fault count by orders of magnitude if you decided so at bake time. None is clever alone. Together they're the difference between a restore that completed and a machine that's ready.

If you want the mechanism underneath all of this — the userfaultfd syscall, region handoff, and the fault-handling loop itself — that's /blog/userfaultfd-explained. For how the streamed cold restore behaves end to end when a host has never held the snapshot, see /blog/thaw-sub-second-cold-restore. And for where the restore step sits inside the full create path — network slot, reflink, VMM spawn, resume, readiness probe — /blog/snapshot-restore-boot-path walks the whole thing in order.