The Snapshot-Restore Thundering Herd, Explained

Every cron expression in the world has a bias toward zero. Not 09:03, not 09:17 — 09:00:00, on the minute boundary of a clock that just synced with NTP. So at the top of the hour a scheduler decides four hundred jobs are due, four hundred sandboxes are requested inside the same tens of milliseconds, and every one of them wants the same template snapshot. Individually, each of those restores is a small, elegant piece of engineering. Collectively they are a stampede, and the thing they are stampeding toward is a single file in an object store.

I'm Ajay, and I build PandaStack, a Firecracker microVM platform where every sandbox create is a snapshot restore. Demand-paged restore is the reason a create is p50 179ms and p99 203ms instead of the roughly three seconds a first-ever cold boot costs — and it is also the reason a burst can behave in ways that look, from the outside, like the platform has simply stopped working. This post is about that gap: why bursts are the normal shape of this workload rather than an edge case, the failure modes in the order you actually hit them, and the handful of mechanisms that turn a herd back into a queue.

Bursts are the shape of the workload

Arrival is not Poisson here. Sandbox creates do not trickle in at a smooth rate with occasional clumping; they arrive in walls, and the walls are structural:

• Cron at :00 — humans write `0 * * * *` and almost never `7 * * * *`, so every tenant's scheduler agrees to fire at the same instant, and NTP enforces the agreement.
• A demo that works — someone posts a link and the traffic curve is not a ramp, it's a step. The pleasant version of an outage.
• CI fan-out — one push expands into a matrix, and a matrix is a burst by definition: twenty cells requested by one orchestrator in one loop with no sleep in it.
• Agent swarms — the spikiest of all. A planner decides to explore forty branches and forty sandboxes are requested in a single `asyncio.gather`. The model has no concept of backpressure and will not develop one.
• Retry storms — the burst your own system creates. Something times out, every client retries at once, and the herd is now synchronized more tightly than it was originally.

What makes these pointed for snapshot restore is homogeneity. It isn't four hundred workloads touching four hundred files — it's four hundred restores of one seed, reading the same chunks of the same memory image in nearly the same order, because they are all resuming the same frozen kernel from the same instruction pointer. The access pattern isn't merely correlated; it is nearly identical. Wonderful news for caching, terrible news for anyone who hasn't built the cache yet.

What a demand-paged restore actually does

Briefly, because the details matter for the failure modes. A baked snapshot is a memory image, a machine-state file, and a root filesystem. The naive restore reads the whole memory image into RAM before resuming the guest — correct, simple, and hopeless once that image is a couple of gigabytes, because every create then pays for bytes the guest will never touch.

So instead you lie to the guest. Register its memory region with `userfaultfd`, hand the file descriptor to the VMM, and resume with essentially nothing paged in. The guest touches an address, the kernel traps the fault, your handler works out which slice of the snapshot that address lives in, fetches it — a 4 MiB chunk, over HTTP Range GET, from object storage — and installs it with `UFFDIO_COPY`. From inside the guest that is a slightly slow memory access and nothing more. A recorded header marks which chunks are entirely zero, so those are filled locally without asking anyone. That is why the restore step lands around 49ms: you aren't moving 2 GiB, you're moving the few megabytes the guest actually reaches for on its way to being useful.

Now multiply by four hundred

Each restore is an independent unit — its own handler, resolver, and HTTP client, with no knowledge that four hundred siblings exist — and each faults on roughly the same chunks in roughly the same order. Without coordination the arithmetic is grim: if the hot working set is three hundred distinct chunks, four hundred restores issue on the order of a hundred and twenty thousand range requests for three hundred chunks' worth of actual bytes. You have taken a 2 GiB file and asked for it four hundred times, concurrently, from one prefix, in one second.

The failure modes then arrive in a specific and fairly reliable order.

1. The object store's opinion

Object stores are extraordinary at aggregate throughput and considerably less extraordinary at ten thousand concurrent requests against one key prefix in one second. You get back some blend of throttling responses, connection resets, and — the sneakier one — an unchanged median with a tail that has grown a second-long shadow. Nothing is down. p50 looks fine. p99 has quietly detached from reality. And because the handler sits on the critical path of a page fault, that tail is not a metric, it is a stalled guest.

2. The host NIC discovers its limits

Suppose the object store is unbothered. The bytes still have to arrive somewhere, and that somewhere is one host's network interface. Duplicate fetches mean the same chunk crosses that NIC once per restore. Saturate the link and every fetch slows together — including fetches for VMs that were nearly done — so the burst degrades as a unit rather than at the margin. This is also the failure mode with a bill attached, since most of that traffic is egress you are paying for twice, four hundred times.

3. Page cache thrash

Now the bytes are on the host and they need to live somewhere. Four hundred guests each materializing a private copy of the same hot pages means four hundred copies of identical data resident in host memory, arriving faster than anything can reclaim. The page cache — which was helpfully holding your root filesystem images, the ones that must stay local because copy-on-write needs a real block device — is evicted to make room. Disk operations that were free start costing, which slows the restores, which extends the window in which the herd is still arriving. The word for this is not "slow." It is "feedback."

4. The ugly one: a live VM that is wedged

Here's the failure mode that costs you a weekend. A `userfaultfd` fault blocks the faulting thread, and if that thread is a vCPU, the vCPU is stopped — not descheduled, stopped — until the handler installs the page. A fetch that takes four seconds because the object store is throttling you halts a guest vCPU for four seconds inside a VM that is, by every external signal, perfectly healthy: process running, metrics green, health endpoint answering if it lands on a different vCPU whose pages are resident. Meanwhile the guest's timekeeping is confused, its watchdogs are firing, and a TCP connection somewhere has given up.

Fix one: single-flight, because N faults are one fetch

The cheapest and highest-leverage fix is coalescing: when a fault arrives for a chunk already being fetched, don't start a second fetch — attach to the first and wait. The standard single-flight pattern, perhaps thirty lines of code, and it collapses in-process request amplification almost completely. It is the difference between a burst that is loud and a burst that is fatal.

// chunkResolver serves 4 MiB slices of a snapshot's memory image to the
// userfaultfd handler. Every guest page fault during restore lands here.
// On a burst, hundreds of faults -- from dozens of unrelated VMs -- ask
// for the SAME chunk within a few milliseconds of each other.
type chunkResolver struct {
	cache *SharedCache // per-host, per-seed-generation, on local disk
	src   ChunkSource  // HTTP Range GET against object storage

	mu       sync.Mutex
	inflight map[uint64]*fetch
}

type fetch struct {
	done chan struct{}
	data []byte
	err  error
}

func (r *chunkResolver) Chunk(ctx context.Context, idx uint64) ([]byte, error) {
	// 1. Local disk first. After the first restore of this seed on this
	//    host, essentially every fault ends on this line and the network
	//    is never involved again.
	if b, ok := r.cache.Get(idx); ok {
		return b, nil
	}

	// 2. Single-flight. N faults for one chunk must become ONE request.
	//    Without this block a 200-VM burst issues ~200 identical GETs for
	//    every hot chunk, which is how you turn a 2 GiB file into 400 GiB
	//    of egress and a rate-limit wall.
	r.mu.Lock()
	if f, ok := r.inflight[idx]; ok {
		r.mu.Unlock()
		select {
		case <-f.done: // ride along with the fetch already in progress
			return f.data, f.err
		case <-ctx.Done():
			return nil, ctx.Err()
		}
	}
	f := &fetch{done: make(chan struct{})}
	r.inflight[idx] = f
	r.mu.Unlock()

	f.data, f.err = r.src.Range(ctx, idx*chunkSize, chunkSize)
	if f.err == nil {
		// Durability BEFORE advertisement: write the chunk, fdatasync it,
		// atomically rename, and only then flip the present bit. A bitmap
		// that claims "present" must never outrun the bytes on disk, or a
		// crash mid-write resurrects as silent guest memory corruption --
		// which presents as a VM that boots fine and then behaves insanely.
		f.err = r.cache.PutDurable(idx, f.data)
	}
	close(f.done)

	r.mu.Lock()
	delete(r.inflight, idx)
	r.mu.Unlock()
	return f.data, f.err
}

Note the ordering in `PutDurable`, because that is the part people get wrong. The present-bitmap is a promise, and a promise must not be made before it can be kept: write the chunk, `fdatasync` it, rename it into place, and only then set the bit. Flip the bit first, lose power mid-write, and the next restore reads a chunk the bitmap swears is valid and gets garbage — which becomes guest memory, which becomes a VM that boots normally and then makes decisions no reasonable computer would make. Entirely avoidable by ordering two operations correctly.

Fix two: a shared per-host chunk cache, keyed so a re-bake self-invalidates

Single-flight dedupes faults inside one restore. It does nothing for the restore that starts ninety seconds later and asks for the same chunks. For that you want a cache that outlives any individual VM: a per-host, on-disk, sparse store of chunks any restore can read and any restore can populate. The first restore on a host pays object-store latency once; every later restore of that seed reads local disk. That is the whole trade, and it is a good one — the herd's cost becomes O(unique chunks) rather than O(unique chunks × VMs).

The design detail that matters more than it sounds is the cache key. Key it by template name and you have built a correctness bug with a delay fuse: re-bake the template, publish a new memory image under the same name, and every host in the fleet cheerfully serves stale chunks from the old image into the new snapshot's address space. Pages from two different bakes of the same kernel produce a VM that is not crashed, merely wrong. Key on a hash of the object's full identity instead — bucket plus object path — so a new bake is a new key, old entries become unreferenced, and invalidation is something you never have to remember to do. Then bound it, because a cache with no budget is a disk-full incident waiting for a quiet Sunday: a size ceiling, LRU eviction by generation, and accounting that measures allocated blocks rather than the apparent size of a sparse file.

Fix three: prefetch a recorded working set instead of faulting cold

Caching removes duplicate work; prefetching removes the latency of the work that remains. A given template's restore path is remarkably deterministic — the same kernel resuming from the same state touches nearly the same pages in nearly the same order every time. So record it: at bake time capture the chunks touched during a warm-up, write that trace next to the snapshot, and on restore walk the trace in the background while the guest starts running. By the time the guest faults, the chunk is resident and the fault is a cache hit rather than a network round trip.

The two compose, provided the prefetcher and the fault path go through the same resolver: a race between "prefetcher is fetching chunk 41" and "guest just faulted on chunk 41" then resolves into one fetch and one waiter. Don't skip coalescing on the theory that prefetch makes it unnecessary — prefetch is a prediction, and predictions are wrong at the worst possible moments.

Fix four: spread the burst, then admit it on purpose

Everything above makes one host survive a herd. The next question is whether the herd should have landed on one host at all. A scheduler scoring purely on free capacity will send an entire burst to whichever host looked emptiest thirty seconds ago, because every placement decision reads the same stale view of the world before any of them take effect. The fix is to account for in-flight placements — reserve against a host the moment you decide to send work there, not when the work arrives. Without it, load spreading works beautifully at steady state and fails precisely when it matters.

There is a real tension here: spreading a burst reduces per-host pressure but multiplies cold caches, since each host pays its own first-restore penalty. Cache locality wants concentration; capacity wants dispersion. Spread anyway — a slow create is recoverable, a saturated host is not — and close the gap with prefetch and pre-warming. Placement capacity is rarely the wall: a host carries 16,384 pre-allocated /30 subnets, so you meet memory and I/O limits long before you run out of anywhere to put a VM.

Finally, admission control — the least fashionable item on the list and the one that keeps you up the least. You cannot serve four hundred simultaneous cold restores well. You can serve forty and queue the rest, and the queued creates will finish sooner than they would have in a free-for-all, because the ones ahead of them warm the cache on their behalf. Bounded concurrency plus a queue plus a clear timeout degrades linearly; unbounded concurrency degrades like a cliff. Given a choice between a create that takes four seconds and a create that returns 200 OK for a VM with a stalled vCPU, take the four seconds every time.

Naive demand paging vs. herd-aware restore

Same architecture, same snapshot, same burst — the difference is entirely in what sits between the fault and the network.

• Requests per burst — Naive: one range GET per fault per VM, so the hot chunk set is fetched once per restore. Herd-aware: one fetch per unique chunk per host per seed generation; every other fault is served from memory or local disk.
• First restore on a host — Naive: pays object-store latency. Herd-aware: also pays it, once. That is the honest cost of the design.
• Hundredth restore on a host — Naive: pays object-store latency again, identically, forever. Herd-aware: a local-disk read, at which point create latency is dominated by everything else in the path.
• Failure under throttling — Naive: fetches stall, vCPUs stall inside apparently-healthy VMs, and the burst extends itself. Herd-aware: one bounded fetch retries on behalf of all waiters, and if it cannot be served the create fails loudly instead of hanging quietly.
• Host memory — Naive: N private copies of identical hot pages, plus page-cache eviction of rootfs images. Herd-aware: one on-disk copy per host, read by everyone, with a size budget and LRU eviction.
• Correctness after a re-bake — Naive: if the cache key is the template name, silently mixed memory from two different bakes. Herd-aware: the key is derived from object identity, so a new bake is a new key and stale entries are orphaned.
• Behavior at the limit — Naive: unbounded concurrency, cliff-shaped degradation, a tail that eats the median. Herd-aware: admission control with a queue, so the burst slows in proportion to its size instead of falling over.

Measuring it: burst first, then read the counters

None of this is worth arguing about in the abstract, because the numbers are cheap to get. Fire a burst at a cold host, fire the same burst at a warm one, and compare distributions — not means. A herd is a tail phenomenon, and averages are designed to hide tails from you.

#!/usr/bin/env bash
# Burst test: fire N creates from ONE template at the same instant and look
# at the SHAPE of the result. A thundering herd does not show up as a
# slower mean -- the mean barely moves. It shows up as a long right tail.
set -uo pipefail
N=${N:-200}

burst() {
  seq 1 "$N" | xargs -P "$N" -I{} sh -c '
    t0=$(date +%s%N)
    curl -sf -o /dev/null -X POST "$PANDASTACK_API/v1/sandboxes" \
      -H "Authorization: Bearer $PANDASTACK_API_KEY" \
      -H "content-type: application/json" \
      -d "{\"template\":\"code-interpreter\",\"ttl_seconds\":120}" || exit 0
    echo $(( ($(date +%s%N) - t0) / 1000000 ))
  ' | sort -n | awk '{a[NR]=$1} END {
      printf "n=%d p50=%sms p90=%sms p99=%sms max=%sms\n",
        NR, a[int(NR*0.50)], a[int(NR*0.90)], a[int(NR*0.99)], a[NR]
    }'
}

# Run 1: cold host, cache empty. Every hot chunk is fetched from object
# storage for the first time. This is your worst honest case.
echo -n "cold  "; burst

# Run 2: same host, same seed, cache warm. If the numbers do not collapse
# toward the steady-state create latency, your cache is not being shared
# across restores -- check the cache key, not the network.
echo -n "warm  "; burst

# Now read the counters that explain the difference. Duplicate fetches are
# the tell: chunk_fetch_total should be far SMALLER than fault_total, and
# the gap between them is exactly what single-flight + cache bought you.
curl -s localhost:9100/metrics | grep -E 'pandastack_(uffd|sandbox_boot)'

The single most diagnostic number is the ratio of chunk fetches to page faults. Roughly equal means no coalescing and no cache — you are downloading the same bytes over and over. A small fraction means the machinery is working. On the warm run the fetch count should be near zero; if it isn't, your cache key is wrong or something is invalidating entries you didn't intend to invalidate.

From the client side, stop pretending the burst is a surprise. If you know a fan-out is coming, pay the cold restore deliberately and in advance with one throwaway sandbox — then bound the fan-out's concurrency so the queue lives in your process, where you can see it, rather than in a page-fault handler, where you cannot.

from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox

TEMPLATE = "code-interpreter"


def prime(template: str) -> None:
    """Pay the object-store latency once, on purpose, before the herd.

    The first restore of a seed on a host populates that host's chunk
    cache. Every restore after it reads from local disk. So if you know a
    burst is coming -- a cron minute, a demo at the top of the hour, a
    scheduled eval sweep -- create and kill one sandbox a few seconds
    early and let it do the expensive part alone.
    """
    sbx = Sandbox.create(template=template, ttl_seconds=60)
    sbx.exec("true")
    sbx.kill()


def run_one(task: str) -> str:
    sbx = Sandbox.create(template=TEMPLATE, ttl_seconds=900)
    try:
        sbx.filesystem.write("/work/task.py", task)
        return sbx.exec("python /work/task.py", timeout_seconds=120).stdout
    finally:
        sbx.kill()


def fan_out(tasks: list[str], concurrency: int = 32) -> list[str]:
    prime(TEMPLATE)  # one cold restore, then 200 warm ones

    # Bounded concurrency is not politeness, it is self-defense. An
    # unbounded pool converts a queue you can see into a tail you cannot.
    with ThreadPoolExecutor(max_workers=concurrency) as pool:
        return list(pool.map(run_one, tasks))

When this is overkill, and what it costs

If you run tens of VMs a day from a snapshot that comfortably fits in host RAM, build none of this. Read the memory image at restore, let the page cache do its job, and go work on your product. Every mechanism here answers a scale problem, and adopting a scale solution before you have the scale problem is how systems acquire complexity nobody can explain a year later. The honest threshold is roughly "a burst regularly exceeds what one host can fetch from object storage inside your latency budget" — until then the simple path is better.

The costs are real. A shared cache is persistent state on a host that was otherwise pleasantly stateless: eviction policy, disk budgeting, corruption handling, and a new class of incident where the cache is the problem. Prefetch traces are build-time artifacts that can go stale and quietly stop helping — worse than not helping, because nobody notices. Single-flight makes one slow fetch slow for every waiter: usually right, occasionally surprising. And admission control means deliberately making some creates slower, an easy decision on a whiteboard and a harder one when a customer asks why their p99 moved.

What you buy is a shape change. Without it, restore latency scales with how many people happened to want a sandbox at the same moment — not a property you can put in a contract. With it, the first restore on a host pays object-store latency once and every restore after that is local-disk fast, so the burst that used to be an incident becomes a line on a dashboard. And the cron jobs keep firing at :00, because they were never going to do anything else.