Snapshot-Restore vs Warm Pools: Two Ways to Kill Sandbox Cold Starts

There are exactly two honest ways to hide a multi-second VM boot from the person waiting on it. You can boot the machines before anyone asks and keep a stack of them idling — a warm pool — and hand one out the instant a request lands. Or you can boot no idle machines at all, bake a fully-initialized VM into a snapshot once, and on every create map that snapshot copy-on-write and restore it — snapshot-restore on demand. Both make a sandbox appear in tens or low-hundreds of milliseconds. They diverge completely on what you pay while nobody is asking, how they behave when a burst hits, and how much capacity planning they force on you. This is the head-to-head: what each strategy actually costs, where each one wins, and why PandaStack runs snapshot-restore on every create with no warm pool of idle VMs.

Strategy 1: the warm pool

A warm pool is the obvious answer, and for good reason: it's conceptually trivial. Boot N VMs ahead of time, park them in a ready state, and keep them in a free list. A request pops one off the list — the boot already happened, so handout is near-instant — and when it's done you either reset the VM and return it to the pool or throw it away and boot a replacement. A background refill loop watches the free list and tops it back up toward N as VMs get claimed, so the pool doesn't drain to empty under steady load.

The catch is that a parked VM is a whole live machine. It holds its guest RAM resident, it has a schedulable vCPU, and the host carries all of that whether or not a single request ever arrives. To absorb a burst you have to size N for the burst — which means for most of the day most of the pool is doing nothing but existing. A warm pool is paying rent on VMs that are asleep. And a pool VM that one tenant has already touched has to be scrubbed or recycled before the next tenant gets it — that reset is its own latency and its own correctness risk.

A warm pool solves cold starts by never letting the machine go cold. That's also the bill: you are paying, continuously, for the privilege of the machine never going cold.

There's a subtler failure mode too. A warm pool is only warm up to its size. If a spike claims VMs faster than the refill loop can boot replacements, the pool drains — and the request that finds it empty falls straight through to a cold boot, the exact multi-second wait the pool existed to hide. So the pool doesn't remove cold starts; it removes them up to a provisioned ceiling and then hands them back to you at the worst possible moment, under load.

Strategy 2: snapshot-restore on demand

Snapshot-restore starts from a different premise: keep no idle VMs at all. Boot and fully initialize one VM, freeze it to disk as a small set of artifacts — the guest's physical RAM (vm.mem), the VMM's device and CPU state (vm.state), and a copy-on-write rootfs — and then, on every create, restore a fresh copy of that frozen machine. A restore isn't a boot. The guest kernel was already up when it was frozen, the devices were already probed, your userspace already reached ready; restore maps the memory file, loads the device state, and unpauses the vCPUs mid-instruction. There is no firmware, no kernel init, no systemd handshake to sit through.

The reason this is cheap enough to do on every single create — instead of hiding the cost in a pool — is that restore does not read the snapshot in. Firecracker memory-maps vm.mem with MAP_PRIVATE, a lazy copy-on-write mapping. At load time nothing is copied; the mapping just establishes that the guest's RAM is backed by the file. A page faults in only when the guest first touches it, and a write triggers a private copy of just that 4 KiB page while the original stays pristine for every other restore. So a create pays, in time and RAM, only for the working set the guest actually touches to become usable — a few hundred megabytes of a 2 GiB guest — not the whole image. Identical read-only pages are even shared across restores in the host page cache, so N sandboxes of one template cost far less than N times the guest's declared memory.

On PandaStack that math lands at a 179ms p50 create and roughly 203ms at p99, of which the snapshot-load step — mapping vm.mem and loading device state — is about 49ms and unpausing the vCPUs is another ~6ms. Every create takes that same path on any host that has the seed; there is no fast path and slow path, no pool to hit or miss. Here's the per-request shape from the SDK — create, use, kill — with nothing kept warm in between:

from pandastack import Sandbox

# create() restores a baked template snapshot on demand.
# No warm pool is consulted and nothing was kept idling for you:
# p50 ~179ms, p99 ~203ms (~49ms of that is the snapshot-load step).
box = Sandbox.create(template="code-interpreter")

# The guest is already booted and reachable — run immediately,
# no "wait for ready" loop (the readiness probe is part of create).
result = box.exec("python -c 'print(2 ** 10)'")
print(result.stdout)  # -> 1024

# Kill it. Between this and the next create, NO VM of yours is
# parked on a host burning RAM. You pay for sandboxes that exist,
# not for a pool that might be needed.
box.delete()

Making restore work across a fleet: stream the memory

MAP_PRIVATE is great when vm.mem is already on the local disk. But in a multi-host fleet, snapshots live in object storage so any host can restore any template — and the memory file might not be local yet. Downloading a multi-gigabyte vm.mem before you can restore would reintroduce exactly the latency snapshotting exists to kill, and most of those bytes are pages this restore will never touch. PandaStack streams instead: with userfaultfd, the agent backs guest memory itself and, on each page fault, fetches the surrounding 4 MiB chunk from GCS over an HTTP Range GET and installs it with UFFDIO_COPY, waking the parked vCPU. All-zero chunks are elided with no fetch, and a bake-time prefetch trace warms the hot set into a shared per-host cache. So a host restores a template it has never held locally without downloading the whole image first — the create still takes the restore path, it just pulls pages over the network on the way. Scope matters: userfaultfd streams memory, not the disk. The rootfs stays a local file because copy-on-write disk cloning (reflink or dm-snapshot) needs a local block device.

The head-to-head

Both make a sandbox appear fast. They differ entirely on what you pay for while nobody is asking, and how each behaves when everybody asks at once.

• Idle cost — Warm pool: high and continuous. Every parked VM holds resident RAM and a vCPU whether used or not; the pool bills 24/7. Snapshot-restore: ~0. Between requests nothing of yours runs — the only standing cost is snapshot artifacts on disk (cheap) plus a small pool of pre-built network slots (cheaper).
• Burst behavior — Warm pool: instant handout until the pool drains, then requests fall through to a real cold boot exactly when load is highest. Snapshot-restore: every create is the same ~179ms restore, bounded by host memory/CPU and restore throughput, not a pre-sized pool — each host just keeps stamping copies from the same on-disk snapshot.
• Memory waste — Warm pool: each idle VM wastes its full private RAM. Snapshot-restore: copy-on-write memory sharing means many restores of one template share read-only pages, so resident cost is sublinear in sandbox count.
• Capacity planning — Warm pool: you must guess N, size it for peak, and run a refill loop; guess low and you drain, guess high and you overpay. Snapshot-restore: nothing to size — no pool depth to tune, no refill loop for idle VMs.
• First-request latency — Warm pool: near-zero to hand out an already-booted VM (the boot happened earlier, off the request path). Snapshot-restore: ~179ms p50 / ~203ms p99 to build a fresh machine from cold artifacts — warm-pool-class, with none of the standing bill. The one exception is a brand-new template's very first spawn, which is a real ~3s cold boot.
• Complexity — Warm pool: simple idea, but you own pool sizing, the refill loop, and per-tenant reset/recycle correctness. Snapshot-restore: simpler to operate (no pool, every create identical) but demands a real substrate — a good snapshot/CoW layer, MAP_PRIVATE lazy page-in, and userfaultfd streaming to make it work across hosts.

Where a warm pool still earns its keep

Being honest about the trade means naming where the warm pool is genuinely the better tool. Snapshot-restore has one cost it cannot make disappear: the first cold boot that produces the snapshot. If your workload constantly spins up brand-new template identities that have never been baked on a given host — so nearly every request is that first ~3s cold boot rather than a restore — a small warm pool in front of the bake step can hide that first-boot latency where snapshot-restore structurally cannot. The same is true if you need truly zero handout latency for a fixed, known-ahead set of VMs and the idle bill is acceptable to you; a warm pool wins on raw handout speed because it does no per-request work at all.

The two aren't even mutually exclusive. You can run snapshot-restore as the default create path and keep a thin warm layer specifically to mask the cold-boot-and-bake of a new template — restore for the 99% steady state, warm pool for the rare first-of-its-kind spawn. PandaStack's own answer is a variant of that idea: no pool of idle VMs, but a warm depth of pre-built network slots (up to 16,384 /30 subnets per agent) so the plumbing is never on the hot path, plus auto-bake so the first cold boot happens once and is amortized across every restore after. The expensive, reusable thing gets built off the critical path; the VM itself never idles.

The corollary: fork is snapshot-restore pointed at your own machine

Once create is "map a memory file copy-on-write and resume," the rest of the lifecycle falls out of the same primitive — and it's a capability a warm pool simply can't offer. A fork snapshots a specific running sandbox and restores that: same MAP_PRIVATE memory map, same reflinked rootfs, just starting from your live machine instead of the template. A same-host fork runs in roughly 400–750ms because the parent's memory is already resident and the rootfs reflinks locally; a cross-host fork is 1.2–3.5s because the artifacts move over the network first. The child shares vm.mem pages with the parent copy-on-write until it writes them, so a fan-out of N branches from one warmed-up machine costs far less RAM than N fresh boots. There is no warm-pool analogue for "give me twenty copies of this exact running machine, cheaply" — that only exists because the substrate is snapshot-restore all the way down.

The summary

Two strategies, one goal. A warm pool hides cold starts by keeping VMs booted and idling — trivially simple, instant to hand out, but you pay for peak-sized idle capacity continuously, you own the sizing and refill and reset logic, and a burst that drains the pool hands the cold boot right back to you. Snapshot-restore hides them by baking a fully-initialized machine once and restoring a fresh copy on every create — MAP_PRIVATE makes the restore pay only for the working set, so a create lands at a 179ms p50 without reading the whole image, and userfaultfd streams vm.mem from object storage so any host can restore without downloading gigabytes first. The one cost it can't erase is the first ~3s cold boot that produces the snapshot, which is exactly where a thin warm layer can still help. PandaStack chose snapshot-restore on every create — no warm pool of idle VMs — because for bursty sandbox workloads it delivers warm-start latency with idle cost that rounds to zero. Compare the numbers against any provider's docs; the strategy is what to reason about first.

Every sandbox, managed Postgres database, and git-driven app on PandaStack runs on this create path, and the control-plane API and per-host agent are open source under Apache-2.0 — so you can run them on your own Linux KVM hosts and watch the restore timings, and the empty idle bill, yourself. For the create pipeline step by step start with /blog/microvm-cold-start-optimization; for the memory mechanics, /blog/microvm-snapshotting-warm-starts.