Kill Cold Starts with microVM Snapshotting: Warm Starts Without a Warm Pool

There are two ways to make a sandbox appear instantly. The first is to keep a warm pool: boot a fleet of VMs ahead of time and leave them idling, so a request just leases one that's already up. It works, and it's a tax — a warm pool is you paying the cloud to keep computers awake in case someone shows up. The second way is snapshotting: boot and fully initialize one VM, freeze it to disk, and restore a fresh copy on demand for every request. Restore lands in tens of milliseconds, so you get warm-start latency without keeping anything warm. This post is about why that second approach wins — what's actually inside a snapshot, why restoring is far faster than booting, how copy-on-write memory and on-demand streaming keep restore cheap, and the economic argument that makes idle cost round to zero.

The cold-start problem, and the expensive way out

A cold start is the wait between "I need compute" and "the compute is ready to run my code." For a real VM that's seconds: firmware, kernel boot, device probing, init, and then your own userspace — package managers, interpreters, framework startup — all coming up before the first useful instruction. Seconds of latency on the request path is unacceptable for an API, an AI agent tool call, or a code interpreter, so the industry's default answer is to pre-pay: boot the machines before anyone asks.

That's the warm pool. Keep N machines booted and parked; a request leases one and hands it back (or throws it away) when done. The latency problem is solved because the boot already happened. But a parked VM is a fully resident machine — it holds its RAM, it has a scheduled vCPU, and the host carries it whether or not anyone ever uses it. To absorb a burst you size N for the burst, which means most of the time most of the pool is idle. You are paying, continuously, for capacity to sit and wait. And a pool VM that's been used by one tenant has to be reset or recycled before the next — its own latency, its own correctness risk.

A warm pool pays to keep machines running so start is fast. Snapshotting makes start fast enough that you never keep a machine running you don't need.

What a microVM snapshot actually is

A microVM is a real machine frozen mid-execution: a guest kernel with live page tables, a vCPU sitting between two instructions, and a set of virtio devices each holding queue and configuration state. A snapshot serializes all of it at one instant into a small, fixed set of artifacts. There are three things in it.

• Guest memory (vm.mem) — the guest's entire physical RAM, byte for byte. This is the large artifact: a 2 GiB guest produces a 2 GiB memory file. Everything the guest was holding — the page cache, running process memory, kernel data structures — lives here.
• Device + CPU state (vm.state) — the VMM's serialized state: vCPU registers, the in-kernel interrupt controller, the clock, and every virtio device's configuration and queue pointers. Tiny next to vm.mem, but it's what lets the guest resume mid-instruction instead of rebooting.
• Rootfs — the disk. It's cloned as a copy-on-write image (XFS reflink, or dm-snapshot) rather than copied block-for-block, so it shares data extents with the template until the guest writes.

The property that matters: this is a paused running machine, not a disk image you boot from. Restore it and the guest sees no reboot — processes that were mid-execution keep running, the page cache stays warm, open files stay open, there is no init or systemd handshake. The machine was booted and fully initialized once, at bake time, and every restore inherits that finished state. That is the whole reason restore is measured in milliseconds while a cold boot is measured in seconds.

Why restore is far faster than a boot

A boot builds a machine up from nothing: the CPU comes out of reset, firmware runs, the kernel decompresses and initializes, devices are enumerated and driven, init launches, and only then does your userspace start doing work. Every one of those phases is real time, and the userspace phase — pulling up interpreters, warming caches, connecting to dependencies — is often the slowest part and the part you most want to skip.

Restore skips all of it. The guest kernel was already up, the devices were already probed, your userspace already reached a ready state — that's what got frozen. Mechanically, restore is closer to "map a memory file and unpause the vCPUs" than to "start a computer." On PandaStack the snapshot-load step — mapping vm.mem and loading device state — is about 49ms, and unpausing the vCPUs is another ~6ms. A full create around that lands at a 179ms p50 and roughly a 203ms p99. Compare that to the ~3s a genuine cold boot takes: restore is more than an order of magnitude faster, and the gap is entirely the boot phases you no longer run.

# Snapshot a fully-booted, fully-initialized guest ONCE, then restore
# a fresh copy per request. Firecracker drives this over its API socket.

# --- Bake: freeze the running machine (do this once) ---
# 1. Pause the vCPUs so state is consistent
curl --unix-socket /run/fc.sock -X PATCH 'http://localhost/vm' \
  -d '{"state": "Paused"}'

# 2. Write vm.state + vm.mem to disk (Full = the whole RAM image)
curl --unix-socket /run/fc.sock -X PUT 'http://localhost/snapshot/create' \
  -d '{
        "snapshot_type": "Full",
        "snapshot_path": "/seed/base/vm.state",
        "mem_file_path": "/seed/base/vm.mem"
      }'

# --- Restore: on every request, into a fresh Firecracker process ---
# 3. Load the snapshot. backend_type "File" maps vm.mem copy-on-write.
curl --unix-socket /run/fc2.sock -X PUT 'http://localhost/snapshot/load' \
  -d '{
        "snapshot_path": "/seed/base/vm.state",
        "mem_backend": {"backend_type": "File", "backend_path": "/seed/base/vm.mem"},
        "resume_vm": false
      }'

# 4. Unpause — the guest resumes exactly where it froze. No reboot.
curl --unix-socket /run/fc2.sock -X PATCH 'http://localhost/vm' \
  -d '{"state": "Resumed"}'

The trick that makes restore cheap: MAP_PRIVATE copy-on-write

The naive mental model of "load a 2 GiB snapshot" is "read 2 GiB off disk into RAM, then run." If that were true, restore latency would scale with guest memory and forking would be ruinously expensive — and snapshotting wouldn't be fast enough to replace a warm pool. It isn't what happens. When Firecracker loads the File backend, it memory-maps vm.mem with MAP_PRIVATE, and that flag does the heavy lifting.

A MAP_PRIVATE mapping is lazy and copy-on-write at page granularity:

• Nothing is eagerly copied at load time. The mapping just establishes that the guest's physical RAM is backed by this file. No bytes move yet, which is why the ~49ms load step doesn't grow with guest RAM size.
• A read fault — the guest touches a page it hasn't touched since restore — is resolved by the kernel mapping that 4 KiB page straight from the file, no copy. Multiple guests restoring the same snapshot share those identical read-only pages in the host page cache.
• A write fault triggers the copy-on-write: the kernel makes a private 4 KiB copy of just that one page for this guest, and the original file page stays pristine for everyone else.

So a restore pays, in time and RAM, only for the pages the guest actually touches between resume and ready — its working set — not for the full image. A 2 GiB guest that touches a few hundred megabytes to become usable pays for a few hundred megabytes; the rest of vm.mem is never read. It's the same copy-on-write trick that makes fork() of a Unix process cheap, applied to an entire virtual machine's RAM. And because identical read-only pages are shared across guests, restoring many copies of the same template is far cheaper in host memory than booting many independent VMs.

Don't download gigabytes: stream memory with userfaultfd

MAP_PRIVATE is great when vm.mem is already on the local disk. But in a multi-host fleet, snapshots live in object storage so any host can restore any template — and the memory file might not be local yet. The naive path is to download the whole multi-gigabyte vm.mem before you can restore, which is pure dead time, and most of those bytes are pages this restore will never touch. That download would reintroduce exactly the latency snapshotting exists to kill.

userfaultfd deletes the wait. It's a Linux kernel feature that delivers page-fault events to a user-space handler. Instead of pointing Firecracker at a local file, the agent hands it a userfaultfd and backs guest memory itself, fetching pages on demand. Per fault:

1. The agent opens a userfaultfd handler and starts Firecracker in UFFD restore mode. Firecracker connects and sends the userfaultfd descriptor (via SCM_RIGHTS) plus the layout mapping guest-physical regions to offsets in vm.mem.
2. The guest resumes and touches a page that isn't resident. The CPU raises a fault; the kernel sees the address is registered with userfaultfd, posts an event, and parks the faulting vCPU instead of resolving it.
3. The handler reads the event, translates the faulting address into an offset in vm.mem, and fetches the surrounding 4 MiB chunk from object storage over an HTTP Range GET.
4. It installs the bytes with UFFDIO_COPY, which atomically places the page(s) and wakes the parked vCPU. The vCPU resumes exactly where it faulted, never knowing the page came over the network.

Now a host can restore a template it has never held locally without first downloading a multi-gigabyte image — the guest pulls only the pages it actually touches. Two optimizations keep it close to local-disk speed: all-zero chunks are elided entirely (a fault in a zeroed region is served with a zero-fill and no fetch — a surprising fraction of guest RAM is untouched zeros), and a prefetch trace recorded at bake time replays the hot page set in the background so chunks are often already in a shared per-host cache by the time a vCPU faults. Be precise about scope, though: userfaultfd streams memory, not the disk. The rootfs still has to be a local file because copy-on-write disk cloning (reflink or dm-snapshot) needs a local block device. Streaming removes the big vm.mem download specifically.

Warm pool vs. snapshot-restore on demand

Both approaches make a sandbox appear fast. They differ entirely in what you pay for while nobody is asking, and how the cost scales.

• Approach — Warm pool: keep N idle VMs booted and parked; a request leases one. Fast to hand out, but you pay for N machines' worth of RAM and CPU continuously, you must size N for peak, each VM must be reset between tenants, and idle cost scales directly with the pool. A slow ramp of real traffic still means a full pool of paid-for idle machines.
• Approach — Snapshot-restore on demand: keep a baked snapshot on disk (plus cheap pre-built network plumbing) and restore a fresh copy per request. Nothing runs between requests, so idle cost is ~0; you pay only for sandboxes that actually exist; every restore starts from the same clean baked baseline (no reset step, no cross-tenant leakage risk); and copy-on-write memory means many restores of one template share host RAM.
• Latency — Warm pool: near-zero to hand out an already-booted VM, but you paid for that speed with continuous idle capacity. Snapshot-restore: ~179ms p50 / ~203ms p99 to build a fresh machine from cold artifacts — warm-pool-class latency with none of the standing bill.
• First-ever start — Warm pool: hidden, because the boot happened before the request. Snapshot-restore: the very first spawn of a template does a real ~3s cold boot to produce the snapshot, then bakes it; every restore after is on the fast path. Cold-boot rarely, restore constantly.
• Scaling to a spike — Warm pool: capped by the pool size you provisioned; overflow requests eat a cold boot anyway. Snapshot-restore: bounded by host memory/CPU and restore throughput, not a pre-sized pool — each host just keeps stamping copies from the same on-disk snapshot.

The economic argument: idle cost near zero

Strip it to the money. A warm pool's cost is (pool size) × (per-VM RAM + CPU) × (time), running whether or not a single request arrives. To hide cold starts during a burst you must keep the pool sized for the burst, so your bill is set by your peak and paid at all times — including the 3am trough when the pool is almost entirely idle. Utilization on a warm pool sized for peak is, by construction, low most of the time, and every idle VM is real money.

Snapshotting changes the shape of the cost. Between requests, nothing of yours is running on the host — the only standing cost is the snapshot artifacts sitting on disk (cheap) and a small pool of pre-built network slots (cheaper). You pay compute for the sandboxes that actually exist, for as long as they exist, and nothing for the gaps between them. Idle cost rounds to zero, so the bill tracks real usage instead of provisioned peak. The one-time ~3s cold boot to bake a template is amortized across every subsequent restore — you cold-boot rarely and restore constantly — and copy-on-write memory sharing makes the resident cost of many same-template sandboxes sublinear. That's the whole pitch: warm-pool speed, without paying the cloud to keep computers awake in case someone shows up.

Same primitive from the SDK: create, snapshot, fork

Once you see "restore" as "map a memory file copy-on-write and resume," the rest of the lifecycle falls out of the same primitive. A create restores the generic baked template snapshot. A snapshot freezes a specific running sandbox, and a fork restores that — same MAP_PRIVATE memory map, same reflinked rootfs, just starting from your live machine instead of the template. A same-host fork runs in roughly 400–750ms because the parent's memory is already resident and the rootfs reflinks locally; a cross-host fork is 1.2–3.5s because the artifacts have to move over the network first. From the SDK it's a couple of lines:

from pandastack import Sandbox

# create() restores a baked template snapshot on demand.
# No warm pool is consulted: p50 ~179ms (~49ms is the snapshot-load step),
# ~203ms p99. Nothing was kept running waiting for you.
box = Sandbox.create(template="base", ttl_seconds=3600)
box.exec("git clone --depth 1 https://github.com/acme/service /work")
box.exec("cd /work && npm ci")  # build up warm in-memory + on-disk state

# snapshot() freezes THIS running machine: its RAM + device state + rootfs.
checkpoint = box.snapshot()

# fork() restores that frozen machine copy-on-write (~400-750ms same-host).
# The child shares vm.mem pages with the parent until it writes them, so a
# fan-out of N branches costs far less RAM than N fresh boots.
branch = box.fork()
print(branch.id, "branched from", checkpoint)

# Between all of this, no VM of yours is parked and idling on the host.
# You pay for the sandboxes that exist, not for a pool that might be needed.

The first spawn of a brand-new template on a host may hit the ~3s cold-boot-and-bake; every create after that takes the restore path. There's no flag to flip and no pool to size — the system bakes once and restores forever, and forking is the same lazy copy-on-write map pointed at your own frozen machine.

The summary

Cold starts have two cures. A warm pool hides them by keeping VMs booted and idling — fast to hand out, but you pay for peak-sized idle capacity continuously and reset each VM between tenants. Snapshotting cures them differently: boot and initialize one machine, freeze its guest RAM (vm.mem), device and CPU state (vm.state), and copy-on-write rootfs to disk, and restore a fresh copy on demand for every request. Restore maps memory with MAP_PRIVATE so pages fault in lazily and copy-on-write, which is why a create lands at a 179ms p50 without reading the whole image and why many same-template sandboxes share host RAM. userfaultfd extends the same on-demand page-in to stream vm.mem from object storage, so a host restores a template it never held locally without downloading gigabytes first. The one-time ~3s bake amortizes across every restore. The result is warm-start latency with idle cost near zero — you stop paying the cloud to keep computers awake in case someone shows up.

Every sandbox, managed Postgres database, and git-driven app on PandaStack runs on exactly this path, and the core is open source under Apache-2.0 — so you can run the control-plane API and per-host agent on your own Linux KVM hosts and watch the restore timings yourself. For the create pipeline step by step start with /blog/snapshot-restore-boot-path; for the memory mechanics, /blog/firecracker-memory-snapshots; for streaming, /blog/userfaultfd-explained.