The Anatomy of a Sub-200ms MicroVM Create

"179ms" is a headline, and headlines hide the work. The number describes a PandaStack microVM create — a fresh Firecracker guest with its own kernel, hardware-isolated, ready to take commands — but it isn't a single measurement of a single thing. It's a sum. Eight distinct stages run to turn an API call into a usable sandbox, and each one has been individually filed down until it contributes single-digit or low-tens of milliseconds. This post is the itemized bill. We'll walk each line, say what it costs, and — more usefully — say why it costs that, because the why is a small catalogue of the same few tricks applied over and over: pre-allocate the expensive thing out of the hot path, copy-on-write instead of copying, and page things in lazily instead of eagerly.

Two companion pieces set the stage this one drills into. /blog/how-firecracker-boots-fast argues why a restore is not a boot; /blog/snapshot-restore-boot-path argues why there's no warm pool. Both are worth reading, and neither does what this post does, which is stand at each stage of the pipeline with a stopwatch and explain the mechanism responsible for that stage's number. If you like knowing exactly where the time goes, this is the one.

The bill, itemized

Here is the whole create, stage by stage, on the fast path — the path every create takes once a template's snapshot has been baked. Read it top to bottom; the sections after it unpack the interesting lines.

1. Allocate a NATID network slot — ~1ms. Grab a pre-built Linux network namespace (netns + veth pair + tap + iptables) from a warm pool. Cold, this would be ~100ms of ip netns add / ip link add; the pool amortizes that setup out of the hot path entirely.
2. Configure the tap device in the namespace — ~6ms. Patch the tap's MAC and routes to match the IP/MAC/gateway the guest was frozen with at bake time, so the restored guest sees the network identity it remembers.
3. Reflink the rootfs — ~4ms. Clone the template's ext4 rootfs with an XFS reflink — an O(metadata) copy-on-write clone, not an O(data) byte copy. Constant-time regardless of image size; blocks stay shared with the template until the guest writes.
4. Fork+exec Firecracker under the jailer — ~25ms. Launch the VMM process in its dropped-privilege jail. No firmware phase, no device enumeration — it comes up ready to be handed a snapshot.
5. POST /snapshot/load — ~80ms. Firecracker memory-maps vm.mem with MAP_PRIVATE and loads vm.state. The mapping is lazy and copy-on-write, so pages fault in only as the guest touches them (~49ms of this is the snapshot-restore step itself; the rest is the surrounding load).
6. POST /snapshot/state Resume — ~6ms. Unpause the vCPUs. The guest is now running mid-instruction, exactly where the snapshot froze it — no reboot, no init, no systemd.
7. Probe TCP :22 for readiness — ~40ms. Confirm the guest's network stack is live and accepting connections before returning success, so create only reports done once the box is actually usable.
8. Insert the sandbox row — ~6ms, async. Bookkeeping in the agent's database, deliberately off the critical path so it never gates readiness.

Two lines dominate: the snapshot load (~80ms) and the readiness probe (~40ms). The other six stages together are smaller than the load alone. That's not an accident — it's the result of moving every avoidable cost off the hot path. Let's take the interesting lines in turn.

Line 1: why the network is ~1ms and not ~100ms

Standing up a Linux network namespace from scratch — ip netns add, create a veth pair, move one end into the namespace, create a tap, wire up iptables — is on the order of 100ms of syscalls and netlink churn. If create did that inline, it alone would blow past the entire budget of the two dominant stages combined. So create doesn't do it inline. PandaStack pre-builds a pool of these NATID slots ahead of time and keeps them warm; a create grabs a ready one and just patches its identity.

This is the single most important idea in the whole pipeline, and it recurs: take the expensive-but-repeatable setup, do it once ahead of time, and amortize it across every create. The slot pool is the warm prebuild depth in front of a much larger address space — 16,384 /30 subnets per agent, the /16 the design tops out at. If the warm pool drains, a slot is built on demand (the ~100ms cold cost, paid only when you're allocating faster than the pool refills), so the pool is a latency optimization, not a concurrency cap. Note the asymmetry with warm-pooling whole VMs: keeping netns plumbing warm costs almost nothing (no RAM, no scheduled vCPU), whereas keeping VMs warm costs a fully resident machine each. You pre-allocate the cheap thing and restore the expensive thing.

Line 3: why the rootfs clone is ~4ms at any size

The guest needs a writable disk, and it must not scribble on the shared template image. The naive way to get that is to copy the template's rootfs — but a multi-hundred-megabyte byte copy is tens to hundreds of milliseconds, and it scales with image size, which is exactly the wrong shape for a latency budget. So PandaStack doesn't copy bytes. It reflinks: an XFS reflink clone shares every block with the template and only records new metadata pointing at those shared blocks.

That makes the clone O(metadata) instead of O(data) — a few milliseconds regardless of whether the rootfs is 200MB or 2GB, because you're writing a handful of pointers, not copying data. Blocks stay shared until the guest writes to one, at which point the filesystem copies just that block (copy-on-write at the block level). The guest gets a private, writable disk; the host stores one copy of the shared bytes plus each guest's diffs. dm-snapshot provides the same property through a different mechanism. The deeper mechanics are in /blog/copy-on-write-rootfs.

Line 5: why the snapshot load is the big one — and why it doesn't get bigger

The snapshot load is the most expensive stage, and it's the one people most misunderstand. The instinct is to model it as "read a 2GiB memory image off disk into RAM, then run," which would be slow and would scale linearly with guest memory. If that were how it worked, sub-200ms create would be impossible for any non-trivial guest, and the no-warm-pool design would collapse. It is not how it works.

When Firecracker loads the File backend it memory-maps vm.mem with MAP_PRIVATE. That flag does the heavy lifting. A MAP_PRIVATE mapping is lazy and copy-on-write at page granularity: at load time nothing is copied — the mapping merely establishes that the guest's physical RAM is backed by the file. A page is materialized only when the guest first touches it, resolved by the kernel mapping that 4KiB page straight from the file (no copy on a read); a write triggers a private copy of just that one page and leaves the original pristine for anyone else restoring the same snapshot. The consequence: the load pays, in time and RAM, only for the guest's working set between resume and ready — not for its declared memory. Restoring a 2GiB guest does not read 2GiB. That's why this line is ~80ms flat instead of a number that grows with the template's RAM. It's fork()'s copy-on-write trick, applied to an entire VM's memory.

Line 7: the ~40ms you can't optimize away (and shouldn't)

The readiness probe is the second-biggest line, and it's the one that's supposed to be there. After resume, the guest is technically running, but "running" and "reachable" aren't the same instant — the network stack needs a beat to come back to accepting connections. The probe polls TCP :22 until the guest answers, and only then does create return success. You could shave it by returning early, but then create would sometimes hand back a sandbox that isn't ready yet, and the next line of the caller's code would race the guest. This is the tax of an honest API: create returns when the box is usable, not when it's merely resumed. The ~40ms buys you the guarantee that box.exec(...) on the very next line works.

The comparison: what a cold boot pays that a restore doesn't

The reason all eight lines stay small is that none of them boots anything. To make that concrete, here's what the ~3s first-spawn cold boot pays for that the restore path simply skips. The cold boot happens exactly once per template per agent — the agent then bakes a snapshot of the ready machine, and every create afterward restores it. So you pay the left column once and the right column forever after.

• Cold boot (~3s, once per template): fork+exec the VMM, then the guest kernel initializes — brings up virtio drivers, mounts the rootfs, starts PID 1; then userspace comes up — systemd (or the init) starts services, the SSH daemon binds, the guest app warms; then the machine finally reaches a ready state. Every one of those is real work the CPU has to grind through.
• Snapshot restore (~179ms, every create): grab a pre-built network slot (~1ms), patch the tap (~6ms), reflink the rootfs (~4ms), fork+exec the jailed VMM (~25ms), map vm.mem copy-on-write and load device state (~80ms), resume the vCPUs (~6ms), probe the guest is reachable (~40ms), async-insert the row (~6ms). No kernel init, no service startup — the kernel was already up and its page cache already warm when the snapshot froze it.

A cold boot grinds a CPU through kernel init and service startup. A restore maps a memory file and resumes vCPUs. The first is measured in seconds because it does work; the second is measured in milliseconds because it mostly refuses to.

The honest framing is that PandaStack cold-boots rarely and restores constantly. Re-baking a template — new kernel, new rootfs contents — invalidates the old snapshot and triggers one fresh cold-boot-and-bake on the next spawn. The 179ms figure describes the constant case, which is essentially all of them.

The asterisk on line 5: when vm.mem lives in object storage

Line 5 assumes vm.mem is already on the host's disk. On a fresh agent, or one that has never served a given template, it might live in object storage instead — and downloading a multi-gigabyte memory image before you can restore would obliterate the entire budget many times over. So PandaStack can stream it on demand rather than downloading it, using userfaultfd, the Linux feature that delivers page-fault events to a userspace handler.

The mechanism reuses the same lazy-paging idea from line 5, just pointed at the network. Instead of a local file, Firecracker is handed a userfaultfd and the agent backs guest memory itself. Per fault: the guest touches a non-resident page → the kernel parks that vCPU and posts the fault → the handler translates the address to an offset in vm.mem, fetches the surrounding 4MiB chunk from object storage over an HTTP Range GET, and installs it with UFFDIO_COPY, waking the vCPU. All-zero chunks are elided (zero-fill, no fetch), a prefetch trace recorded at bake time warms the hot set in the background, and a shared per-host cache means the first restore pays object-storage latency once while later restores of the same template read locally. Crucially, streaming applies to memory, not disk: the rootfs must always be a local file because copy-on-write cloning needs a local block device. The full mechanics are in /blog/userfaultfd-explained; the point for the bill is that an agent can serve restores without first paying for the whole vm.mem download.

The pipeline, in code

Here's the same eight stages as an annotated timeline — roughly what the agent's create path does, with each line's contribution called out. It reads top to bottom, but remember that in production several of these overlap.

// Agent create path (elided). Comments are the per-stage bill.
func (a *Agent) Create(ctx context.Context, tmpl string) (*Sandbox, error) {
    t0 := time.Now()

    slot := a.natid.Allocate(tmpl)              // ~1ms  pre-built netns/veth/tap
    a.tap.PatchIdentity(slot, tmpl)             // ~6ms  MAC + routes -> baked identity
    rootfs := a.disk.Reflink(tmpl)              // ~4ms  O(metadata) CoW clone (XFS reflink)

    fc := a.firecracker.Spawn(slot, rootfs)     // ~25ms fork+exec VMM under the jailer

    // ~80ms: mmap vm.mem MAP_PRIVATE (lazy, copy-on-write) + load vm.state.
    // Does NOT read the full image -- pages fault in on first touch.
    fc.Post("/snapshot/load", loadBody(tmpl))
    fc.Patch("/vm", `{"state":"Resumed"}`)      // ~6ms  unpause vCPUs; guest runs mid-instruction

    a.probeTCP(slot, 22)                         // ~40ms wait until the guest is actually reachable

    go a.db.InsertSandbox(slot, tmpl)            // ~6ms  async -- off the critical path

    // t0 -> here: ~179ms p50, ~203ms p99. No kernel boot happened.
    return &Sandbox{Slot: slot, BootMS: time.Since(t0).Milliseconds()}, nil
}

And from the caller's side, all of that is one line. The SDK create returns a ready sandbox — the readiness probe has already confirmed the guest is up — so the next line can run a command in it immediately.

from pandastack import Sandbox

# Restores the baked "base" snapshot on demand. p50 ~179ms end to end;
# ~80ms of that is the snapshot-load line, ~40ms the readiness probe.
# Nothing was kept warm waiting for you -- it was built from cold artifacts.
box = Sandbox.create(template="base", ttl_seconds=3600)

# create() already probed TCP :22, so the box is usable right now.
out = box.exec("python -c 'print(2 ** 10)'")
print(out.stdout)  # 1024

The pattern behind the bill

Step back from the eight lines and there are really only three ideas doing the work. Pre-allocation: take the expensive-but-repeatable setup (the network namespace) and do it once ahead of time, so the hot path just grabs a warm slot — ~1ms instead of ~100ms. Copy-on-write: never copy what you can share until it's written, whether that's the rootfs (reflink, ~4ms at any size) or the guest's RAM (MAP_PRIVATE, so the load pays only for the working set). Lazy paging: materialize pages on first touch instead of eagerly up front — the same trick that keeps the local load flat also lets userfaultfd stream memory from object storage without a giant download. Snapshot-restore-versus-cold-boot is what makes all three usable at once: you pay the ~3s boot exactly once per template, bake the result, and then every create is a restore that these three tricks keep under 200ms.

For fork, the same primitive points at a live machine instead of the clean template — same copy-on-write memory and reflinked rootfs, starting from your running sandbox. A same-host fork lands in 400–750ms because the parent's memory is already resident locally; cross-host is 1.2–3.5s because the artifacts have to cross the network first. It's the create bill again, with a different starting snapshot.

Every sandbox, managed Postgres database, and git-driven app on PandaStack runs on exactly this path, and the core is open source under Apache-2.0 — so you can run the control-plane API and per-host agent on your own Linux KVM hosts and read your own itemized bill off the boot metrics. For the boot-vs-restore framing start with /blog/how-firecracker-boots-fast; for the no-warm-pool argument, /blog/snapshot-restore-boot-path; for the memory internals, /blog/firecracker-memory-snapshots.