Firecracker Snapshot/Restore vs CRIU Checkpoint/Restore

CRIU (Checkpoint/Restore In Userspace) and Firecracker snapshot/restore are often reached for to solve the same-sounding problem — freeze a running thing to disk, thaw it later, skip the cold start — but they operate at completely different layers of the stack, and that layer difference decides everything: what gets captured, what isolation boundary the restored thing lands in, what breaks, and what portability you get. CRIU checkpoints a Linux process tree: its memory, open file descriptors, sockets, even live TCP connections, freezing all of it as a set of image files you can restore back onto a host kernel. Firecracker snapshots an entire microVM — the guest kernel, its physical RAM, and every emulated device's state — at the hypervisor level, so restoring resumes a whole virtual machine mid-instruction. This post compares the two honestly: what each one actually captures, the isolation boundary you get back on restore, the fidelity gotchas each fights, how userfaultfd shows up in both, and why PandaStack builds its sub-second create path on VM-level snapshots specifically. (For CRIU-specific mechanics, verify against the CRIU project docs — this post describes it qualitatively and reserves concrete numbers for PandaStack.)

Same idea, different layer of the stack

The cleanest way to hold the two apart is to ask: when you thaw, what comes back? With CRIU, a set of processes comes back — running on the same kind of Linux kernel they were checkpointed on, sharing that kernel with everything else on the host. With Firecracker, a whole machine comes back — its own guest kernel, its own page tables, its own virtual devices — running under a hypervisor, sharing nothing with other guests but the physical CPU and the host's KVM layer.

That single distinction — process tree vs entire virtual machine — cascades into every other difference. CRIU has to understand and reconstruct kernel-managed resources (which FD numbers were open, what they pointed at, what state a socket was in) because those live in the host kernel, outside the process. Firecracker doesn't reconstruct any of that, because the guest kernel that owns those resources is itself part of the snapshot. The guest's open files, its sockets, its process table — all of it is just bytes inside vm.mem, frozen and thawed as one opaque blob from the host's point of view.

What CRIU captures: a process tree, kernel resources and all

CRIU works entirely in userspace (with kernel support behind it) to freeze a process or process tree. Conceptually it walks everything the kernel knows about those processes and serializes it: the virtual memory areas and their contents, the register state of every thread, the open file descriptors and what they reference, pipes, UNIX and network sockets, timers, and the process hierarchy itself. On restore, it re-creates processes with the right PIDs, reopens files, re-maps memory, re-establishes the relationships, and lets the tree run again. The specifics of exactly which resource types are supported and how they're serialized are an evolving matter — verify against the CRIU docs for your version — but the shape is: rebuild kernel-visible state from image files.

This is what makes CRIU powerful for container live-migration and for warm-starting long-init processes. Freeze a container's process tree on host A, ship the images to host B, restore, and the application keeps running as if it never moved. Its canonical homes are container checkpoint/restore (integrated into container runtimes) and reducing startup latency for processes with expensive initialization.

The famous party trick is TCP: CRIU can, under the right conditions, checkpoint and restore a live TCP connection — draining and re-establishing socket state so the peer never notices the process was frozen and moved. When it works it feels like a magic trick. It's also exactly where the fidelity problems concentrate, because a live socket's correctness depends on state that lives partly in the kernel, partly on the wire, and partly in a peer you don't control.

# Illustrative CRIU usage — verify flags/behavior against current CRIU docs.

# Checkpoint ("dump") a running process tree to an image directory.
# --shell-job for a process started from a shell; --tcp-established to
# attempt capturing live TCP connections.
criu dump \
  --tree $(pgrep -f my-service) \
  --images-dir /checkpoint/my-service \
  --shell-job \
  --tcp-established \
  --leave-running          # keep the original running after dump

# ... later, on the same host or another compatible one ...

# Restore the process tree from the images. It comes back on THIS host's
# kernel, sharing it with everything else already running here.
criu restore \
  --images-dir /checkpoint/my-service \
  --shell-job \
  --tcp-established \
  --restore-detached

What Firecracker captures: the whole microVM

A Firecracker snapshot is a running microVM frozen at one instant, serialized to a small fixed set of files: vm.mem is the guest's entire physical RAM byte for byte; vm.state is the VMM's serialized device and CPU state — vCPU registers, the in-kernel interrupt controller, the clock, and every virtio device's configuration and queue pointers; and the rootfs is the disk the guest was running against. Restoring is not a boot. The host launches a fresh Firecracker process, maps vm.mem, loads vm.state, and resumes the vCPUs — the guest picks up mid-instruction with its page cache warm, its processes still running, its open files still open. There is no init, no device re-probe, no systemd handshake.

Crucially, the guest kernel is inside the snapshot. Everything CRIU has to painstakingly reconstruct — the file descriptor table, the socket state, the process tree — is just guest-kernel data structures sitting in vm.mem. Firecracker doesn't know or care what any of those bytes mean; it maps the memory and the guest's own kernel finds its data structures exactly where it left them. The isolation boundary you restore into is a fresh guest kernel behind KVM hardware virtualization, not a shared host kernel. The mechanics of how that memory map stays cheap live in /blog/firecracker-memory-snapshots.

The comparison, dimension by dimension

Laid side by side, the trade-offs follow directly from the layer each one operates at.

• Unit captured — CRIU: a process tree (memory, threads, FDs, sockets, timers) running on the host kernel. Firecracker: an entire microVM — guest kernel, all of guest RAM, and every emulated device's state.
• Isolation boundary on restore — CRIU: processes land back on a shared host kernel; isolation is whatever namespaces/cgroups/seccomp wrapped them. Firecracker: the guest resumes behind its own kernel and KVM hardware virtualization — a VM-strength boundary, restored intact.
• Kernel coupling — CRIU: restore host must present a compatible kernel and feature set to reconstruct kernel-visible resources; verify constraints against CRIU docs. Firecracker: the guest kernel is in the snapshot, so it's decoupled from the host kernel — but the snapshot pins that specific guest kernel/version.
• Portability — CRIU: portable across hosts within kernel/architecture compatibility limits; used for container live-migration. Firecracker: portable across hosts, with CPU templates masking the CPU feature set so a snapshot restores on differing hardware — but you carry the guest kernel and rootfs with it.
• Lazy restore — CRIU: supports lazy-pages restore, streaming memory back on demand via userfaultfd instead of copying it all up front. Firecracker: supports a userfaultfd (UFFD) backend, so guest RAM can be paged in on demand — locally copy-on-write, or streamed from object storage.
• Typical use — CRIU: container live-migration, checkpoint/restore of long-init processes, saving/moving running workloads. Firecracker: killing microVM cold starts — restore a baked snapshot on every create instead of booting, with a full VM boundary each time.

Fidelity and the gotchas each one fights

Neither is free magic, and the failure modes are as layer-specific as the strengths.

Where CRIU gets hard

Because CRIU reconstructs kernel-visible state, anything that reference something outside the process is where it struggles. External resources — a file the checkpoint assumes still exists at restore, a device node, a connection to a peer the restore host can't reach — have to be present and consistent, or the restore is meaningless even if it technically succeeds. Certain kinds of open file descriptors and exotic kernel objects have historically been the difficult cases (verify which against the CRIU docs for your version). Live TCP restore depends on the peer, the network path, and kernel cooperation all lining up. And there's kernel-version coupling: because CRIU is re-establishing resources the host kernel owns, the restore host's kernel has to be able to present what the images describe. The general principle: CRIU is only as good as its coverage of every resource type your process touches, and that coverage is a moving target worth checking against upstream.

Where Firecracker snapshots get hard

Firecracker sidesteps the kernel-resource reconstruction problem entirely — but pays elsewhere. A snapshot pins the guest kernel: whatever kernel was running is what you restore, so you don't get host-kernel upgrades for free inside old snapshots. Baked-in RAM and vCPU count are fixed at snapshot time — you can't grow a restored guest's memory beyond what the snapshot froze. And two guest-state hazards deserve real caution. First, the guest clock is frozen at bake time, so a restored guest wakes believing it's the moment it was snapshotted; anything time-sensitive — TLS certificate validity, tokens, timers — needs a clock resync on resume or it breaks. Second, anything a running guest treats as unique-per-boot — random seeds, session keys, in-memory secrets — is identical across every restore of the same snapshot unless you deliberately reseed, which has real security implications (see /blog/firecracker-snapshot-secrets-security). The isolation is excellent; the sameness is the sharp edge.

userfaultfd shows up in both

Here's the satisfying convergence: both systems reach for the same Linux primitive to make restore fast. userfaultfd is a kernel feature that delivers page-fault events to a user-space handler instead of the kernel resolving them itself — so a program can start before its full memory image is present, and pages get populated on demand, exactly when and only when the program touches them (the deep dive is at /blog/userfaultfd-explained).

CRIU uses it for lazy-pages restore: rather than copy the whole memory image back before the process runs, it registers the restored memory with userfaultfd and streams pages from the image (or a remote source, for post-copy migration) as the process faults on them. This shrinks restore-to-first-instruction time and is central to post-copy live migration.

Firecracker uses the very same mechanism for its UFFD restore backend. Instead of pointing the VMM at a local memory file, you hand it a userfaultfd and back guest RAM from your own handler. PandaStack uses that to stream vm.mem from object storage: the guest faults on a page, the kernel parks the vCPU and posts the fault to the agent, which fetches the surrounding 4 MiB chunk over an HTTP Range GET and installs it with UFFDIO_COPY — so an agent can restore a template it has never held locally without first downloading a multi-gigabyte memory image. Same kernel primitive, same lazy-by-default philosophy; one applies it to a process's address space, the other to a whole guest's physical RAM.

Why PandaStack builds on VM-level snapshots

PandaStack's job is to run untrusted, arbitrary code for AI agents and multi-tenant workloads, and to do it with sub-second creates and no warm pool. That job selects VM-level snapshots almost by itself. The isolation boundary is the deciding factor: restoring a Firecracker snapshot lands you behind a fresh guest kernel and KVM hardware virtualization, which is the boundary you actually want between two tenants' code. A process-level checkpoint restores onto a shared host kernel — a boundary that's fine for migrating your own trusted workload but not the one you'd choose for running someone's arbitrary Python.

The other factor is that VM snapshots make the whole restore opaque and uniform. Because the guest kernel and all its resources are just bytes in vm.mem, there's no per-resource reconstruction to get right and no host-kernel-compatibility negotiation on every restore — the host maps memory and resumes vCPUs, full stop. That uniformity is what lets PandaStack run the identical restore path for every create, and it's why the numbers are what they are: every create restores a baked snapshot at a 179ms p50 (roughly 203ms p99), with only the first spawn of a template paying a ~3s cold boot to bake the snapshot in the first place. Fork rides the same primitive — same-host forks land in 400–750ms, cross-host in 1.2–3.5s once artifacts move over the network — and a managed Postgres database, which blocks on real bootstrap, takes 30–90s. Networking is pre-provisioned as 16,384 /30 subnets per agent so the network slot is warm before restore even begins. The full create pipeline is in /blog/snapshot-restore-boot-path.

from pandastack import Sandbox

# create() restores a baked microVM snapshot on demand — a full guest
# kernel behind KVM, not a process tree on a shared host kernel.
# p50 ~179ms, ~203ms p99. First-ever spawn of a template cold-boots (~3s)
# then bakes; every create after that restores.
box = Sandbox.create(template="base", ttl_seconds=3600)
box.exec("pip install numpy && python -c 'import numpy; print(numpy.__version__)'")

# snapshot() freezes THIS running microVM — guest RAM + device state + rootfs.
checkpoint = box.snapshot()

# fork() restores that frozen machine copy-on-write. The child is a whole
# separate microVM sharing vm.mem pages with the parent until it writes.
# ~400-750ms same-host; 1.2-3.5s cross-host (artifacts move over the network).
branch = box.fork()
print(branch.id, "forked from", checkpoint)

None of this makes CRIU wrong — for live-migrating a trusted container or warm-starting a slow-init process on infrastructure you own, checkpointing a process tree is exactly the right tool, and its ability to move a running workload without a full VM around it is a genuine strength. The point is that the layer you checkpoint at is a design decision, and it should follow the isolation boundary you need. PandaStack needs a VM-strength boundary on every restore, so it snapshots the VM.

The summary

CRIU checkpoints a Linux process tree at the process level — its memory, FDs, sockets, even live TCP — and restores it onto a shared host kernel, which makes it excellent for container live-migration and warm-starting slow-init processes, but means it must faithfully reconstruct kernel-visible state and external resources (verify the exact coverage against the CRIU docs). Firecracker snapshots an entire microVM at the hypervisor level — guest kernel, all of guest RAM, device state — and restores it behind a fresh guest kernel and KVM, so there's nothing to reconstruct: the guest never knows it was frozen, but you inherit a pinned kernel, fixed RAM, a frozen clock, and byte-identical restores to manage. Both use userfaultfd to make restore lazy — CRIU for lazy-pages restore, Firecracker for on-demand memory streaming. PandaStack builds on VM-level snapshots because the isolation boundary is the whole point for untrusted multi-tenant code, and the core is open source under Apache-2.0 — so you can run the control-plane API and per-host agent on your own Linux KVM hosts and watch the restore path yourself. For the memory mechanics start with /blog/firecracker-memory-snapshots; for the create pipeline, /blog/snapshot-restore-boot-path; for the streaming primitive, /blog/userfaultfd-explained.