Your Firecracker Snapshot Restore Failed: A Field Guide

A Firecracker snapshot restore has two modes. It works, in a few tens of milliseconds, and the guest resumes mid-instruction as if nothing happened — which still feels like a magic trick after a few thousand of them. Or it fails, and you get a terse HTTP 400 naming something you didn't think was involved, or worse, nothing at all: a VMM process that exits before it ever answers the socket.

I'm Ajay; I built PandaStack, where every sandbox create is a snapshot restore rather than a boot, so I've debugged an unreasonable number of these. This is the field guide I wish I'd had: seven classes of restore failure, what each looks like from the outside, why it happens, and the fix. Then a bisection method that separates "the snapshot is bad" from "my restore config is bad," which is the only question that matters at 2am.

What a restore actually is (and therefore what can disagree)

A restore is not a boot. Booting is a negotiation: firmware and kernel discover hardware, drivers bind, and everything ends up consistent because it was built from scratch. A restore skips the negotiation and asserts a result. Three things have to agree for that assertion to hold.

• The VMM state file (`vm.state`): serialized vCPU registers, device model state, and VMM-internal structures. The authoritative record of what the guest believes about its hardware.
• The guest memory image (`vm.mem`): every page of guest RAM, either handed to the VMM as a file to map or served on demand by your own userfaultfd handler.
• The restore-time config: drive IDs and host paths, network interfaces and their host taps, the vsock config, and the choice of memory backend.

Almost every restore failure is a disagreement between those three, not corruption. The state file says two block devices with specific IDs; your config offers one. The image is 4 GiB per the state file and 3.7 GiB per the filesystem, because a download got truncated. The load path is a series of consistency checks, and each class below is one of them failing.

# The two shapes of PUT /snapshot/load. Same snapshot, different memory backend.
FC_SOCK=/tmp/fc.sock
rm -f "$FC_SOCK"                     # a stale socket file is its own failure class
firecracker --api-sock "$FC_SOCK" &

# Variant A: file-backed. The VMM mmaps vm.mem itself, MAP_PRIVATE, lazily.
curl -sS --unix-socket "$FC_SOCK" -X PUT 'http://localhost/snapshot/load' \
  -H 'Content-Type: application/json' \
  -d '{
    "snapshot_path": "/snap/gen-42/vm.state",
    "mem_backend": { "backend_type": "File", "backend_path": "/snap/gen-42/vm.mem" },
    "enable_diff_snapshots": false,
    "resume_vm": false
  }'

# Variant B: userfaultfd. backend_path is a UNIX socket YOUR handler is already
# listening on; it receives the uffd + region layout over SCM_RIGHTS and answers
# page faults. Required for hugepage snapshots -- see below.
curl -sS --unix-socket "$FC_SOCK" -X PUT 'http://localhost/snapshot/load' \
  -H 'Content-Type: application/json' \
  -d '{
    "snapshot_path": "/snap/gen-42/vm.state",
    "mem_backend": { "backend_type": "Uffd", "backend_path": "/run/fc/uffd.sock" },
    "enable_diff_snapshots": false,
    "resume_vm": false
  }'

# Load with resume_vm=false, health-check, THEN resume. A failure you can see is
# worth more than the few milliseconds you saved.
curl -sS --unix-socket "$FC_SOCK" -X PUT 'http://localhost/vm' \
  -H 'Content-Type: application/json' -d '{"state": "Resumed"}'

Classes 1 and 2: the binary and the backend

Version and state incompatibility

What it looks like: the load call fails immediately, before anything touches memory, complaining about the snapshot version, the state format, or a field the VMM can't deserialize. Nothing in the guest ever runs. If you upgraded Firecracker recently, check this first.

Why: `vm.state` isn't a documented interchange format, it's a serialization of VMM-internal structures. Firecracker versions its snapshot format and supports a bounded compatibility window, but that window is a per-release promise — new device state fields, changed layouts, and removed features all move it. Restoring across a version boundary is either refused or, in the more exciting failure mode, accepted into a guest that misbehaves later.

The fix is organizational. Pin the VMM version per snapshot generation and store it beside the artifacts, so the restore path refuses a mismatch loudly instead of discovering it inside a customer request. Treat an upgrade as a re-bake event: publish a new generation with the new binary, keep the old one restorable by the old binary until nothing references it. Rolling a VMM upgrade across a fleet without re-baking is how a routine deploy becomes a fleet-wide restore outage.

Firecracker's `snapshot-editor` is the instrument here: it reports what a state file claims about itself, and rebases a diff memory file onto its base. Cross-CPU restore is a related axis — a snapshot carries the CPUID the guest saw at bake time, so moving it to a different CPU model can load fine and then trap on an instruction the guest was told it had. CPU templates normalize that.

Memory backend mismatch

What it looks like: the load is rejected on the backend field itself, or it hangs because you pointed at a UFFD socket path nothing is listening on. Pure configuration — and the class most likely to arrive as "but this worked yesterday."

File backing and a userfaultfd socket aren't interchangeable transports for the same thing; they're different contracts. With the file backend the VMM maps the image and the kernel pages it in behind your back. With UFFD the VMM registers the regions with userfaultfd, hands your handler the descriptor and region layout, and every first touch becomes an event your process must answer. That's what makes on-demand streaming from object storage possible — and it means a bug in your handler shows up as a hung guest rather than a failed load.

• Who supplies pages — File: the kernel, from an mmap of the image. UFFD: your handler process, one fault at a time, over a UNIX socket.
• Hugepage-backed snapshots — File: rejected outright. UFFD: the only supported path.
• Whole-image availability — File: the entire image must exist locally before load. UFFD: only the pages actually touched, so cross-host restore doesn't wait on a download.
• Failure timing — File: usually at load, or at first touch of the bad region. UFFD: whenever the guest happens to fault that page, which can be minutes in.
• Failure shape — File: an error in the log. UFFD: a wedged guest with no error anywhere, because from the VMM's side it's still waiting.
• Operational cost — File: nothing to run. UFFD: a second process per VM that must outlive the load and never die while the guest is alive.

Class 3: the memory image is missing, truncated, or the wrong size

What it looks like: a load that fails on a size or mapping error, or — much more fun — a restore that succeeds, runs a while, then dies or hangs the first time the guest touches a page that isn't really there. Late failure is the signature of this class, which is why it gets misdiagnosed as a guest kernel problem.

The image must match the guest memory size recorded in the state file exactly. Not approximately, not "at least." The classic cause is a partially-downloaded `vm.mem` from object storage: a reset at 94%, a retry that wrote to the same path, a sync tool that decided a same-name file was already fresh. With the file backend you often get an honest failure at load. With UFFD you get the worst version: the load succeeds because nobody read the whole image, and three minutes later the guest faults a page in the missing tail.

Verify size and checksum before the load, not after the incident. Use `stat`, not `du` — memory images are sparse, so `du` reports allocated blocks and will happily tell you a perfectly good 4 GiB image is 900 MB.

#!/usr/bin/env bash
# preflight.sh -- run before every restore, not after every incident.
set -euo pipefail

SNAP="${1:?usage: preflight.sh /snap/gen-42}"
# Recorded at bake time from the machine-config you used. The state file is
# authoritative; this is the copy you can compare against cheaply.
MEM_SIZE_MIB=$(jq -r .mem_size_mib "$SNAP/meta.json")

expected=$(( MEM_SIZE_MIB * 1024 * 1024 ))
# stat, not du: the image is sparse and du reports ALLOCATED blocks.
actual=$(stat -c %s "$SNAP/vm.mem")

if [ "$actual" -ne "$expected" ]; then
  echo "FATAL: vm.mem is $actual bytes, state expects $expected" >&2
  echo "       short by $(( expected - actual )) -- almost certainly a truncated download" >&2
  exit 1
fi

# Checksums catch the other half: right length, wrong bytes.
( cd "$SNAP" && sha256sum -c manifest.sha256 --quiet ) || {
  echo "FATAL: checksum mismatch in $SNAP -- re-pull the generation" >&2
  exit 1
}

# Hugepage snapshots restore ONLY through UFFD. Refuse early and clearly.
if [ -f "$SNAP/hugepages" ]; then
  echo "note: generation demands the Uffd backend (hugepage-backed snapshot)"
fi

echo "ok: $SNAP verified (${MEM_SIZE_MIB} MiB guest memory)"

Classes 4 and 5: device topology drift and network identity

These two are cousins: the host you're restoring onto no longer matches the hardware the guest was frozen believing in. One fails the load. The other doesn't fail anything at all.

Paths may move; IDs may not

The state file encodes a specific device model: how many block devices, how many NICs, whether there's a vsock, and critically what each is called. Host paths are allowed to move — that's the point of restoring onto a fresh copy-on-write clone of the rootfs — but you update them by PATCHing the device by its ID after load, and the ID is the join key back to the snapshotted state.

Rename `rootfs.ext4` to `clone.ext4` on disk and you're fine. Change `drive_id` from `rootfs` to `root` between bake and restore and you have nothing to patch. Same for NICs: `iface_id` must match even though `host_dev_name` changes every restore. Count matters too — if the guest was snapshotted with a vsock, the restore needs one. Firecracker v1.16 added a UDS path override at restore precisely so each restored VM gets its own socket path without re-declaring the device.

It restored, and the network is dead

The load returns 200, the guest resumes, your health check on port 22 times out, and the serial console shows a perfectly healthy Linux with no connectivity. Nothing failed. That's the problem — there's no error to grep for.

Two things are true after a restore. The tap the VMM wants must exist, in the namespace the VMM process is actually in, with the name the interface config points at — a tap created in the root namespace while Firecracker runs inside `ns-<id>` is invisible to it. And the guest still believes the network identity it had at snapshot time: IP, route, ARP entries, and MAC are baked into the memory image. It will not DHCP and will not re-ARP, because from its point of view no time passed.

So the host conforms to the guest, not the other way round. Recreate the tap in the correct namespace, set its MAC and the host-side addressing to the values frozen at bake time, and reinstall NAT and routing for that slot. This is why we pre-allocate 16,384 /30 subnets per agent as ready-made namespace-plus-veth-plus-tap slots: at restore we patch a MAC to match what the guest already thinks. Debug from both ends, but it's almost always the namespace.

Class 6: the host said no

What it looks like: no API response at all, because the VMM died before it could give you one, or the socket never appeared. If `curl` can't connect, you don't have a snapshot problem — you have a process problem, and the answer is in the VMM's stderr or the jailer's output.

• No KVM access: `/dev/kvm` missing, or the (possibly jailed) uid can't open it. Nested-virt hosts love this one.
• Hugepages demanded but unavailable. Reserved hugepages aren't reclaimable, so a host can be simultaneously out of hugepages and mostly idle.
• Out of memory: the file backend maps lazily, but a guest that touches its working set commits it. Restoring more VMs than the host can back is a load that succeeds and an OOM kill that follows.
• Jailer chroot gaps: every path the VMM opens must exist inside the chroot, owned by the jailed uid. A restore that works unjailed and fails jailed is this, essentially every time.
• Stale API socket: the file still exists from a previous process, so binding fails. Unlink before launch.
• File descriptor limits: a host that restores 40 VMs happily and fails at 200 is worth checking here.

Class 7: it restored, it's alive, and it's still wrong

These aren't load failures, which is why they get filed as application bugs and burn a week. Same root cause: a restored guest is a guest that time-travelled, and some state is only correct relative to when it was captured.

The clock is frozen at bake time

The guest's wall-clock time was serialized with everything else, so a snapshot baked three weeks ago produces a guest that believes it is three weeks ago. The first HTTPS request then fails certificate validation because the cert isn't valid yet, package indexes are rejected as being from the future, and JWTs fail their `nbf` checks. It presents as a TLS bug and it's a clock bug. Set the guest clock explicitly on restore, resume, and wake — don't hope NTP notices, because NTP needs the network and reaching it may need TLS.

Duplicated identity and randomness

Every VM restored from one snapshot starts with the same machine-id, the same SSH host keys, the same hostname, and an entropy pool in the same state. Fork it a hundred times and you have a hundred guests agreeing on things they should disagree about — a correctness problem for anything keyed on machine identity, and a security problem for anything that generates a secret shortly after boot. Re-seed the RNG and regenerate identity post-resume, and generate long-lived keys after the fork rather than baking them in.

Stale sockets, stale DNS, and mid-write filesystems

Any TCP connection open at snapshot time is restored into a guest that still thinks it's connected while the peer forgot it exists — a socket that hangs until timeout rather than resetting cleanly. Cached DNS answers, including negative ones, come back with the guest and can be arbitrarily stale. And the disk was captured mid-write, so the guest sees a crash-consistent filesystem, and crash-consistent means fsck might have opinions. Quiesce before you capture, and design the guest to reconnect rather than assume its sockets survived a nap.

Methodology: read the log, then bisect

The temptation with a failed restore is to start changing the restore config, because that's the part you own. Resist it for five minutes. The goal is to answer one binary question — is the snapshot bad, or is my config bad — and everything else is downstream of that answer.

1. Confirm the process is alive and the API socket answers at all. If it doesn't, this is Class 6 and the snapshot is irrelevant.
2. Read the Firecracker log you configured via PUT /logger — configure the logger BEFORE the load, or you'll be blind on the one request that mattered.
3. Read the serial console. A load that succeeds and a guest that dies afterwards leaves its evidence there and nowhere else.
4. Bisect: restore the same artifacts, unjailed, on a host you control, with the simplest possible config — file backend, no NIC, no vsock, resume_vm false.
5. If the minimal restore works, the snapshot is fine and your config is the bug. Add devices back one at a time; the last thing you added is the answer.
6. If it fails, it's the snapshot or the binary. Check the VMM version against the generation's recorded version, then verify the image's size and checksum.
7. Only then change production config, with a hypothesis instead of a hunch.

# bisect.sh -- the control experiment. Simplest possible restore, no devices.
set -euo pipefail
SNAP=/snap/gen-42
WORK=$(mktemp -d)
SOCK="$WORK/fc.sock"

# Work on COPIES. You do not want to debug a snapshot you have already damaged.
cp --reflink=auto "$SNAP/vm.state" "$SNAP/vm.mem" "$WORK/"
cp --reflink=auto "$SNAP/rootfs.ext4" "$WORK/clone.ext4"

firecracker --api-sock "$SOCK" --level Debug --log-path "$WORK/fc.log" &
FC_PID=$!
until [ -S "$SOCK" ]; do kill -0 "$FC_PID"; done   # dies here => Class 6

curl -sS --unix-socket "$SOCK" -X PUT 'http://localhost/snapshot/load' \
  -H 'Content-Type: application/json' \
  -d "{\"snapshot_path\":\"$WORK/vm.state\",
       \"mem_backend\":{\"backend_type\":\"File\",\"backend_path\":\"$WORK/vm.mem\"},
       \"resume_vm\":false}"

# Paths may move; IDs may not. Patch by the drive_id recorded in the snapshot.
curl -sS --unix-socket "$SOCK" -X PATCH 'http://localhost/drives/rootfs' \
  -H 'Content-Type: application/json' \
  -d "{\"drive_id\":\"rootfs\",\"path_on_host\":\"$WORK/clone.ext4\"}"

# Same rule for the NIC: iface_id must match, host_dev_name is expected to change.
# curl ... -X PATCH 'http://localhost/network-interfaces/eth0' \
#   -d '{"iface_id":"eth0","host_dev_name":"tap0"}'

curl -sS --unix-socket "$SOCK" -X PUT 'http://localhost/vm' \
  -H 'Content-Type: application/json' -d '{"state":"Resumed"}'

tail -f "$WORK/fc.log"

The checklist: before every restore, and after

Every item here exists because it once cost someone a night. Encode them as code in your restore path, not as a wiki page nobody opens.

1. The VMM binary version matches the version recorded for this snapshot generation. Refuse loudly on mismatch instead of trying.
2. The memory backend matches the snapshot's declared requirement. A hugepage marker means UFFD — no exceptions, no flag that overrides it.
3. vm.mem exists and its byte length exactly equals the recorded guest memory size, measured with stat, not du.
4. State file and memory image checksums match the generation manifest. A checksum run after the incident is a post-mortem, not a check.
5. Every drive_id and iface_id matches what was snapshotted, and every path_on_host exists and is readable by the uid the VMM runs as.
6. If jailed: all of those paths exist inside the chroot with the right ownership.
7. The tap exists in the namespace the VMM will run in, is up, and carries the baked MAC; NAT and routes for that slot are installed.
8. /dev/kvm is openable, the socket path is free, and the host has the memory (and hugepages, if demanded) the guest will touch.
9. Load with resume_vm false, health-check, then resume — so a broken restore is caught before it takes traffic.
10. Post-resume hooks are queued and idempotent: set the clock, re-seed randomness, regenerate machine identity, reconnect long-lived sockets.

A 200 from the load endpoint means the VMM accepted your state file. It does not mean the guest is usable, and Classes 5 and 7 live entirely in that gap. So the last thing in a restore path should be a health gate that runs inside the guest and checks exactly what restores break: clock skew, a working route, DNS, a real TLS handshake, and whether this guest's identity is distinct from its siblings.

from pandastack import Sandbox

# A restore that "succeeded" is not the same as a guest that works. These five
# checks catch the restore-specific failure modes. Run them before the sandbox
# is handed to a caller, not after a support ticket.
CHECKS = {
    "clock":      "date -u +%s",
    "route":      "ip route get 1.1.1.1",
    "dns":        "getent hosts pypi.org",
    "tls":        "curl -sS -o /dev/null -w '%{http_code}' https://pypi.org/simple/",
    "machine_id": "cat /etc/machine-id",
}


def gate(sbx, host_epoch: int) -> list[str]:
    problems = []
    for name, cmd in CHECKS.items():
        r = sbx.exec(cmd, timeout_seconds=15)
        out = r.stdout.strip()
        if r.exit_code != 0:
            problems.append(f"{name}: exit={r.exit_code} {r.stderr.strip()[:80]}")
            continue
        if name == "clock":
            # A frozen clock is the #1 cause of "TLS broke after a restore".
            skew = abs(int(out) - host_epoch)
            if skew > 30:
                problems.append(f"clock: guest is {skew}s off -- sync before resume")
        print(f"  {name:<11} {out[:60]}")
    return problems


if __name__ == "__main__":
    import time

    with Sandbox.create(template="base", ttl_seconds=300) as sbx:
        issues = gate(sbx, host_epoch=int(time.time()))
        print("HEALTHY" if not issues else "DEGRADED:\n  " + "\n  ".join(issues))

Two habits do most of the work. Verify the artifacts before you load them, because a truncated memory image is cheap to detect and expensive to debug. And gate on the guest afterwards, because the interesting restore failures don't return an error code — they return a healthy-looking VM whose clock says it's three weeks ago. A snapshot is a promise about the world, and your job at restore time is to make the world keep it.