What's Actually Inside a Firecracker Snapshot

Ask what a Firecracker snapshot is and you'll usually get "a saved VM." Look on disk and you get something far more concrete: two files with wildly different personalities. One is small, highly structured, version-coupled, and read in full on every restore. The other is enormous, completely unstructured, mostly zeros, and — done right — barely read at all. Which is which explains essentially every design decision in this space, including the ones that look like black magic from outside.

I'm Ajay — I build PandaStack, an open-source Firecracker microVM platform where there is no warm pool: every sandbox, managed Postgres database, and hosted app is a snapshot restore. That means I spend an unreasonable fraction of my life staring at these two files. This post is the anatomy lesson: what's in the state file, what's in the memory file, why `ls -l` lies to you about the second one, what restore actually does with each, and — the part that generates the most production incidents — the four things a snapshot conspicuously does not contain.

Two files: one tiny, one enormous

`PUT /snapshot/create` writes exactly two artifacts. The state file (`vm.state` by convention) is a serialized `MicrovmState` — the VMM's structured view of the machine. The memory file (`vm.mem`) is a flat dump of guest physical RAM. That's it. There is no archive, no manifest, no self-describing container; you get two paths because you passed two paths, and keeping them together is entirely your problem.

• Size — State file: kilobytes to low megabytes, and roughly constant regardless of how much RAM the guest has. Memory file: exactly the guest's configured RAM, logically. A 4 GiB guest produces a 4 GiB memory file.
• Contents — State file: structured VMM state, i.e. vCPU register sets, MSRs, the captured CPUID, per-device virtio state, interrupt-controller state, and memory-region descriptors. Memory file: no structure at all — offset N in the file is guest physical byte N of the corresponding region.
• Format — State file: a versioned binary serialization with a header, and an implementation detail you should not hand-parse. Memory file: not a format, just bytes.
• Sparseness — State file: dense; every byte means something. Memory file: typically very sparse, because most of a guest's RAM was never written and reads back as zeros or as filesystem holes.
• Restore cost — State file: always read in full, and it's small enough that nobody cares. Memory file: ideally not read up front at all — it's mapped copy-on-write, or served fault-by-fault by a userspace handler.
• Portability — State file: tightly version-coupled to the Firecracker build, the CPU features it captured, and the guest kernel that was running. Memory file: portable bytes, and completely meaningless without its exact state-file partner.
• Where the engineering goes — State file: compatibility checks and refusing bad restores. Memory file: literally everything else — sparseness, chunking, caching, prefetch, streaming, dedup, encryption.

That last row is the thesis. The state file is a solved problem: it's small, you read it, it validates or it doesn't. The memory file is the only artifact whose size scales with the guest, which makes it the only one that can make restore slow, storage expensive, transfer painful, and secrets leak. Every optimization in this space — lazy paging, copy-on-write sharing, zero elision, chunk caches, prefetch traces, streaming — is a trick played on the memory file. Nobody optimizes the state file, because there's nothing there to win.

# Take a snapshot. Pause first — you can't serialize a moving target.
curl --unix-socket /run/fc.sock -X PATCH 'http://localhost/vm' \
  -d '{"state": "Paused"}'

# Two paths in, two files out. (Verify exact field names against the
# Firecracker API docs for the version you actually run.)
curl --unix-socket /run/fc.sock -X PUT 'http://localhost/snapshot/create' \
  -d '{
        "snapshot_type": "Full",
        "snapshot_path": "/snap/vm.state",
        "mem_file_path": "/snap/vm.mem"
      }'

# Now look at what you got. `ls -l` shows LOGICAL size for both.
ls -l /snap
#   vm.state   <-- kilobytes. The whole MicrovmState.
#   vm.mem     <-- exactly the guest's RAM size. Or so ls claims.

# `du --apparent-size` agrees with ls: it reports the logical length.
# Plain `du` reports blocks ACTUALLY ALLOCATED on the filesystem.
du --apparent-size -h /snap/vm.mem   # logical: full guest RAM
du -h /snap/vm.mem                   # allocated: only non-hole blocks

# Same file. Two very different numbers. Which one is 'the size' of your
# snapshot depends entirely on what you're about to do with it.

Inside the state file: a serialized MicrovmState

The state file is everything needed to reconstitute the machine around the RAM. Conceptually it's one big struct, serialized:

• vCPU state, one set per vCPU — the general-purpose and special registers, the program counter sitting between two instructions, segment/system registers, the FPU and vector register state, and on x86 the model-specific registers (MSRs) that carry things like the TSC.
• The captured CPUID — the exact feature bits the guest was told it had. This is the single most operationally dangerous field in the file, because the guest's software cached those bits and will keep emitting instructions that assume them.
• Device model state, per virtio device — for each block, net, vsock, balloon and MMIO device: its configuration, its virtqueue descriptor/available/used ring pointers, and its interrupt status. This is why in-flight I/O survives a snapshot instead of turning into corruption.
• Interrupt controller state — the in-kernel controller's registers and pending interrupts (the KVM IRQ chip on x86, the GIC on aarch64), plus the clock/timer state.
• Memory region descriptors — the map from guest physical address ranges to offsets in the memory file. Without these the memory file is an anonymous pile of bytes.

Serialization, and why cross-version restore is refused rather than attempted

That struct is written as a versioned binary serialization with a header — not JSON, not protobuf, and deliberately not a public interchange format. The snapshot data format carries its own version number, and each Firecracker release declares which format version it produces and which ones it can consume. Verify the specifics against the upstream snapshot-support and versioning docs for the exact version you run; the point here is the shape of the design, not a matrix you should memorize.

The consequence is worth stating plainly: because the layout is an internal implementation detail rather than a stable contract, a mismatched restore is refused at load time rather than attempted optimistically. That's the correct choice, and also the one people complain about. A snapshot is a serialized picture of another program's internal memory layout — let a binary that disagrees about that layout try its luck and you don't get a helpful error, you get a guest whose interrupt-controller state is being read as register state. Loud refusal is a feature. (Full compatibility story at /blog/firecracker-snapshot-version-compatibility.)

Firecracker ships a `snapshot-editor` utility for the cases where you legitimately need to look inside or modify these artifacts. Qualitatively it does two families of things: inspect the vmstate — read out the snapshot's format version, dump vCPU/VM state — and edit the memory file, most notably rebasing a diff snapshot's memory onto its base to produce a standalone full memory file. There's also aarch64 register surgery for stripping registers a target host won't accept. Treat it as the supported way to poke at a snapshot, and check its actual subcommands against the upstream docs for your release rather than trusting a blog post's flag spelling — mine included.

The memory file is sparse, and `ls -l` is lying to you

Here's where people lose an afternoon. A guest with 4 GiB of RAM produces a memory file that is 4 GiB long, but most of a freshly-booted guest's RAM was never written — zeros the kernel allocated and never touched, freed pages, page-cache slack. On a filesystem that supports sparse files those zero ranges become holes: extents that exist in the file's logical length with no blocks allocated behind them. The file is 4 GiB long and occupies dramatically less than that, and both numbers are correct answers to different questions. `ls -l`, `stat`, and `du --apparent-size` report the logical length; plain `du` reports allocated blocks. If you've ever had a snapshot bucket bill that didn't match your local disk usage, this is why: your disk saw the holes and your object store did not.

# Every one of these will happily inflate holes into real, billable zeros
# unless you tell it not to:
cp /snap/vm.mem /backup/vm.mem              # may materialize every hole
tar -cf snap.tar /snap/vm.mem              # ditto
cat /snap/vm.mem | gzip > vm.mem.gz        # reads all 4 GiB of mostly-zero

# Sparse-aware alternatives:
cp --sparse=always /snap/vm.mem /backup/vm.mem
rsync -S /snap/vm.mem  backup:/snap/       # -S == --sparse
tar -S -cf snap.tar /snap/vm.mem           # -S == --sparse

# Where the holes actually are, straight from the kernel:
#   lseek(fd, off, SEEK_DATA) -> next byte offset with data
#   lseek(fd, off, SEEK_HOLE) -> next byte offset that is a hole
# Useful locally. Useless the moment the file leaves the filesystem.

# Object storage has NO concept of a hole. A naive upload streams the
# LOGICAL bytes, so you pay full guest-RAM egress and full guest-RAM
# storage to persist a file that is mostly nothing. The fix is not a
# smarter uploader; it's to stop treating it as one opaque blob.

What restore does with that file: mmap vs. a fault handler

There are two ways to hand a memory file to a restoring Firecracker, and the choice determines what's possible for the rest of your architecture.

# --- Backend A: File. The VMM maps the memory file itself. ---
curl --unix-socket /run/fc2.sock -X PUT 'http://localhost/snapshot/load' \
  -d '{
        "snapshot_path": "/snap/vm.state",
        "mem_backend": {
          "backend_type": "File",
          "backend_path": "/snap/vm.mem"
        },
        "resume_vm": false
      }'

# --- Backend B: Uffd. A userspace handler serves the page faults. ---
# backend_path here is NOT the memory file — it's a Unix socket your
# handler is already listening on. Firecracker connects, sends the
# userfaultfd descriptor over SCM_RIGHTS plus the guest memory region
# layout, and from then on every un-populated guest page fault becomes
# a message your process must answer.
curl --unix-socket /run/fc2.sock -X PUT 'http://localhost/snapshot/load' \
  -d '{
        "snapshot_path": "/snap/vm.state",
        "mem_backend": {
          "backend_type": "Uffd",
          "backend_path": "/run/uffd.sock"
        },
        "resume_vm": false
      }'

# Either way, the guest is loaded but frozen. Unpause to resume it
# mid-instruction, exactly where it was when you paused the original.
curl --unix-socket /run/fc2.sock -X PATCH 'http://localhost/vm' \
  -d '{"state": "Resumed"}'

# (Field names/shape: verify against the upstream API docs for your version.)

The `File` backend maps the memory file `MAP_PRIVATE`. Nothing is copied at load time; the mapping just declares that guest RAM is backed by this file. A read fault is resolved by the kernel mapping the page straight from the page cache, no copy. A write fault triggers copy-on-write: a private copy of that one page for this guest, original untouched. Two enormously useful properties fall out. First, restore cost tracks the guest's working set, not the file size — which is why the load step on PandaStack sits around 49ms inside a ~179ms p50 create, and doesn't grow when the template gets more RAM. Second, N restores of the *same* memory file share every clean page in the page cache, so the tenth concurrent clone of a template costs the host almost nothing in RAM beyond what it dirties. (The mmap mechanics get their own post at /blog/firecracker-memory-file-mmap-explained.)

The `Uffd` backend replaces the kernel's "read it from the file" answer with "ask this process." That indirection is the whole ballgame, because the process is under no obligation to have a local file at all. It can fetch the bytes from anywhere, and it can answer a fault in a zero region by installing zeros with no I/O whatsoever. This is what turns a memory file from something you must possess into something you can stream — a host can begin restoring a template it has never held locally, pulling only the pages the guest actually touches, instead of downloading multiple gigabytes of mostly-zero before the guest is allowed to run. It's also the mechanism behind 2 MiB hugepage-backed guests, since a hugepage snapshot can only be restored through a fault handler that serves whole huge pages. Mechanics in detail at /blog/userfaultfd-explained.

The layers people build on top: a non-zero index and a prefetch trace

Once a userspace handler owns the faults, the memory file stops being a file and becomes a *content-addressed range service*, and two general optimizations become obvious. Both are things PandaStack does; neither is exotic, and I'd expect any serious implementation to converge on the same two.

The first is a sidecar index recording which chunks of the memory file contain non-zero bytes. You compute it once at bake time and ship it next to the snapshot. At restore, a fault landing in a zero chunk is served with a local zero-fill and never touches the network; only chunks with real content are fetched. It also lets your uploader skip all-zero chunks entirely, which is how you stop paying object-storage prices to durably preserve nothing. Chunk at a size that amortizes a request across many pages — a 4 MiB chunk covers 1,024 four-KiB pages, so one round trip answers a thousand potential faults.

// Bake time: one pass over the memory file yields a sidecar bitmap of
// "which chunks are worth fetching". At 4 MiB chunks that's ~32 KiB of
// bitmap per TiB of address space — cheap enough to be unconditional.
const chunkSize = 4 << 20

func buildNonZeroIndex(memPath string) ([]byte, error) {
	f, err := os.Open(memPath)
	if err != nil {
		return nil, err
	}
	defer f.Close()

	fi, err := f.Stat()
	if err != nil {
		return nil, err
	}
	// fi.Size() is the LOGICAL size, holes included — which is what we want:
	// chunk indices have to address the guest's full RAM.
	nChunks := (fi.Size() + chunkSize - 1) / chunkSize

	bitmap := make([]byte, (nChunks+7)/8) // 1 = has data, 0 = all zeros
	buf := make([]byte, chunkSize)
	zero := make([]byte, chunkSize)

	for i := int64(0); i < nChunks; i++ {
		n, err := f.ReadAt(buf, i*chunkSize)
		if err != nil && err != io.EOF {
			return nil, err
		}
		if !bytes.Equal(buf[:n], zero[:n]) {
			bitmap[i/8] |= 1 << uint(i%8)
		}
	}
	return bitmap, nil
}

// Restore time, inside the UFFD handler:
//
//	offset := fileOffsetFor(gpa)      // honour the region descriptors!
//	idx    := offset / chunkSize
//	if bitmap[idx/8]&(1<<uint(idx%8)) == 0 {
//	    installZeroPage(faultAddr)    // UFFDIO_ZEROPAGE — no bytes, no network
//	    return
//	}
//	chunk := cache.GetOrRangeGet(idx) // HTTP Range GET on a cache miss
//	installCopy(faultAddr, chunk)     // UFFDIO_COPY wakes the parked vCPU

The second layer is a prefetch trace. The path from resume to "ready to do work" is remarkably deterministic — the same frozen machine replaying the same wake-up touches close to the same pages every time. So you record that hot chunk set once at bake time, ship it beside the index, and replay it in the background the instant a restore starts. The streamer races ahead of the guest, so by the time a vCPU faults, the chunk is frequently already local — a memory read instead of a network round trip. Add a shared per-host chunk cache keyed by the snapshot's generation and the first restore on a cold host pays the network cost once on behalf of every restore after it.

What a snapshot does NOT contain (this is the part that pages you)

Everything above is what's in the files. The production incidents come from what isn't. A snapshot captures a machine's *memory and CPU*, and four things that feel like they belong to a machine turn out to live outside that boundary.

No block-device data

The state file holds the block device's *state* — virtqueue pointers, configuration, in-flight requests. It holds zero bytes of the disk itself. The rootfs is a separate file you must keep in lockstep, and the coupling is absolute: the guest's page cache, sitting in the memory file, holds inodes and data blocks read from that specific disk image at that specific instant. Restore against a rootfs that has moved on — a rebuilt image, a newer template, a disk someone else already wrote to — and you don't get an error. You get a running guest whose in-memory view of the filesystem disagrees with the blocks underneath it, which is a long word for corruption. Version the pair, publish the pair, garbage-collect the pair. A snapshot is three files pretending to be two.

No host clock — the guest wakes up in the past

The captured state includes the guest's clock as of bake time. It does not include any notion of "what time is it now." So a guest restored from a template baked six weeks ago genuinely believes it is six weeks ago, confidently and without complaint. `date` is wrong, timers are wrong, and — the one that actually bites — TLS breaks. Not immediately, which is the cruel part: everything works fine right up until an upstream service rotates its certificate to one whose `notBefore` is *after* your bake date, at which point every HTTPS call from every restored guest starts failing certificate validation simultaneously, on a template nobody has touched in weeks. We hit exactly this. The fix is that something has to re-sync the clock on every restore, resume, and wake, before the guest gets a chance to make an outbound connection.

No fresh entropy — every clone shares an RNG state

A snapshot captures the kernel's random pool exactly as it was. Restore it a thousand times and you have a thousand guests holding byte-identical entropy state, which means they can produce identical "random" values: same UUIDs, same session tokens, same nonces, same ephemeral keys. This is a security problem dressed as a curiosity. The mitigation is to re-seed on restore — from a virtio-rng device, from injected host entropy, from something that differs per clone — and to be suspicious of any userspace RNG that grabbed a seed before the snapshot and cached it. I dug into this failure mode at /blog/snapshot-clone-randomness-problem-explained.

No network identity — MAC and IP are frozen

The guest's view of its network — its interface's MAC, its configured IP, its default route, its ARP cache — is memory, so it's frozen at bake time and comes back identical on every restore. The host side is not in the snapshot at all. Restore a hundred clones and you have a hundred guests all convinced they are the same host at the same address. There are two ways out: give every guest a genuinely unique identity on restore, or make the frozen identity true by construction. PandaStack does the second — each agent pre-allocates 16,384 /30 subnets, each sandbox lands in its own network namespace with its own tap device, and the agent patches the tap's MAC and routes to match what the guest already believes. The guest is never told it moved, because inside its own namespace, it didn't.

Security: it's a core dump you deliberately keep forever

A memory file is a complete, unencrypted, byte-for-byte copy of everything that was in the guest's RAM. Not a summary, not a redacted export. Every environment variable, every decrypted secret, every private key a process had loaded, every database password in a connection pool, every token in an HTTP client, every plaintext buffer not yet overwritten. We have a name for a file like this when it's produced by accident — a core dump — and we treat those as sensitive. This one is produced on purpose, kept indefinitely, replicated to object storage, and shared across a fleet.

A memory snapshot is a core dump you deliberately keep. And yes, `grep -a` works on it exactly as well as you're now afraid it does.

That's not a hypothetical — searching a memory file for a known string is a genuinely useful debugging technique, which is precisely why it's a genuinely useful attack. The operational posture follows directly:

• Never bake a secret into a template snapshot. Anything resident in RAM at bake time is in the artifact permanently, copied to every host that restores it, inherited by every clone. Inject secrets after restore, per sandbox, never before capture.
• Encrypt at rest and in transit, and scope bucket access hard. Snapshot buckets deserve the same paranoia as database backups, for the same reason: they are your database backups, plus the RAM that was reading them.
• Scope per tenant. A memory file from one customer's workload must never be reachable by a restore path serving another. Treat cross-tenant snapshot access as a breach, not a bug.
• Set retention. A snapshot doesn't age out of sensitivity — a key leaked in a nine-month-old artifact is just as valid as one leaked today, and probably hasn't been rotated.
• Watch the derived artifacts. Chunk caches, prefetch traces, and diff snapshots contain real guest memory bytes and need the same encryption, scoping, and retention as the file they came from.

There's a longer treatment of the threat model at /blog/firecracker-snapshot-secrets-security. The compressed version: if you would not email the file, do not leave it in a world-readable bucket.

Seeing all of this from the outside

The nice thing about this being two concrete files is that every claim in this post is observable from inside a sandbox in about thirty seconds. A PandaStack create *is* a restore of a baked snapshot pair — around 179ms p50 end to end, versus roughly 3s for the cold boot that produces the snapshot in the first place — so anything that's true about restore is true about every sandbox you launch:

from pandastack import Sandbox

# This create is a snapshot restore: state file loaded, memory file mapped
# (or streamed), vCPUs unpaused. ~179ms p50. No boot, no init, no systemd.
box = Sandbox.create(template="base", ttl_seconds=3600)

# The guest resumed mid-life, not from power-on. Uptime continues from the
# moment of the bake — it did not start counting when you called create().
print(box.exec("cat /proc/uptime").stdout)

# Everything the guest believes about its network was frozen at bake time.
# It looks identical in every clone, and it's correct in every clone,
# because each one lives in its own /30 in its own network namespace.
print(box.exec("ip -4 addr show; ip route").stdout)

# The clock is the one belief that MUST be corrected on restore, or TLS
# starts failing the day an upstream cert rotates past the bake date.
print(box.exec("date -u; curl -sSI https://example.com | head -1").stdout)

# Fork it: snapshot this live machine, restore the pair copy-on-write.
# 400-750ms same-host (memory is already resident, rootfs reflinks locally),
# 1.2-3.5s cross-host (the artifacts have to move first).
child = box.fork()
print(child.id, "shares clean pages with", box.id, "until it writes them")

The bottom line

A Firecracker snapshot is a small structured state file and a large unstructured memory file. The state file is a serialized `MicrovmState` — vCPU registers and MSRs, the captured CPUID, per-device virtio state, the interrupt controller, and the memory-region descriptors that make the second file interpretable — written in a versioned binary format that's an internal implementation detail, which is why an incompatible restore is refused rather than gambled on, and why `snapshot-editor` exists. The memory file is a flat dump of guest RAM: sparse on disk, full-size the moment it leaves the filesystem, mapped `MAP_PRIVATE` for lazy copy-on-write restore or served fault-by-fault by a UFFD handler that can stream it from object storage — with a non-zero chunk index so you never fetch zeros and a prefetch trace so the hot set arrives before the guest asks for it.

Then there's the negative space, which is where the pager goes off: no disk data (version the rootfs with the snapshot or ship yourself corruption), no host clock (the guest wakes in the past and TLS dies the day a cert rotates past your bake date), no fresh entropy (every clone shares an RNG state until something re-seeds it), no network identity (frozen MAC and IP you have to make true on restore). Plus the thing that's always true and easy to forget: the memory file is a complete copy of guest RAM, secrets included, and `grep -a` does not care how you feel about that. PandaStack's core is Apache-2.0 — run the agent on your own KVM host and go look at the two files yourself.