Should You Compress Firecracker Memory Snapshots?

A Firecracker memory snapshot is not a clever encoding of your guest's state. It is your guest's state. When you POST to /snapshot/create, the VMM writes out guest physical memory byte for byte, plus a much smaller vm.state file holding vCPU registers and device state. A guest configured with 4 GiB of RAM produces a 4 GiB vm.mem. There is no compaction pass, no "only the used parts" step. The file is the RAM.

That's fine for one snapshot. It stops being fine when you multiply: every template, every generation of every template, every region you replicate to, plus the user snapshots and fork parents your customers create because you told them forking was cheap. Eventually someone opens the storage bill, notices the largest line item is memory nobody has read since March, and asks the obvious question: why aren't we just gzipping these?

Good question, annoying answer. For archival, yes, absolutely. For the restore path, almost never — and the reason isn't ratios or CPU cost, it's that compression takes away the one property fast restore is built on.

What you're actually storing

Split a snapshot into its two artifacts and the asymmetry is immediate. vm.state is small and structured: CPU state, virtio queue positions, interrupt controller state — the bookkeeping needed to reconstitute the machine. Nobody has ever had a storage problem because of vm.state. vm.mem is a flat image of guest physical memory, and it is the entire cost.

# Pause the guest, then ask Firecracker to write the snapshot out.
curl --unix-socket /tmp/fc.sock -X PATCH 'http://localhost/vm' \
  -H 'Content-Type: application/json' \
  -d '{"state": "Paused"}'

curl --unix-socket /tmp/fc.sock -X PUT 'http://localhost/snapshot/create' \
  -H 'Content-Type: application/json' \
  -d '{
        "snapshot_type": "Full",
        "snapshot_path": "/snap/vm.state",
        "mem_file_path": "/snap/vm.mem"
      }'

# Two artifacts, wildly different sizes. vm.state is registers and device
# state. vm.mem is the guest's RAM, 1:1, and it is the entire storage bill.
ls -l /snap/

# The restore side is what constrains the format: mem_backend can be a FILE
# (mmapped, faulted in lazily) or a UFFD socket (faults served on demand).
# Both need to address memory by OFFSET. Neither can read a compressed stream.
curl --unix-socket /tmp/fc2.sock -X PUT 'http://localhost/snapshot/load' \
  -H 'Content-Type: application/json' \
  -d '{
        "snapshot_path": "/snap/vm.state",
        "mem_backend": {"backend_type": "Uffd", "backend_path": "/snap/uffd.sock"},
        "resume_vm": true
      }'

The flatness is the feature. Because vm.mem is a linear image, guest physical address X lives at file offset X, modulo the regions carved out around MMIO holes. That mapping is what lets a restore be lazy: Firecracker can mmap the file and let the guest fault pages in as it touches them, or hand fault handling to a userfaultfd process that fetches only the pages the guest asks for. The address-to-offset arithmetic is a subtraction. Everything below is a consequence of that one sentence — a snapshot format is fast to restore exactly to the degree that it supports random access by guest address.

Why compression is so tempting

Guest memory is boring in the specific way general-purpose compressors love. A freshly booted guest's RAM is dominated by pages that were never written and are therefore all zeros. Much of what remains is page cache — literal copies of files that came off the rootfs, so the snapshot contains a second copy of your libc and your Python standard library. And kernel data structures are extremely regular: page tables, slab objects, and struct arrays repeat the same field layout thousands of times.

Zeros collapse to nothing. Repeated structure is exactly what an LZ match window is for. Compared to random data, where you're lucky to break even, a memory image is soft — which is why the first experiment always looks so good. I won't quote you a ratio, because the honest one is "it depends entirely on your template and how much of its RAM the boot actually dirtied." Run it on your own images. The mechanism is real; that part of the intuition is correct. The part that's wrong is assuming it helps the thing you care about.

The fatal problem: compression destroys random access

Here's the thing a fast restore does not do: read the memory file. On PandaStack every sandbox create is a snapshot restore, and the restore step lands around 49ms inside a ~179ms p50 create. That is not the speed of reading 4 GiB from anywhere. It's the speed of setting up mappings and resuming vCPUs, then letting the guest fault in the handful of pages it needs to reach a prompt. The unread majority of the file is never touched. You pay, in latency, only for what the guest looks at.

Now compress that file with gzip or plain zstd. A compressed stream is a chain: symbols are coded against a sliding window of everything before them, so there is no "byte N" without decoding bytes 0 through N-1. The file no longer has offsets. It has a beginning.

So the restore path collapses into a different algorithm. You can't mmap it, because a page fault on a compressed stream has no meaningful answer. You can't serve faults from it on demand, because each fault would mean decompressing from the start. What's left is: decompress the whole thing to scratch, then restore from that. You traded a lazy setup for a linear read plus a full decompress of gigabytes you were never going to read.

Seekable frames: the partial escape hatch

There's a middle path worth knowing about even if you don't take it. Instead of one long stream, compress the image in independent, block-aligned frames — say every 4 MiB of guest memory becomes its own self-contained frame — and write a small index mapping frame number to byte range. Zstandard's seekable format is the well-known implementation; check its current status and API against upstream's own documentation, since it lives outside the core format spec.

Random access is back, at a coarser granularity: a fault on guest page P becomes find-the-frame, read its byte range, decompress it, hand back the page. That composes fine with streaming over a network, and when bandwidth is your bottleneck it can be a genuine win.

The costs are equally concrete. You decompress a whole frame to serve one page, so read amplification climbs unless prefetching is good. That decompression now sits on the critical path of a fault, with a vCPU blocked, turning a latency problem into CPU contention on a host that's probably already oversubscribed. Independent frames also compress worse than one stream, because each starts with a cold window. Measure it on the fault path before believing it.

What actually wins on the restore path

The good news is that compression is competing for a job better tools already do. Everything below shrinks storage, network, or both, without putting a decompressor between the guest and its memory.

Sparse files and zero elision

Start here, because people skip it and it's usually the biggest single win. Most of a boot-fresh guest's RAM is zeros, and a zero page doesn't need to be stored, transferred, or decompressed — it needs to be produced. The kernel hands you one for free; that's what it does for every anonymous mapping in every process on the machine.

On local disk you get part of this automatically, since a filesystem with sparse-file support never allocates blocks for regions that were never written. So before any compression project, look at what the file already costs you versus what it claims to be.

# Three different questions about the same snapshot. Run these on your own
# vm.mem before you buy anyone's compression story.

# 1) What does the file CLAIM to be? (== the guest's configured RAM)
ls -l /var/lib/pandastack/seeds/base/current/vm.mem

# 2) Same claim, human-readable. --apparent-size ignores sparseness.
du --apparent-size -h /var/lib/pandastack/seeds/base/current/vm.mem

# 3) What does it actually OCCUPY? du without --apparent-size counts
#    allocated blocks, so holes cost nothing. On a freshly baked template
#    this is usually a small fraction of (1) -- that gap is zero pages the
#    filesystem never allocated, i.e. savings you already have.
du -h /var/lib/pandastack/seeds/base/current/vm.mem

# Confirm the holes are real instead of trusting du:
filefrag -v /var/lib/pandastack/seeds/base/current/vm.mem | head -20

# 4) Now the tempting experiment. Compare the result against (3), NOT (1) --
#    comparing to the apparent size double-counts zeros that sparseness
#    already gave you for free.
zstd -3 -T0 --long=27 -f \
  -o /tmp/vm.mem.zst /var/lib/pandastack/seeds/base/current/vm.mem
ls -l /tmp/vm.mem.zst

# Whatever ratio you got, here is what it costs: /tmp/vm.mem.zst has no
# byte N. To restore from it you must first write the whole thing back out.
time zstd -d -f -o /tmp/vm.mem.restored /tmp/vm.mem.zst   # <-- the real price

Sparseness on disk is only half of it, because holes don't survive a naive upload — PUT that file to object storage with a plain read loop and you transfer every zero. So record the sparseness explicitly: walk the image in fixed chunks, note which chunks are entirely zero, and store a small header alongside the object. Zero chunks are never uploaded and never fetched; on restore the handler fills them locally. That's compression-shaped savings at a decompression cost of exactly nothing, because expanding a zero chunk is a memset the kernel was going to do anyway.

#!/usr/bin/env python3
"""Build a zero-chunk bitmap header over a Firecracker vm.mem.

Output layout (little-endian):
    magic   4s   b"PSM1"
    version u32  1
    chunk   u32  chunk size in bytes
    nchunks u64  ceil(filesize / chunk)
    bitmap  ceil(nchunks/8) bytes, bit i set == chunk i has NON-zero data

Restore reads the header first. Chunks whose bit is clear are never
fetched from object storage -- they are zero-filled locally.
"""
import os
import struct
import sys

MAGIC = b"PSM1"
CHUNK = 4 * 1024 * 1024  # 4 MiB: big enough to amortize a range GET


def build(mem_path: str, out_path: str, chunk: int = CHUNK) -> None:
    size = os.path.getsize(mem_path)
    nchunks = (size + chunk - 1) // chunk
    bitmap = bytearray((nchunks + 7) // 8)

    zero = bytes(chunk)  # compared against, not reallocated per iteration
    nonzero = 0

    with open(mem_path, "rb") as f:
        for i in range(nchunks):
            buf = f.read(chunk)
            # Short final chunk: compare against a same-length zero slice.
            if buf != (zero if len(buf) == chunk else bytes(len(buf))):
                bitmap[i >> 3] |= 1 << (i & 7)
                nonzero += 1

    with open(out_path, "wb") as out:
        out.write(struct.pack("<4sIIQ", MAGIC, 1, chunk, nchunks))
        out.write(bitmap)

    elided = (nchunks - nonzero) * chunk
    print(f"{mem_path}: {nchunks} chunks, {nonzero} non-zero")
    print(f"never uploaded, never fetched: {elided / 2**30:.2f} GiB of zeros")


if __name__ == "__main__":
    build(sys.argv[1], sys.argv[1] + ".header")

A production version uses SEEK_HOLE and SEEK_DATA so the filesystem tells you where the holes are instead of reading gigabytes to discover they're empty, and keys the header to the snapshot generation so a re-bake invalidates stale copies. But the shape is exactly this: a few kilobytes of metadata that eliminate most of a boot-fresh image.

Copy-on-write sharing: store it once, use it a thousand times

The best way to store the same 4 GiB a thousand times is to store it once. When many VMs restore from one snapshot they can map the same backing pages and diverge only where they write: reads hit shared pages, the first write triggers a copy, everything untouched stays shared. The same logic covers the rootfs via reflinks, where a clone is an O(metadata) operation and data blocks are shared until somebody dirties them. It's why same-host forking runs 400-750ms on PandaStack, and it's an axis compression can't reach — compression makes one copy smaller, sharing makes the other 999 free.

Diff snapshots: store what changed, not what exists

Firecracker can take a diff snapshot: using dirty-page tracking, it writes out only pages modified since the last snapshot. You keep a full base plus a chain of small deltas, and restore layers the deltas over the base. For long-lived VMs snapshotted repeatedly, or a template family whose variants differ from a common ancestor by a modest working set, that beats any compressor on the full image — the redundant bytes aren't compressed, they're absent.

The tradeoff is chain management. Long chains slow restores and turn one lost object into a corrupt lineage, so you periodically collapse a chain into a fresh full snapshot. Worth it exactly when snapshot frequency is high relative to the rate of change.

On-demand streaming: don't move what you don't read

The strongest move is to stop transferring the image at all. Keep vm.mem in object storage, register a userfaultfd handler for the guest's memory regions, and when the guest faults, fetch the containing chunk with an HTTP range request. The unread majority of the file costs zero bytes of network. PandaStack does this on the restore path, with the zero-chunk header above skipping empty regions and a prefetch trace replaying the known working set so faults land in cache instead of on the wire.

Notice how badly this composes with whole-file compression and how well it composes with everything else. Range requests need offsets; a gzip stream has none. Sparse headers, per-chunk caching, working-set prefetch, and dedup all speak the same chunk-addressed model, and they stack.

Deduplication across generations

Re-baking a template feels like producing a new artifact and at the byte level mostly isn't: you bumped a package, the kernel is identical, the page cache holds largely the same files. If you're already chunk-addressed for streaming, content-hashing those chunks and storing each distinct one once turns N generations into one generation plus deltas, without the chain-restore fragility of diff snapshots. Rollback becomes free, since the old generation's chunks never left.

One caution from experience: measure the dedup rate before building the machinery. Cross-template dedup in particular can be a rounding error, and a content-addressed store you didn't need is a lot of code to maintain for one.

The options, side by side

• Restore latency — Whole-file zstd: worst; decompress the entire image before the guest runs. Seekable/framed: good, minus a decompress per faulted frame on the critical path. Sparse + zero elision: best; zero chunks are filled locally. Diff snapshots: good for short chains, degrades as the chain grows. On-demand streaming: best with a prefetched working set; first-touch faults pay a round trip.
• Storage saved — Whole-file zstd: large, and the whole reason anyone proposes it. Seekable/framed: large but worse than whole-file, since each frame starts cold. Sparse + zero elision: large on boot-fresh images, near zero on a guest that dirtied all its RAM. Diff snapshots: very large when you snapshot often relative to how much changes. On-demand streaming: none by itself — it's a transfer optimization, not a storage one.
• Network saved — Whole-file zstd: large for a full transfer, useless if you only needed 200 MB of pages. Seekable/framed: large; fewer bytes per fetched chunk. Sparse + zero elision: large; zero chunks never cross the wire. Diff snapshots: large when the base is already resident on the target host. On-demand streaming: the largest, because unread pages are never transferred in any form.
• CPU on restore — Whole-file zstd: high and unavoidable, proportional to the full image. Seekable/framed: moderate, and paid with a vCPU blocked on a fault. Sparse + zero elision: negligible; a zero fill is cheaper than a read. Diff snapshots: low, mostly layering and page installs. On-demand streaming: low; the cost is network latency, not cycles.
• Complexity — Whole-file zstd: trivial, which is exactly why it keeps getting proposed. Seekable/framed: moderate; a frame index, a version, a new corruption surface. Sparse + zero elision: low; a header format and a scan. Diff snapshots: high; chain lifecycle, collapse policy, lineage integrity. On-demand streaming: highest; a fault handler, a chunk cache, prefetch traces, and a hard dependency on object storage staying up mid-restore.
• Best fit — Whole-file zstd: cold archive, backup retention, cross-region seeding of artifacts read whole. Seekable/framed: network-bound restores on hosts with idle CPU. Sparse + zero elision: every hot snapshot you own, no exceptions. Diff snapshots: frequently snapshotted long-lived VMs. On-demand streaming: fleets where the image lives in object storage and hosts are cattle.

Where compression genuinely is the right call

None of this makes compression wrong. It makes it wrong in one specific place. The rule that falls out is about read patterns: compress artifacts written once and read rarely, or read whole. Leave alone the ones read partially and unpredictably.

Cold archival is the clearest case — you keep old template generations for rollback and forensics, and the honest expected number of reads for generation N-12 is zero. Cross-region replication is the second, because the shape of the read changes: seeding a new region transfers the whole artifact rather than faulting pages out of it, and egress often dominates. Compress for the wire, decompress on arrival, and make sure what lands is the uncompressed, sparse, chunk-addressed image restores actually use. Backup retention is the third: snapshots kept for durability rather than speed, where restore is an exceptional event and the storage-months are the entire cost.

What to actually do, by which resource is hurting

"Should we compress snapshots?" is usually a proxy for a more specific pain. Name the pain and the answer stops being ambiguous.

If you're storage-bound

1. Verify sparseness first. Compare du against du --apparent-size on a representative vm.mem. If the numbers are close, something in your pipeline is materializing zeros — a linear upload path, a cp that filled the holes, a filesystem without sparse support — and fixing that is a smaller change with a bigger payoff than any compressor.
2. Add a zero-chunk header so sparseness survives the trip to object storage, not just the local disk.
3. Apply a lifecycle policy: keep the current generation and one rollback target hot and uncompressed, compress everything older, delete past retention. Most snapshot bills are a deletion problem wearing a compression costume.
4. Only then consider dedup across generations, and only after measuring the real duplicate rate on your own images.

If you're network-bound

1. Stop transferring whole images. Demand paging with range requests beats compressing a full transfer, because the fastest way to move a byte is not to move it.
2. Add a working-set prefetch so predictable pages arrive before the guest asks, turning first-touch faults into cache hits.
3. Cache fetched chunks on the host, keyed to the snapshot generation, so the first restore pays the network cost and later ones read local disk.
4. Still bandwidth-limited after that? This is where framed compression earns its keep — fewer bytes per chunk, paid for in CPU you apparently have spare. Benchmark it on the fault path, not on a sequential decompress.

If you're latency-bound

1. Do not compress the restore path. Whatever you were about to do, this is the one prohibition.
2. Make sure the restore is genuinely lazy — mapped or fault-served, not read into a buffer first. One well-meaning "read the file to checksum it" turns a lazy restore into a full read.
3. Prefetch the working set and keep it warm, so the pages needed in the guest's first milliseconds are local.
4. Prefer copy-on-write forking over fresh restores where the workload allows it. A fork shares memory until it's written, which is strictly less work than any restore.

The lesson generalizes past Firecracker. Compression optimizes bytes at rest; fast restore optimizes bytes never read. Those goals point in opposite directions, which is why sparse files, copy-on-write, dirty-page diffs, and demand paging all beat compression on the hot path — they attack the second problem directly. The zero page you never fetched is better compressed than the one you fetched at a 1000:1 ratio, and unlike the compressor, it cost nothing to decode.