Merging and Inspecting Firecracker Snapshots: snapshot-editor and the Rebase Workflow

Most writing about Firecracker snapshots stops at the moment of capture: you PUT to /snapshot/create, you get files, the end. That is the easy half. The interesting half starts a second later, when you are holding a directory of memory files and state files and have to answer questions like: which of these can actually be restored on their own? Which one is the base that everything else depends on? What Firecracker version wrote this thing? Can I delete last Tuesday's snapshots, or is that a load-bearing wall? This post is about the tooling and the lifecycle around snapshot files after they exist — merging diffs into bases, reading state files to debug a failing restore, running a snapshot chain with compaction, and the artifact discipline that keeps the whole scheme from turning into an archaeology project.

I'm Ajay — I build PandaStack, an open-source Firecracker microVM platform where every sandbox, database, and app is created by restoring a baked snapshot rather than booting a VM. That means snapshot artifacts are our build outputs, and the pipeline that produces, verifies, publishes, and garbage-collects them is production infrastructure, not a script somebody ran once. Everything below is the mechanism plus the operational rules, including the ones we learned by breaking them.

What you're actually holding after a snapshot

A Firecracker snapshot is two files that are not independently useful. The state file is the serialized VMM state — vCPU registers and MSRs, the interrupt controller, the clock, every virtio device's configuration and queue pointers, the memory-region layout. It is small and always written completely. The memory file is the guest's physical RAM, and this is the one whose contents depend on the snapshot type: a full snapshot writes every page, while a diff snapshot writes only the pages the guest dirtied since the last snapshot, leaving holes everywhere else.

That asymmetry is the whole reason a post-processing toolchain exists. A full snapshot is standalone — hand the pair to any compatible host and it restores. A diff snapshot is a delta with a dangling reference: sparse by construction, real bytes only where the guest wrote, nothing at all elsewhere. Point /snapshot/load at that file alone and you are asking Firecracker to resume a machine whose RAM is mostly missing. It will not politely fill in the blanks.

Taking the diff in the first place

For context, here's the capture side, because the shape of the rebase falls out of it. You enable dirty-page tracking before boot, take a full base, let the workload run, then take diffs that capture only what moved. Every snapshot requires a paused VM — you cannot serialize a moving target.

# Illustrative. Check the API spec for your Firecracker version.
SOCK=/run/fc.sock
api() { curl -s --unix-socket "$SOCK" -X "$1" "http://localhost$2" \
          -H 'Content-Type: application/json' -d "$3"; }

# Dirty-page tracking must be on BEFORE boot for diffs to be possible.
api PUT /machine-config '{"vcpu_count":2,"mem_size_mib":2048,
  "track_dirty_pages":true}'

# ... boot, warm up the workload (deps installed, caches hot) ...

# --- Base: a FULL, self-contained memory image ---
api PATCH /vm '{"state":"Paused"}'
api PUT /snapshot/create '{
  "snapshot_type":"Full",
  "snapshot_path":"/snap/gen-07/base.state",
  "mem_file_path":"/snap/gen-07/base.mem"}'
api PATCH /vm '{"state":"Resumed"}'

# ... more work happens; a fraction of RAM is dirtied ...

# --- Diff: ONLY the pages dirtied since the previous snapshot ---
api PATCH /vm '{"state":"Paused"}'
api PUT /snapshot/create '{
  "snapshot_type":"Diff",
  "snapshot_path":"/snap/gen-07/stage-01.state",
  "mem_file_path":"/snap/gen-07/stage-01.mem"}'

# stage-01.mem is SPARSE. On its own it restores nothing.
# stage-01.state, by contrast, is complete -- it describes the machine
# as of this instant, and it is the state file you keep after rebasing.

# --- Restore, once you have a complete memory image ---
api PUT /snapshot/load '{
  "snapshot_path":"/snap/gen-07/stage-01.state",
  "mem_backend":{"backend_type":"File",
                 "backend_path":"/snap/gen-07/merged.mem"},
  "resume_vm":true}'

Note the last call carefully. The state file it loads is the diff's — stage-01.state, the machine as of that instant — but the memory backend it points at is a complete image. The state file and the memory image are a matched pair that must describe the same moment; getting them out of sync is the most common way to produce a restore that fails somewhere confusing.

snapshot-editor: the tool for the after-party

Firecracker ships a companion binary, snapshot-editor, for working on snapshot artifacts outside a running VMM. It superseded the older standalone rebase-snap utility, which did exactly one thing (merge a diff memory file into a base) and is the tool most older blog posts and Stack Overflow answers still reference. If you are on a current Firecracker and reaching for rebase-snap, you probably want snapshot-editor instead.

The tool splits into two families of operation, and the split is genuinely useful as a mental model: things that modify memory files, and things that read state files.

Family one: memory-file rebase

The rebase operation takes a base memory file and a diff memory file and merges the diff's populated pages into the base, in place. Emphasis on in place: the base file is what gets modified. After the operation, the base file is a complete, standalone memory image reflecting the state at the moment the diff was taken, and the diff file has served its purpose.

That in-place behaviour is a footgun with excellent aim. If you rebase directly onto the only copy of your base, you have destroyed the base — every other diff that referenced it is now garbage, because it is layered against bytes that no longer exist. Copy first. Always copy first.

# ILLUSTRATIVE. Subcommand and flag names have moved between Firecracker
# releases -- run `snapshot-editor --help` and
# `snapshot-editor edit-memory --help` for YOUR version before scripting this.

# --- WRONG: rebasing onto the only copy of the base. ---
# snapshot-editor edit-memory rebase \
#   --memory-path /snap/gen-07/base.mem \
#   --diff-path   /snap/gen-07/stage-01.mem
# base.mem is now stage-01. Every other diff taken against the ORIGINAL
# base is now unrestorable. Congratulations, you have compacted your
# entire snapshot tree into one leaf.

# --- RIGHT: copy, then merge into the copy. ---
cp --reflink=auto /snap/gen-07/base.mem /snap/gen-07/merged.mem

snapshot-editor edit-memory rebase \
  --memory-path /snap/gen-07/merged.mem \
  --diff-path   /snap/gen-07/stage-01.mem

# merged.mem is now a COMPLETE, standalone memory image equivalent to a
# full snapshot taken at stage-01. Pair it with stage-01.state and it
# restores anywhere compatible, with no chain to carry.

# --- Longer chain: apply diffs OLDEST to NEWEST so later writes win. ---
cp --reflink=auto /snap/gen-07/base.mem /snap/gen-07/compacted.mem
for d in stage-01 stage-02 stage-03; do
  snapshot-editor edit-memory rebase \
    --memory-path /snap/gen-07/compacted.mem \
    --diff-path   "/snap/gen-07/${d}.mem" || { echo "rebase $d failed"; exit 1; }
done
# Pair compacted.mem with stage-03.state -- the NEWEST state file, because
# that is the machine description matching these merged bytes.

Order matters and is not negotiable: oldest to newest, so that a page written in stage-01 and rewritten in stage-03 ends up holding the stage-03 value. Apply them backwards and you get a memory image that is internally inconsistent in a way nothing will warn you about — a guest that restores fine and then behaves like it has amnesia about the last ten minutes.

Family two: state-file inspection

The other half of the tool reads state files and tells you what is in them. This is your debugger for "why won't this restore," and it is also the raw material for building automated compatibility checks. The two things you'll reach for constantly are the snapshot's format version and a dump of the VM state description; there is also vCPU and device state inspection for the deeper cases.

# ILLUSTRATIVE. Verify subcommand names with `snapshot-editor info-vmstate --help`.

# 1) Which snapshot data-format version wrote this state file? This is the
#    first question to ask when /snapshot/load fails on one host and works
#    on another -- restore is a compatibility check between this version
#    and the RESTORING Firecracker binary's supported range.
snapshot-editor info-vmstate version \
  --vmstate-path /snap/gen-07/stage-03.state

# 2) Dump the full VM state description: device model, memory regions,
#    the machine config the guest was frozen with. Use this to confirm the
#    snapshot's mem_size matches the memory image you're about to pair it
#    with, and that the device set matches what the restore host provides.
snapshot-editor info-vmstate vm-state \
  --vmstate-path /snap/gen-07/stage-03.state

# 3) Per-vCPU state, for the deep cases -- CPUID/MSR-shaped mysteries where
#    a snapshot restores cleanly and the guest dies later on an illegal
#    instruction because it was baked on richer silicon than it landed on.
snapshot-editor info-vmstate vcpu-states \
  --vmstate-path /snap/gen-07/stage-03.state

# Wire (1) into CI: record the format version in your manifest at bake time,
# and have the restore path refuse an incompatible pairing UP FRONT rather
# than discovering it halfway through a 500 on a customer's create.

That last comment is the point of the whole family. Inspection turns snapshot compatibility from a runtime surprise into a build-time assertion: record the format version at bake time, check it at restore time, and an incompatible snapshot fails as a clear refusal instead of an opaque error deep in the load path.

The lifecycle this enables: chains with compaction

Put capture and rebase together and you get a workflow that will feel deeply familiar to anyone who has built a log-structured store. Writes are cheap and append-only; reads require reconstruction; periodically you compact so reconstruction doesn't get expensive. Same shape, different bytes.

1. Bake a full base. Cold-boot the template, warm it up — install dependencies, prime caches, get the runtime to a ready state — then pause and take a full snapshot. This is the expensive operation you do once.
2. Restore and advance. Load the base, resume, and run the next stage of setup: a language toolchain, a model download, a dataset, whatever layer sits on top.
3. Diff-snapshot each stage. Pause, take a Diff, resume. Each one writes only the pages that stage dirtied, so a stage that changes little costs little to checkpoint.
4. Compact periodically. Copy the base, rebase the accumulated diffs onto the copy oldest-to-newest, pair the result with the newest state file, and publish that as a new full base. The chain resets to depth zero.
5. Retire the old chain — but only after the new base is verified by an actual restore. Deleting the inputs before you've proven the output works is how a compaction becomes an outage.

The reason to compact is not tidiness. It's that chain depth is a real operational liability, and it compounds silently.

Chain depth is a dependency graph, and you own it

Every diff you keep is a restore dependency. A chain of base plus twelve diffs means restoring the twelfth requires all thirteen files, present, intact, and applied in the correct order. That is thirteen chances for a bit of storage lifecycle policy, an overzealous cleanup cron, or a well-meaning engineer freeing disk space to break the whole thing. The failure is not graceful degradation — a missing base doesn't give you a slightly older machine, it gives you an unrestorable pile of sparse files.

Three things get worse as the chain grows: restore work, because you either pre-merge N files or resolve every page fault through an N-deep lookup; blast radius, because more files means more single points of failure for one restore; and your ability to reason about any of it, because at depth twelve across a fleet, "is this file safe to delete" is a question nobody can answer confidently. Compaction converts an N-deep dependency into one self-contained artifact — you spend the merge once so every subsequent restore is dependency-free.

Full snapshot vs diff-plus-rebase, side by side

• Capture cost — Full snapshot: serializes the entire guest RAM image every time, so cost scales with total memory. Diff snapshot + rebase: writes only dirtied pages at capture time, so cost scales with what actually changed — the reason diffs exist at all.
• Restore readiness — Full snapshot: pair the state and memory files and load; nothing else needed. Diff snapshot + rebase: not restorable until you merge onto the base, or resolve pages through the chain at fault time.
• Artifact count — Full snapshot: two files, self-describing. Diff snapshot + rebase: a base plus every intermediate diff, all of which must survive together or none of them matter.
• Failure mode — Full snapshot: file missing or corrupt, and you know immediately which artifact failed. Diff snapshot + rebase: a missing base or a diff applied out of order can fail loudly at load, or quietly produce an inconsistent guest.
• Runtime overhead — Full snapshot: no dirty-page tracking required on the running guest. Diff snapshot + rebase: needs track_dirty_pages enabled before boot, which adds a small write-protection cost while the guest runs.
• Portability — Full snapshot: ships to any compatible host as a unit. Diff snapshot + rebase: portable only as an entire chain, which is a worse thing to move and a much worse thing to version.
• Best fit — Full snapshot: baked templates and golden images restored constantly and independently. Diff snapshot + rebase: incremental checkpointing of long-running guests, staged build pipelines, and any case where you take far more snapshots than you restore.

Practical rules for handling snapshot artifacts

Four rules, each of which exists because ignoring it produces a specific and annoying incident.

First: never rebase across incompatible bases. Snapshots are version-coupled and CPU-model-sensitive. A diff is a set of page writes taken against one specific base, produced by one specific Firecracker version on one specific advertised CPUID. Merging a diff onto a base from a different bake generation, a different FC version, or a different CPU template is not a merge — it's writing arbitrary bytes into a memory image at plausible-looking offsets. There is no checksum in the merge itself that will save you; the operation will very likely succeed and hand you a memory image that is nonsense. Encode the base identity in your manifest and refuse mismatched merges before you run them.

Second: the memory file and the state file are a matched pair. They travel together, version together, and get deleted together. After a rebase, the merged image pairs with the newest state file in the chain, not the base's. Half a snapshot is not a degraded snapshot; it's a rounding error with a filename.

Third: checksum and manifest everything. These are multi-gigabyte binaries that move across networks and object stores, and a silently truncated upload looks exactly like a healthy file until a guest faults on a page in the corrupted region — minutes or hours after the restore reported success. A manifest with per-file SHA256 turns that into a fast, loud failure at the moment of download.

GEN=/snap/gen-07

# 1) Manifest every artifact in the generation, with a checksum and the
#    compatibility metadata a restore host needs to make a decision.
cd "$GEN"
sha256sum base.mem base.state compacted.mem stage-03.state > SHA256SUMS

FC_VERSION=$(firecracker --version | head -n1)
FMT_VERSION=$(snapshot-editor info-vmstate version \
                --vmstate-path stage-03.state)

cat > manifest.json <<EOF
{
  "generation": "gen-07",
  "template": "code-interpreter",
  "fc_version": "${FC_VERSION}",
  "snapshot_format_version": "${FMT_VERSION}",
  "cpu_template": "fleet-intersection-v2",
  "guest_kernel": "5.10",
  "mem_file": "compacted.mem",
  "state_file": "stage-03.state",
  "rebased_from": ["base.mem", "stage-01.mem", "stage-02.mem", "stage-03.mem"]
}
EOF

# 2) Verify on every download, BEFORE the artifact is eligible to serve.
#    A truncated multi-GB upload is indistinguishable from a healthy one
#    until a guest faults on a page inside the missing region.
sha256sum -c SHA256SUMS --quiet || { echo "checksum mismatch"; exit 1; }

# 3) Publish atomically, then flip a pointer. Never mutate a live generation.
#    Readers resolve CURRENT -> gen-07 in one step, so there is no window
#    where a restore sees a half-written directory.
printf 'gen-07\n' > CURRENT.tmp && mv CURRENT.tmp CURRENT

# 4) Garbage-collect only generations strictly BELOW the current pointer,
#    and only after a real restore of the new generation has succeeded.
#    Deleting a base is not reversible and not partial.

Fourth, and the one people forget: a snapshot is a secrets-bearing artifact. The memory file contains guest RAM verbatim — every environment variable, every API token the guest loaded, every private key in a process's heap, decrypted TLS session material, whatever was in a buffer at the instant you paused. It is a memory dump with a friendlier name. Treat it with the access controls, encryption-at-rest, and retention policy you'd apply to a core dump from production, because that is precisely what it is. Snapshot artifacts sitting in a world-readable bucket are a credential leak that hasn't been noticed yet.

How this maps onto a platform

PandaStack sits at the far end of this spectrum: we bake a template snapshot once and restore it thousands of times, with no warm pool of idle VMs anywhere. That inverts the usual economics. In a checkpointing workload you snapshot often and restore rarely, so cheap capture wins and diffs are the right tool. We restore constantly and bake occasionally, so a self-contained full snapshot is the right tool on the hot path — it costs more to produce, but it restores with zero dependency resolution, which is what lets a create land at roughly 179ms p50 and 203ms p99, with the restore step itself around 49ms. A first-ever cold boot with no snapshot yet takes about 3 seconds; every create after that is restore-priced.

The rebase workflow still shows up, just at bake time rather than create time. Building a template is inherently staged — base OS, then runtime toolchains, then warm caches — and diffs plus compaction are a natural fit for that pipeline. The output is what matters: a single, standalone, verified memory image published as a generation, with a SHA256 manifest and an atomic CURRENT pointer that agents resolve to find the snapshot they should be serving. Old generations get garbage-collected strictly below the current pointer, never above it, and never before the new generation has demonstrated a successful restore.

That pointer discipline is not theoretical hygiene. We shipped a bug once where a fan-out across many agents raced a mutable pointer with no compare-and-swap, briefly leaving the pointer naming a generation that garbage collection had already removed. Every pull failed. The fix was a monotonic compare-and-swap flip plus a GC that only ever collects below the current pointer — which is the same lesson as "don't rebase onto your only base," wearing a different hat. Snapshot artifacts have a dependency graph, and if you don't model it explicitly, you will eventually delete a node that something needed.

from pandastack import Sandbox

# Every create here is a restore of a baked, compacted, manifest-verified
# template snapshot -- no chain to resolve, no base to hunt down.
sbx = Sandbox.create(template="code-interpreter", ttl_seconds=600)

setup = sbx.exec("pip install -q pandas pyarrow", timeout_seconds=300)
if setup.exit_code != 0:
    raise RuntimeError(setup.stdout)

# Freeze this warm machine. The platform handles the artifact lifecycle:
# capture, checksum, manifest, publish, and eventual GC.
snap = sbx.snapshot()

# Branch it instead of rebuilding it. Same-host forks land in 400-750ms
# because memory pages and disk extents are shared copy-on-write until
# something writes -- the read-side twin of the diff/rebase idea.
child = sbx.fork()
out = child.exec("python -c 'import pandas; print(pandas.__version__)'",
                 timeout_seconds=30)
print(out.stdout, out.exit_code)

The bottom line

A diff snapshot is cheap to take and worthless alone, so the tooling that turns it back into something restorable is not optional infrastructure — it's the other half of the feature. snapshot-editor is where that lives in modern Firecracker, having replaced the older standalone rebase-snap: one family of operations merges a diff memory file into a base in place, and another reads state files so you can check format versions, dump the VM state description, and inspect vCPU state when a restore is misbehaving. Verify the exact subcommands and flags with --help against the version you actually run; the durable behaviour is stable, the CLI surface has moved between releases.

The lifecycle it unlocks is a log-structured store for machine state: base, warm up, diff per stage, compact periodically into a fresh base. Keep the chain shallow, because every diff you retain is a restore dependency and the base is the root of a tree that rm traverses depth-first. Never rebase across incompatible bases, keep the memory file and state file together as the matched pair they are, checksum and manifest every artifact, publish generations behind an atomic pointer, garbage-collect only below it, and treat every memory file as the verbatim RAM dump it is. Do that and snapshots become build artifacts with a boring, reliable pipeline. Skip it and they become a directory of large binary files that somebody will eventually be afraid to delete — which is its own kind of outage, just a slower one.

The core of PandaStack is Apache-2.0, so you can run the control-plane API and per-host agent on your own Linux KVM hosts and inspect the bake pipeline yourself. For the capture-side mechanics, see /blog/firecracker-diff-snapshots-explained; for the compatibility axes that make a cross-generation rebase dangerous, /blog/firecracker-snapshot-version-compatibility.