fsfreeze and the Three Tiers of Snapshot Consistency

Copy the disk of a running machine and you have not taken a backup. You have taken a crash-consistent image: precisely the bytes that would be on disk if someone had walked into the rack and pulled the power cord at that instant. Sometimes that is fine. The journal replays, the mount succeeds, the database comes up and tells you it recovered. Other times you get a mountable filesystem containing a half-written upload, a job file pointing at rows that were never committed, and a lockfile owned by a PID that no longer exists. This post is about the three tiers of snapshot consistency, the fsfreeze ioctl that gets you from the first tier to the second, why a full Firecracker VM snapshot is a structurally stronger position than a bare disk copy, and the specific things it still cannot save you from.

What “crash-consistent” really means

Crash-consistent means the image is a valid point-in-time view of the block device, and nothing more. Every block you copied is a block that really existed at some instant. Modern journalling filesystems are specifically built to survive this: ext4, XFS and btrfs all order their metadata writes so that a replay of the journal on the next mount produces a structurally sound filesystem. That is a real and underrated guarantee. It is also a guarantee about the filesystem, not about you.

Two things go wrong at this tier. The first is the page cache: a write your application issued may still be sitting in guest RAM, never having reached the virtual disk at all. A raw copy of the disk cannot see it, so it simply does not exist in your image. The second is ordering across independent things. A filesystem journal orders its own metadata; it does not order your application's writes against each other, and it certainly does not order writes to two different volumes. If your data files are on one disk and your write-ahead log is on another, a “simultaneous” copy of both is only simultaneous if something enforced that, and usually nothing did.

The three tiers, precisely

The industry names these tiers consistently enough that it is worth being exact about where each boundary sits, because vendors love to describe tier one using tier three's vocabulary.

• What is guaranteed — Crash-consistent: the blocks are real and the filesystem is repairable on mount. Filesystem-consistent: dirty pages are flushed and the on-disk image is a cleanly quiesced mount point, no journal replay needed. Application-consistent: the application's own on-disk invariants hold, because the application was told to make them hold.
• Who has to cooperate — Crash-consistent: nobody. Filesystem-consistent: the kernel (via the FIFREEZE ioctl on each mount). Application-consistent: the application itself, through its own backup or checkpoint protocol.
• Cost at capture time — Crash-consistent: zero, capture whenever you like. Filesystem-consistent: a freeze window during which every writer on that filesystem blocks. Application-consistent: the freeze window plus whatever the app's quiesce protocol costs, which can be substantial.
• What recovery looks like — Crash-consistent: mount, replay the journal, run the app's own crash recovery, hope. Filesystem-consistent: mount clean, then run the app's crash recovery. Application-consistent: mount clean, start the app, no recovery path taken at all.
• How it fails — Crash-consistent: torn application state that fsck cheerfully declares healthy. Filesystem-consistent: same torn application state, but with a clean filesystem underneath it. Application-consistent: it fails by being slow and by requiring you to actually integrate with the app.

Notice the trap in the middle row. Filesystem-consistent is a genuine improvement — you stop losing writes that were stuck in the page cache — but it does not promote your application's state one inch. A frozen filesystem containing a half-finished multi-file transaction is a very clean filesystem containing a half-finished multi-file transaction.

What application-consistent actually requires

There are only three ways to get to the top tier, and they are all forms of asking the writer to stop lying to you. You can stop the writers entirely — crude, effective, and by far the most common thing people actually do. You can take the application's own lock or checkpoint. Or you can use the application's documented hot-backup protocol, which is what any serious database ships.

For PostgreSQL, the modern protocol is pg_backup_start() and pg_backup_stop(), which bracket a physical backup and produce the label and WAL information the restore needs. Older releases spelled these pg_start_backup() and pg_stop_backup() with a separate exclusive-mode variant that has since been removed — check the function names against the exact major version you are running before you wire them into a script. A plain CHECKPOINT is a different and weaker thing: it flushes dirty buffers and shortens the subsequent recovery, but it does not bracket a backup and it does not by itself make an image application-consistent.

How fsfreeze actually works

fsfreeze is a util-linux command that wraps two ioctls: FIFREEZE and FITHAW. You give it a mount point, not a device — the operation is per-superblock. On freeze the kernel flushes dirty pages belonging to that filesystem, drives the journal to a committed state, marks the superblock frozen, and then blocks new write-side operations. Reads that do not themselves cause a write continue to work. Everything that tries to modify the filesystem parks in uninterruptible sleep until the thaw.

• It needs privilege — the ioctls require CAP_SYS_ADMIN, so this is a root-equivalent operation inside the guest.
• It is per-mount, not per-disk — freezing /data does nothing whatsoever for /var/lib/postgresql if that is a separate mount. Every filesystem holding state you care about needs its own freeze.
• It is not universal — the filesystem has to implement freeze. ext4, XFS and btrfs do; plenty of others return an error. Test on the exact filesystem you plan to freeze rather than assuming.
• Freezing twice is an error, not a no-op — a second freeze of an already-frozen filesystem fails rather than incrementing a counter, so concurrent snapshot jobs need their own mutual exclusion. Confirm the current behaviour against your kernel's documentation.
• It says nothing about the application — fsfreeze quiesces a filesystem. It has no idea what a transaction is.

The deadlock, which you will absolutely hit once

Here is the failure everyone learns the hard way. You freeze a filesystem, and then the process that issued the freeze tries to write to it. Maybe it appends to a log. Maybe your shell writes history. Maybe a monitoring agent flushes a metrics file, or the orchestrator persists a state record, or systemd-journald decides now is the moment. That process blocks in D state on a filesystem it is the only one who can unfreeze, and the guest is now a very consistent brick.

Freezing the root filesystem from a shell whose stdout lives on the root filesystem is the canonical way to turn a 200-millisecond operation into an incident review. The rule that follows is simple: drive the freeze from outside the guest, or from a small in-guest agent that holds no file descriptors, no logs, no cwd and no libraries it might page in from the frozen filesystem. And whatever you do, make the freeze short and give it a deadline.

The freeze/thaw sequence, with a dead-man switch

Two safety nets, because they cover different failures. A trap handles the case where your script errors out or gets Ctrl-C'd. A background watchdog handles the case where your script is SIGKILLed and the trap never runs — which is exactly what happens when an orchestrator times out and reaps the job it was babysitting.

#!/usr/bin/env bash
# freeze-snapshot-thaw.sh — runs on the HOST (or in a tiny agent that owns
# nothing on the frozen filesystem). Never run this from a shell whose
# stdout, cwd, or logs live on $MNT.
set -euo pipefail

MNT="/data"            # the mount point to quiesce
THAW_DEADLINE=15       # seconds — dead-man switch if we die mid-snapshot

# 0. Quiesce the application FIRST. fsfreeze knows nothing about your app.
psql -qc 'CHECKPOINT;'

# 1. Thaw on any exit path: error, Ctrl-C, or a normal return.
trap 'fsfreeze -u "$MNT" 2>/dev/null || true' EXIT

# 2. Thaw even if this process is SIGKILLed and the trap never runs.
( sleep "$THAW_DEADLINE"; fsfreeze -u "$MNT" 2>/dev/null || true ) &
WATCHDOG=$!

# 3. Freeze: flush dirty pages, commit the journal, block new writes.
#    From here until the thaw, every writer on $MNT is in uninterruptible sleep.
fsfreeze -f "$MNT"

# 4. Take the image. Keep this step SHORT — it is the whole freeze window.
cp --reflink=always /var/lib/vm/rootfs.ext4 /snap/rootfs.ext4

# 5. Thaw explicitly, then disarm the watchdog.
fsfreeze -u "$MNT"
trap - EXIT
kill "$WATCHDOG" 2>/dev/null || true

echo 'image captured: filesystem-consistent'

Why a full VM snapshot is a stronger starting position

A Firecracker snapshot is not a disk copy. It is the guest's entire physical RAM, the VMM's serialized device and CPU state, and the disk, captured with the vCPUs paused. That difference matters for consistency in a way that is easy to miss: the page cache is inside the memory image. Those dirty pages that a bare disk copy silently loses are captured, because they were in RAM and RAM is part of the snapshot. The filesystem's in-memory state and its on-disk state are captured at the same instant and describe each other.

The second difference is that there is no “during.” The vCPUs are stopped between two instructions before anything is serialized, so there is no window in which the guest keeps writing while you copy. Restore is a resume, not a boot: the guest never crashed, so there is no journal to replay and no crash-recovery path to take. On PandaStack that restore is the normal create path — 179ms p50, 203ms p99 end to end, with the snapshot-load step itself around 49ms; only the first spawn of a template pays the roughly 3-second cold boot that bakes the snapshot in the first place.

# A full VM snapshot, driven from the host over Firecracker's API socket.
# The guest is not asked to cooperate — it is simply stopped between two
# instructions and serialized whole.

# 1. Pause the vCPUs. You cannot serialize a moving target.
curl --unix-socket /run/fc.sock -X PATCH 'http://localhost/vm' \
  -d '{"state": "Paused"}'

# 2. Write device/CPU state + the entire guest RAM image.
#    The page cache is inside vm.mem, so dirty pages that never reached
#    the disk are captured too — that is the part a bare disk copy loses.
curl --unix-socket /run/fc.sock -X PUT 'http://localhost/snapshot/create' \
  -d '{
        "snapshot_type": "Full",
        "snapshot_path": "/snap/vm.state",
        "mem_file_path": "/snap/vm.mem"
      }'

# 3. Clone the disk while the guest is still paused, so memory and disk
#    describe the same instant.
cp --reflink=always /var/lib/vm/rootfs.ext4 /snap/rootfs.ext4

# 4. Resume. Total stop-the-world window: the pause, not the copy.
curl --unix-socket /run/fc.sock -X PATCH 'http://localhost/vm' \
  -d '{"state": "Resumed"}'

So a full VM snapshot lands you at least at filesystem-consistent for free, and arguably somewhere better, since you also captured the in-flight application state that lived in RAM — the half-parsed request, the connection pool, the open file descriptors. What it does not do is make anything application-consistent, because consistency and correctness are not the same word.

What a VM snapshot still cannot fix

A snapshot captures everything inside the box perfectly. Anything whose other half lives outside the box is captured in a state that is now a lie.

• Open TCP connections — the guest's socket is preserved down to the sequence numbers. The peer's is not; it was never in the snapshot. After any real pause the peer has timed out, the FIN or RST it sent hit a stopped guest and vanished, and host NAT state has been swept. Your application resumes holding a connection that exists only in its own imagination.
• Replication streams — a replica restored from a snapshot resumes asking for a WAL position the primary may have long since recycled. The stream doesn't resume; it fails, and hopefully your tooling notices.
• Leased locks — a lease from etcd, Consul, Redis or a database advisory lock expired while the guest was paused, and has very likely been handed to someone else. The restored guest still believes it is the leader. This is how you get two leaders.
• The clock — a restored guest wakes up believing it is the moment of the snapshot. Certificate validity windows, token expiry, cron catch-up and anything that reasons about elapsed time all resume with a stale worldview until something resyncs the clock.
• Anything with a nonce — restore the same snapshot twice and both children start from byte-identical memory, including seeds, session keys and counters that were supposed to be unique.

A snapshot freezes your machine. It does not freeze the internet.

The ordering recipe

Put together, the sequence is boring, which is the highest compliment you can pay an operational procedure.

1. Quiesce the application. Stop the writers, take the app's lock, or call its hot-backup entry point (pg_backup_start for Postgres 15 and later). This is the step everyone skips and the only one that buys the top tier.
2. Flush what the application owns — a CHECKPOINT, an explicit fsync, whatever the app offers — so the freeze has less work to do.
3. Arm the thaw deadline BEFORE the freeze. A watchdog you start afterwards is a watchdog that never starts if the freeze is the thing that hangs.
4. Freeze each mount you care about. With nested mounts, freeze the deepest first so an outer freeze never blocks the tooling walking into an inner one.
5. Take the image, and only the image. Do not upload, compress, or checksum inside the freeze window — clone cheaply (reflink, dm-snapshot, or a full VM snapshot) and do the expensive work after the thaw.
6. Thaw in reverse mount order, then disarm the watchdog.
7. Release the application: pg_backup_stop, drop the lock, restart the writers.
8. Record the tier as metadata next to the artifact. An image whose consistency tier is unknown is, operationally, crash-consistent — and you will find out which it really was at the worst possible moment.

Driving it from outside the guest

The safest place to issue a freeze is not inside the filesystem you are freezing. With the PandaStack SDK the control flow lives on your machine and only the freeze itself runs in the guest, which keeps your orchestration off the frozen mount — and the in-guest script still carries its own dead-man switch, because defence in depth costs three lines.

from pandastack import Sandbox

# A freeze/thaw drill you can actually run. `with Sandbox.create(...) as sbx:`
# works too and kills the VM on exit; here we do it explicitly so the
# teardown is visible.
sbx = Sandbox.create(template="base", ttl_seconds=600)

FREEZE = r'''#!/bin/sh
set -e
MNT="$1"
# Dead-man thaw: if this script is killed, the guest still wakes up.
( sleep 10; fsfreeze -u "$MNT" 2>/dev/null || true ) &
trap 'fsfreeze -u "$MNT" 2>/dev/null || true' EXIT
sync
fsfreeze -f "$MNT"
echo frozen
# <- the snapshot would be taken here, from OUTSIDE the guest
fsfreeze -u "$MNT"
echo thawed
'''

try:
    sbx.filesystem.write("/root/freeze.sh", FREEZE)
    sbx.exec("chmod +x /root/freeze.sh", timeout_seconds=10)

    # Does this filesystem even implement FIFREEZE? Not all of them do.
    r = sbx.exec("/root/freeze.sh /", timeout_seconds=30)
    print(r.exit_code, r.stdout.strip(), r.stderr.strip())

    # Application tier: quiesce the writer, not just the filesystem.
    sbx.exec("psql -qc 'CHECKPOINT;' || true", timeout_seconds=30)

    # Record which tier this artifact actually is, next to the artifact.
    sbx.filesystem.write("/root/consistency.txt", "tier=filesystem\n")
    print(sbx.filesystem.read("/root/consistency.txt").decode())
finally:
    sbx.kill()

Note the shape: the script that freezes is tiny, static, holds nothing open on the target mount, and thaws itself on a timer regardless of what happens to the caller. If the exec times out or your process dies, the guest wakes up anyway. That is the entire design goal.

Fork and clone inherit the tier they were born with

This is the part that bites teams who adopt fork-based workflows. A fork is a copy-on-write clone of a snapshot's memory and disk — fast precisely because it copies nothing up front. On PandaStack a same-host fork lands in 400–750ms and a cross-host fork in 1.2–3.5s, because the child is a lazily-materialized view of the parent rather than a new machine.

Which means the child's consistency tier is exactly the parent snapshot's tier. Fan out fifty children from a crash-consistent image and you have not created fifty independent problems; you have created one problem, fifty times, and every child holds the same torn upload and the same stale lease. Conversely, pay the quiesce cost once on the base image and every fork inherits a clean starting point for free. If you are branching a database, that is the whole argument for spending 30–90 seconds provisioning a properly quiesced base rather than cloning whatever was on disk when you happened to look.

What to actually do

• Decide the tier you need per workload, not per platform. A scratch build cache is fine crash-consistent. A customer's database is not.
• If you are copying a live disk and nothing else, say the words “crash-consistent” out loud in the design doc so nobody later assumes otherwise.
• Use a full VM snapshot where you can — pausing the vCPUs and capturing RAM removes the whole class of “the write was still in the page cache” bugs without asking the guest for anything.
• Add fsfreeze when you must copy a disk out from under a running guest, and never without a trap and a watchdog.
• Add the application's own protocol when the data has an owner who would be upset to lose it.
• Assume every external relationship — connections, leases, replication, clocks — is broken on restore, and make reconnection the guest's normal startup path rather than an exceptional one.

The tiers are a ladder, and each rung costs something real: a freeze window, an integration with your application, a slower capture. The mistake is not picking a low rung. The mistake is picking a low rung and then describing it, in the runbook, using the vocabulary of a high one.