Firecracker Block Device Cache Modes Explained

A Firecracker guest writes a block, and somewhere between its filesystem and the physical platter that block sits in a host page cache waiting to be flushed. Whether it survives a host crash before that flush happens is not an accident — it's a field you set in the drive config called cache_type, and it has exactly two values. The default respects the guest's flush requests, so when the guest's filesystem says "commit this now," Firecracker makes the host honor it. The other value, Unsafe, lets those flushes be no-ops and leaves the data in the host's write-back cache to be written whenever the kernel gets around to it. That's faster, and it's a perfectly good idea for a disk that's going to be deleted the moment the VM dies. It's a data-loss bug for a disk you promised a customer would persist. This post is about which is which, why the tradeoff exists, how it interacts with the io_engine choice, and how PandaStack splits ephemeral CoW rootfs from durable database volumes along exactly this line.

The guest disk is a host file (again)

Start where every block-device story starts: the guest's disk is a virtio-block device, and behind that device is an ordinary file on the host. The guest's own filesystem — ext4 formatted inside a rootfs.ext4 image — issues reads and writes against what it thinks is a physical disk, and Firecracker's virtio-block device translates each one into I/O against the backing file. We cover that request path and the two I/O engines that service it in /blog/firecracker-io-uring-block-io; the short version is that a guest write descriptor becomes a write to the host file at some offset. What we're adding here is the layer between that write and durable storage: the host's page cache, and whether the guest is allowed to force data through it.

This matters because a real filesystem doesn't just write — it writes and then flushes. When a database commits a transaction, or ext4 closes out a journal, the guest kernel issues a flush (and often marks specific writes Force-Unit-Access) to say: do not consider this done until it is actually on stable media. On bare metal that instruction travels to the disk controller. Inside a Firecracker VM, that flush arrives at the virtio-block device, and the device has to decide what to do with it. cache_type is that decision, made once at config time for the whole drive.

Write-back vs write-through, on the host

The two-value knob maps onto a much older idea: write-back versus write-through caching. When Firecracker writes to the backing file, that write lands first in the host kernel's page cache — a region of host RAM holding recently written file pages. From there the kernel eventually writes the pages down to the real storage device. The question is only ever about timing: when is the write guaranteed to be on durable media rather than sitting in that RAM cache?

In a write-through discipline, a flush from the guest forces the cached pages down to storage and doesn't return until they're durable — the guest's "commit this now" is honored end to end. In a write-back discipline, writes accumulate in the host page cache and get flushed lazily on the kernel's schedule; a guest flush that isn't forwarded is effectively a no-op, and the data is durable only whenever the kernel decides to write it back. Firecracker's cache_type picks which behavior the guest's flushes get. The flush-respecting mode passes the guest's flush through so committed data is truly committed; the Unsafe mode swallows the flush and lets everything ride the host's write-back cache.

The tradeoff: crash safety for latency

The reason anyone would ever choose Unsafe is that flushes are expensive. Every honored flush is a synchronous barrier: the guest stalls until the host confirms durability, which means waiting on the real storage device instead of returning the instant the write is in RAM. A write-heavy guest that flushes often — a database doing fsync on every commit, a filesystem journaling aggressively — pays that barrier over and over. Drop the flushes and those barriers vanish: writes return as soon as they're in the host page cache, and the guest never blocks waiting on physical storage. On boot and on write-heavy workloads that's a real, felt reduction in write latency.

The reason not to choose it is equally blunt: if the host crashes — kernel panic, power loss, the machine falls over — everything still sitting in the host's write-back cache that hasn't been flushed to disk is gone. The guest's filesystem believes those writes committed, because from inside the VM the flush "succeeded." It didn't. After the host comes back, the guest's disk can be inconsistent or simply missing its most recent committed data. For an ephemeral disk that dies with the VM anyway, that loss window is meaningless — the disk was never going to outlive the crash. For a disk that's supposed to be durable, it's silent corruption of exactly the data you promised to keep.

"cache=unsafe" is Firecracker being refreshingly honest about what you signed up for: the word in the config is the warning label. Most systems hide this behind a benign-sounding tuning flag; this one just tells you.

The two cache modes, side by side

Both modes back the same virtio-block device onto the same host file. They differ only in how a guest flush is treated and, therefore, in the crash-durability guarantee. Side by side:

• Guest flush handling — Default (flush-respecting): flush/FUA requests are forwarded to the host and honored. Unsafe: flush requests are dropped (treated as no-ops).
• Durability on host crash — Default: data the guest flushed is on stable storage and survives a host crash. Unsafe: un-written-back data in the host page cache is lost if the host crashes before lazy write-back.
• Write latency — Default: flushes are synchronous barriers, so flush-heavy workloads pay for durability. Unsafe: writes return once in the host page cache; no per-flush stall, lower write latency.
• What the guest believes — Default: a successful flush means truly committed. Unsafe: a successful flush means "in host RAM," which the guest can't distinguish from durable.
• Right fit — Default: durable volumes, databases, anything whose data must outlive a host crash. Unsafe: throwaway CoW rootfs and scratch disks that are deleted when the VM ends.
• Failure mode if wrong — Default on a scratch disk: harmless, you just paid for durability you didn't need. Unsafe on a durable disk: silent data loss of committed writes after a host crash.

Setting it in the drive config

Like io_engine, cache_type is a per-drive field you PUT to Firecracker's API before boot. It lives right next to the file path and the engine choice, and the two are orthogonal knobs that answer different questions: io_engine is about how efficiently permitted I/O is serviced (blocking Sync vs batched io_uring Async), and cache_type is about whether the guest's flushes are honored. You set both on the same drive object.

// PUT /drives/rootfs — an EPHEMERAL CoW rootfs.
// Unsafe cache: flushes are dropped, writes ride the host page cache.
// Safe here because the disk is deleted when the VM dies.
{
  "drive_id": "rootfs",
  "path_on_host": "/var/lib/pandastack/vms/demo/rootfs.ext4",
  "is_root_device": true,
  "is_read_only": false,
  "cache_type": "Unsafe",
  "io_engine": "Async"
}

// PUT /drives/pgdata — a DURABLE database volume.
// Default (flush-respecting) cache: the guest's fsync means fsync,
// so a committed Postgres transaction survives a host crash.
{
  "drive_id": "pgdata",
  "path_on_host": "/var/lib/pandastack/volumes/db-42/data.ext4",
  "is_root_device": false,
  "is_read_only": false,
  "cache_type": "Writeback",
  "io_engine": "Async"
}

Two things worth naming. First, the field values: Firecracker exposes cache_type as "Unsafe" (drop flushes) and "Writeback" — and despite the name, Writeback is the flush-respecting mode, because it forwards the guest's flush/FUA down so a write-back host cache can be forced durable on demand. If you omit cache_type, you get the safe, flush-respecting default; you have to opt into Unsafe deliberately, which is the right ergonomics for a footgun. Second, note that io_engine is set independently on both drives — cache_type does not change which engine services the I/O, only what a flush does once it arrives.

How PandaStack splits ephemeral from durable

PandaStack's storage model falls into two categories that map cleanly onto the two cache modes, which is what makes the choice easy rather than agonizing. The first category is the ephemeral copy-on-write rootfs every sandbox boots on: a reflink clone of a template's rootfs.ext4 (XFS reflink, O(metadata) to create), which exists only for the life of that sandbox and is discarded when it's deleted. Nothing on that disk is supposed to outlive the VM — it's a scratch surface for the guest's OS and the user's code. A host crash destroys the sandbox anyway, so the loss window that Unsafe opens is a window onto data that was already going to disappear. Aggressive host caching there is free speed with no durability cost, because there is no durability to lose.

The second category is the durable volume behind a managed database. A managed Postgres microVM is not backed by throwaway CoW rootfs for its data — it writes to a durable volume that is explicitly meant to survive, the whole reason someone stores data in a database instead of a scratch file. Here the guest's flushes must be honored: when Postgres fsyncs its WAL on commit, that has to mean the bytes are on stable storage, so that a host crash the millisecond after the commit returns cannot un-commit the transaction. That's the flush-respecting default, non-negotiable. The category boundary — disposable-with-the-VM versus must-outlive-a-crash — is exactly the boundary cache_type is designed to sit on.

Why the distinction matters

The honest framing is that cache_type is a durability contract expressed as a config field, and the only mistake is signing the wrong contract for a given disk. For the ephemeral rootfs — reflinked per sandbox, gone when the sandbox is deleted, restored from a snapshot in ~49ms on create (179ms p50, ~203ms p99 end to end) — dropping flushes is pure upside: the guest's OS and scratch writes never needed to survive a crash the disk itself wouldn't survive. For a managed database volume — created in the 30–90s it takes to bring Postgres up and meant to persist indefinitely — honoring flushes is the entire point of calling it durable, and Unsafe would quietly break the one promise that matters. Same virtio-block device, same host file, same io_engine choices underneath; one field decides whether a guest's fsync tells the truth.

None of this touches the isolation boundary — cache_type is about durability, not security; the guest still only ever sees a thin, audited virtio-block translator over a host file. What it governs is the correctness of persistence, which is a different axis entirely from how fast the I/O is (the engine) or how isolated the guest is (KVM plus the minimal device model). PandaStack's core is open source under Apache-2.0, so you can stand up the per-host agent on your own KVM hosts, set cache_type on a drive, and confirm for yourself that a flush-respecting durable volume survives a hard host reset that an Unsafe one wouldn't. For the request path and the Sync/Async engines, start at /blog/firecracker-io-uring-block-io; for the CoW rootfs mechanics, /blog/copy-on-write-rootfs; for the device model as a whole, /blog/firecracker-virtio-devices.

from pandastack import Sandbox

# Ephemeral sandbox: its CoW rootfs is disposable, so the platform
# can cache aggressively on the host. Nothing here needs to outlive
# the VM — the disk is reflinked on create and deleted on close.
sbx = Sandbox.create(template="code-interpreter")
sbx.exec("python -c 'print(2 + 2)'")
# ...work happens on a throwaway disk...
sbx.close()  # rootfs is discarded; a host crash would have lost nothing durable

# A managed database is the opposite contract: its volume is durable
# and flush-respecting, so a committed write is on stable storage.
# You don't tune cache_type yourself — the platform picks the safe
# default for durable volumes and the fast one for throwaway rootfs.