MicroVM Rootfs Filesystems: ext4 vs XFS vs Btrfs

"What filesystem should I use for my microVM rootfs?" is one of those questions that generates long threads and short answers, because it is secretly two questions. There is the filesystem inside the rootfs image — the one the guest kernel mounts at boot, the one that holds /usr and /home and whatever your workload writes. And there is the filesystem on the host — the one that stores those image files and, if you want creates to be fast, clones them copy-on-write. These two are almost always conflated, which is a shame, because they have nearly opposite requirements. The guest one wants to be boring, small, and universally supported. The host one wants a feature ext4 simply does not have. PandaStack ships ext4 inside the image and XFS with reflink underneath it, and that combination is not a compromise — it's the answer to two separate questions that happen to both be about filesystems.

Two filesystems, not one

Concretely: on the host there's a file, call it /var/lib/pandastack/templates/base/rootfs.ext4. It is a few gigabytes of raw block data — a complete filesystem image. The host does not mount it; it hands the file to Firecracker as a virtio-blk backing file. Inside the guest, the kernel sees a block device (/dev/vda), reads the superblock, discovers ext4, and mounts it as /. So the bytes inside that file are formatted by one filesystem, and the file itself is stored by another. Nothing requires them to match, and in practice they shouldn't. The guest filesystem is chosen for boot reliability and mount speed on a small ephemeral disk. The host filesystem is chosen for one property above all: can it clone a multi-gigabyte file in O(metadata) instead of O(data).

The conflation is understandable — people say "the rootfs is ext4" and mean the image, then someone reads that as "PandaStack runs on ext4" and concludes reflinks are impossible. Both statements are true of different layers. The image is ext4. The host directory it lives in is XFS with reflink=1. That's how you get an ext4 guest disk that still clones in a few milliseconds: the cloning happens below the image, not inside it.

Guest side: what goes inside the image

For the guest, ext4 is the boring correct default, and "boring" is the entire argument. It is supported by every Linux kernel you might reasonably boot, including the older stable kernels microVM platforms tend to pin. It has a small on-disk footprint and a simple mount path — no multi-device scan, no chunk tree, no background workers to start before the filesystem is usable. Most importantly, it is what the kernel you're using was tested with: Firecracker's own CI rootfs images are ext4, so you are walking a well-trodden path where bugs have already been found by someone else. On an ephemeral disk that lives for minutes and gets thrown away, exotic filesystem features buy you nothing and add ways to be surprised at 3am.

XFS in the guest is a fine filesystem that solves problems you don't have here. Its strengths — allocation-group parallelism, extremely good behaviour on large files and deep concurrency, mature online growth — are strengths at scale, on big multi-disk arrays, under sustained parallel I/O. A 2 GiB ephemeral disk backed by a single virtio-blk device is the opposite of that workload. You also can't shrink an XFS filesystem, which is genuinely annoying when your build pipeline wants to trim a freshly-populated image down to its minimum size before publishing it. If your guest workload is a database doing heavy parallel writes on a durable volume, XFS there is a real conversation. For the rootfs, it's a lateral move.

Btrfs in the guest is the one that sounds most appealing and delivers least, because its headline feature is already provided one layer up. Btrfs gives you subvolumes and snapshots — but the hypervisor already snapshots the entire disk, atomically, from outside the guest, along with the guest's RAM. A Btrfs snapshot inside the guest is a strictly smaller-scoped version of something you get for free, and it costs you a more complex mount path and a filesystem with more moving parts inside a VM you intend to discard. Btrfs also has known unhappiness with certain workloads (heavy random overwrite in databases, for instance) that you'd rather not debug through two layers of virtualization. Where Btrfs earns its keep in this architecture is on the host, not in the guest — see below.

The genuinely interesting alternative is going read-only: format the rootfs as erofs (or squashfs) and mount a small writable layer over it with overlayfs. This inverts the model — the base image becomes immutable and compressed, and everything the guest writes lands in an upper layer that can be a tmpfs, a small second block device, or a separate ext4 image. The wins are real: a compressed read-only image is smaller to store and ship, the guest physically cannot corrupt its own base, and there is no journal to replay because there is nothing to write. erofs in particular was designed for exactly this use case (immutable system images, fast random read, low memory overhead) and is the modern choice over squashfs. The costs are equally real: you now have two images and an overlay to assemble at boot, anything that expects a writable path outside the overlay breaks, and package installs at runtime write into the upper layer where they may be unexpectedly ephemeral. It's an excellent fit for locked-down, single-purpose guests. It's a poor fit for a general-purpose sandbox where users run apt install.

# Build an ext4 rootfs image for a microVM guest.
# Start oversized; we shrink to the real content size at the end.
$ truncate -s 4G rootfs.ext4

# mkfs flags that actually matter for an ephemeral guest disk:
#   -m 0       no reserved-for-root blocks; this isn't a shared system disk,
#              so 5% of the image reserved for nobody is pure waste.
#   -I 256     256-byte inodes (room for nanosecond timestamps + xattrs).
#   -O ^resize_inode  we resize offline at build time, not online in the guest.
#   lazy_*_init=0  do ALL inode-table + journal init NOW, at build time.
#              Otherwise the guest's first boot does it in the background --
#              which dirties blocks that were supposed to stay shared with
#              every other clone. Deterministic images clone better.
$ mkfs.ext4 -F -q -L PDSROOT \
    -m 0 -I 256 -O ^resize_inode \
    -E lazy_itable_init=0,lazy_journal_init=0 \
    rootfs.ext4

# Populate it from a container export (or debootstrap, or a build stage).
$ mkdir -p mnt && sudo mount -o loop rootfs.ext4 mnt
$ docker export "$(docker create my/base-image)" | sudo tar -x -C mnt
$ sudo umount mnt

# Never let a time- or mount-count-based fsck fire inside a restored guest:
# a snapshot-restored VM wakes with a clock from bake time, so "days since
# last check" arithmetic is nonsense. Disable both triggers.
$ sudo tune2fs -c 0 -i 0 rootfs.ext4

# Shrink to the minimum the content needs, then truncate the file to match.
$ sudo e2fsck -fp rootfs.ext4
$ sudo resize2fs -M rootfs.ext4
$ BLOCKS=$(sudo dumpe2fs -h rootfs.ext4 2>/dev/null | awk '/Block count/{print $3}')
$ BSIZE=$(sudo dumpe2fs -h rootfs.ext4 2>/dev/null | awk '/Block size/{print $3}')
$ truncate -s $(( BLOCKS * BSIZE )) rootfs.ext4
$ ls -lh rootfs.ext4

Host side: where the choice actually matters

Now the question that has a real answer. Every microVM create needs a private, writable disk cloned from a shared golden image. If the host filesystem can reflink, that clone is a metadata operation: the new file points at the same physical extents as the template, the filesystem reference-counts them, and blocks are only copied when the guest writes to them. XFS with reflink=1 (the default on modern mkfs.xfs) does this. Btrfs does this natively — it is copy-on-write by construction, and cp --reflink is essentially its normal mode of operation. ext4 does not, has no reference-counted shared extents, and never will without an on-disk format change. On ext4, cp --reflink=auto silently degrades to a full byte-for-byte copy of the entire image.

That degradation is not a rounding error — it changes the shape of your platform. A snapshot-restore create in PandaStack lands at a 179ms p50 and roughly 203ms p99, and the rootfs clone is one small stage inside that budget. Replace the reflink with a full copy of a multi-gigabyte image and you've put seconds of real disk I/O on the critical path, per create, forever. The same logic governs density: reflinked clones consume physical space only for the blocks a sandbox actually writes, so a hundred sandboxes from one template cost one template plus a hundred small deltas. Full copies cost a hundred templates. On a host with finite NVMe, that difference is the difference between running a hundred sandboxes and running six.

If you're stuck on ext4 for the host — an existing fleet, a managed volume you don't control, a distro default nobody wants to relitigate — the escape hatch is to do copy-on-write one layer lower, at the block device, with device-mapper's dm-snapshot. It is filesystem-agnostic precisely because it works underneath the filesystem, and PandaStack supports it as an alternative CoW path. It comes with its own tradeoffs (chunk-size write amplification, weaker page-cache sharing across clones, explicit ordered teardown), all of which are laid out in /blog/dm-snapshot-vs-reflink-cow. The mechanism behind reflink itself, and why the clone is O(metadata) rather than O(data), is in /blog/copy-on-write-rootfs.

# 1) Does the host filesystem actually support reflinks?
$ xfs_info /var/lib/pandastack | grep -o 'reflink=[01]'
reflink=1
# (On Btrfs, reflink is unconditional -- there's nothing to check.)
# (On ext4, this command finds nothing, which is your answer.)

# 2) Clone the golden image. --reflink=always FAILS LOUDLY if the filesystem
#    can't do it. Never use =auto in a build script: it silently falls back
#    to a full copy and you find out from your latency graphs.
$ cp --reflink=always templates/base/rootfs.ext4 vms/vm-01/rootfs.ext4

# 3) Prove the extents are shared, not copied. filefrag flags them:
$ filefrag -v vms/vm-01/rootfs.ext4 | head -5
Filesystem type is: 58465342
File size of vms/vm-01/rootfs.ext4 is 2147483648 (524288 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..  524287:   1048576.. 1572863: 524288:             shared

# 4) The honest space check is the filesystem's used count, not du.
#    Cloning a 2 GiB image should move this number by ~nothing.
$ df --output=used /var/lib/pandastack | tail -1
$ cp --reflink=always templates/base/rootfs.ext4 vms/vm-02/rootfs.ext4
$ df --output=used /var/lib/pandastack | tail -1

# 5) Watch divergence begin: one write un-shares one extent, not the file.
$ dd if=/dev/urandom of=vms/vm-01/rootfs.ext4 bs=4k count=1 conv=notrunc
$ filefrag vms/vm-01/rootfs.ext4   # one extent split off; the rest still shared

The comparison, both layers at once

• ext4 — In the guest: the correct default. Universal kernel support, small, mounts fast, extensively tested with the kernels microVMs pin, shrinkable at build time. On the host: the one thing you cannot use, because it has no reference-counted shared extents — cp --reflink fails and degrades to a full multi-gigabyte copy per create. Host ext4 forces you down to dm-snapshot or accept the copy.
• XFS — In the guest: solves problems a 2 GiB ephemeral disk doesn't have, and can't be shrunk, which annoys image build pipelines. On the host: the recommended choice. reflink=1 is the modern mkfs default, clones are O(metadata), shared extents can dedup in the page cache so many sandboxes booting the same template cache its unchanged blocks once, and cleanup is a plain rm.
• Btrfs — In the guest: gives you snapshots you already get from the hypervisor, at the cost of a more complex mount path and more surface area inside a VM you plan to discard. On the host: genuinely viable — copy-on-write by construction, reflinks work natively, plus subvolumes and checksumming if you want them. Weigh its behaviour under your actual write pattern; it is a different performance profile from XFS, not a strictly better one.
• erofs + overlayfs — In the guest: the interesting option for immutable, single-purpose guests. A compressed read-only base means a smaller image, no journal, and a guest that physically cannot corrupt its own root, with all writes landing in a small overlay. Costs you two artifacts to assemble at boot and breaks anything that expects a writable path outside the overlay. On the host: not applicable — erofs is read-only and can't store or clone mutable image files.

Mount options that actually matter

For an ephemeral guest disk, most mount tuning is superstition, but three options are worth understanding. noatime is the easy one: without it, every read updates an inode's access time, which turns reads into writes. In a copy-on-write world that is worse than merely wasteful — an atime update dirties a block that was shared with the template, so a purely read-only workload starts un-sharing extents and consuming real disk. Modern kernels default to relatime, which cuts most of it, but for a rootfs that nothing sane depends on atime for, noatime is free and strictly correct.

Journal behaviour is the interesting one. ext4's default data=ordered forces data blocks out before the metadata that references them, which is what stops you from reading someone else's deleted data after a crash. data=writeback relaxes that ordering for a bit less write traffic and, on a genuinely disposable disk, arguably acceptable risk. Some people go further and build the image with the journal disabled entirely (-O ^has_journal), which is a defensible choice for a rootfs that is discarded on every teardown and reconstructed from a template. Understand what you're trading: no journal means a crash leaves a filesystem needing a full fsck rather than a log replay, and "we just throw it away" only holds until someone stores something they care about on that disk. For anything durable — a managed database volume, a persistent workspace — keep the journal and keep the ordering.

discard is the one with the most cross-layer consequences. When the guest deletes files, the filesystem can issue discard requests that travel down through virtio-blk to the host, telling the backing store those blocks are free. That's how deleted guest data actually returns space to a sparse image or a thin-provisioned host volume — without it, an image file only ever grows toward its maximum size, no matter how much the guest deletes. The catch is that inline discard (the discard mount option) puts that work on the delete path, where it can stall a workload doing lots of small deletions. Periodic fstrim, run on a timer or explicitly before a snapshot, batches it into one predictable pass instead. The block-layer mechanics — including whether your virtio-blk device even advertises discard support — are in /blog/firecracker-virtio-blk-discard-trim-explained, and how the host caches those writes on the way through is in /blog/firecracker-block-device-cache-modes.

The snapshot angle: you are freezing a mounted filesystem mid-flight

Here is the part people discover the hard way. A Firecracker snapshot captures the guest's memory and device state at an arbitrary instant, and the rootfs is captured alongside it. That rootfs is a mounted, live, in-use filesystem. It has dirty pages in the guest's page cache, in-flight journal transactions, and metadata updates that are half-applied. You are not snapshotting a filesystem at rest; you are photographing a filesystem in the middle of a sentence.

The good news is that this is exactly the crash-consistency case journaling filesystems were designed for. The restored guest resumes, the journal replays, and ext4 arrives at a consistent state — which is why guests come back rather than exploding. The less-good news is what "consistent" means: consistent at the filesystem level, not at the application level. A half-written file is either fully there or fully not, but the database that was mid-transaction has no such guarantee unless it did its own fsync work. So quiesce before you capture: sync the guest, ideally freeze the filesystem, take the snapshot, thaw. It costs a moment of blocked writers and buys you a snapshot that doesn't need to replay anything to be correct.

Then there's the UUID problem, which is subtle and extremely annoying. A filesystem's UUID lives in its superblock — inside the image. Clone the image and you clone the superblock, so every clone has the identical filesystem UUID. That is completely fine while each clone is alone inside its own microVM. It stops being fine the moment two of them are visible to the same kernel, or anything resolves storage by UUID: /etc/fstab entries using UUID=, a root=UUID= kernel argument, systemd device units, a mount helper, or a host-side tool that scans /dev and tries to build a coherent picture. The classic failure is a host that loop-mounts two clones for inspection and gets a blkid cache that confidently reports the wrong device. Prefer device paths (/dev/vda) or labels inside the guest, and if you truly need distinct UUIDs, set them offline per-clone with tune2fs -U — remembering that this writes to the superblock, which un-shares that one block, and that any UUID= reference baked into the image must be updated to match or the guest won't boot.

# --- Quiesce before capture (run inside the guest) -----------------
# 1) Push dirty pages down to the virtual block device.
$ sync

# 2) Optional: return freed blocks to the host image before snapshotting,
#    so the captured disk isn't carrying deleted data.
$ fstrim -av

# 3) Freeze writers so the on-disk state can't move under the snapshot.
#    Keep this window SHORT -- every writer on / blocks until you thaw.
$ fsfreeze -f /
#    ... host takes the Firecracker snapshot here ...
$ fsfreeze -u /

# --- The duplicate-UUID check (on the host) ------------------------
# Every clone of an image carries the same superblock, so:
$ blkid -o value -s UUID templates/base/rootfs.ext4
7c2f5e10-9c1a-4c4e-8f0d-2b6a1d3e5f70
$ blkid -o value -s UUID vms/vm-01/rootfs.ext4
7c2f5e10-9c1a-4c4e-8f0d-2b6a1d3e5f70   # identical -- by design

# Fine in isolation; a problem the moment two are visible to one kernel,
# or anything resolves storage by UUID= (fstab, root=, systemd units).
# If you need distinct UUIDs, set them offline, per clone:
$ tune2fs -U random vms/vm-01/rootfs.ext4
#   NOTE: this writes the superblock, which un-shares that one block, and
#   any UUID= reference baked into the image must be updated to match.

The recommendation, and its limits

For a general-purpose microVM platform: ext4 in the guest, XFS with reflink on the host. The guest choice is about not being clever — ext4 is what your kernel expects, mounts without ceremony, and can be shrunk to size at build time. The host choice is about the only thing on the host that changes the platform's economics: reflink turns "clone a multi-gigabyte disk" into a metadata write, which is what lets a create land inside a 179ms p50 and lets a same-host fork branch a live machine in 400–750ms rather than moving gigabytes first. Btrfs on the host is a legitimate alternative if you want its subvolumes or checksumming and you've measured it against your write pattern. Host ext4 is the one configuration to actively avoid — and if you're stuck with it, use dm-snapshot rather than paying for a full copy on every create.

The honest limits. If your guests are immutable and single-purpose, erofs plus a small overlay is probably better than ext4 and you should try it — the compressed base and absent journal are real wins, and the assembly cost is a one-time engineering expense rather than a per-boot one. If your guest is a database with a durable volume, that volume is a separate decision from the rootfs and deserves its own analysis. And none of the performance claims here are numbers I'm going to invent for you: filesystems differ by mechanism in ways this post describes, but how those mechanisms cash out depends entirely on your image size, your write pattern, your density, and your storage hardware. Build both, run your real workload, and measure. PandaStack's core is open source under Apache-2.0, so the honest version of this experiment is available: point the rootfs directory at XFS and time a create, flip it to ext4, and watch the difference show up in the only graph that matters.

The guest filesystem should be boring. The host filesystem should be able to lie about copying. Confusing the two is how you end up copying gigabytes on every create and wondering where your latency went.

If you're still deciding whether the guest even needs a full rootfs image — or whether an initrd would serve better — /blog/firecracker-initrd-vs-rootfs-boot-explained covers that fork in the road, including why a first cold boot takes around 3s while a snapshot restore doesn't.