Copy-on-Write Rootfs: dm-snapshot vs reflink for MicroVMs

A sandbox platform that spawns thousands of ephemeral microVMs needs to hand each one a private, writable rootfs cloned from a shared golden image — without actually copying gigabytes off disk every time. There are two mainstream ways to do this on Linux, and they are both, at heart, the same con: a lie you tell the filesystem so it stops copying gigabytes and hands out a reference instead. One tells the lie at the file layer (reflink, via XFS or Btrfs). The other tells it one layer lower, at the block device (device-mapper's dm-snapshot). PandaStack supports both. This post is about how each actually works, where they differ in ways that matter at scale — clone latency, write amplification, page-cache sharing, cleanup — and how to pick.

Reflink: copy-on-write at the file layer

Both mechanisms exist to make one operation cheap: "give me a writable copy of this 2 GiB rootfs, now, without moving 2 GiB." A full cp reads and writes the whole image — seconds of I/O and a second physical copy. Copy-on-write instead clones a reference: the new disk shares the golden image's storage, nothing is copied up front, and the two diverge later, one unit at a time, as the guest writes. The clone is O(metadata) — proportional to how the image is laid out, not how big it is — so a 2 GiB and a 20 GiB rootfs clone in the same handful of milliseconds. The difference between reflink and dm-snapshot is not that contract; it's the layer at which the CoW bookkeeping lives, and everything downstream flows from that. Reflink (the cp --reflink operation, backed by the FICLONE ioctl) does it at the file layer: it creates a new file whose contents are identical to the source but shares the source's data extents — the filesystem's runs of on-disk blocks — and records in metadata that those extents now have more than one owner. When the guest writes a shared block, the filesystem allocates a fresh block and re-points only the clone's metadata; the golden image is untouched. This needs a filesystem that tracks reference-counted shared extents: XFS (with reflink=1, the default on modern mkfs.xfs) or Btrfs. ext4 has none of it, so cp --reflink fails on ext4 and silently degrades to a full byte copy.

# Confirm the host filesystem actually supports reflinks.
$ xfs_info /var/lib/pandastack | grep reflink
reflink=1        bigtime=1 inobtcount=1

# A reflink clone of the golden rootfs: O(metadata), not O(data).
$ time cp --reflink=always golden.ext4 clone.ext4
real    0m0.004s

# Both files map to the SAME physical extents — filefrag proves it.
# Note the identical physical_offset ranges and the "shared" flag.
$ filefrag -v clone.ext4 | grep -m1 shared
   0:        0..    8191:      98304..    106495:   8192:  shared

# First write to the clone copies only the touched block(s), then they diverge.
$ dd if=/dev/zero of=clone.ext4 bs=4k count=1 conv=notrunc 2>/dev/null
$ filefrag clone.ext4   # one extent now split off; the rest still shared

The operational shape of reflink is its selling point: creating a clone is one cp, deleting one is one rm. No device to set up, no second store to track, no teardown ordering — the clone is a normal file that happens to share storage, and the filesystem's reference counting frees shared blocks when the last owner goes away. For a platform churning thousands of ephemeral sandboxes, that simplicity is worth a lot.

dm-snapshot: copy-on-write at the block layer

dm-snapshot does the same thing one layer down. Device-mapper composes virtual block devices out of targets, and the snapshot target presents a writable view of a read-only base device (the "origin") plus a separate "COW store" for changed blocks. Reads of untouched regions pass through to the shared origin; the first write to a region copies that region into the COW store ("copy-out") and redirects future access there. The unit of that copy-out is the chunk size, and it is tunable — a knob with no equivalent in reflink. dm-snapshot doesn't care what filesystem the origin sits on, or whether there's a filesystem at all; it works over any block device, which is exactly why it's the fallback when the host filesystem isn't reflink-capable. One origin can back many snapshot devices at once — the golden image is read-only and every clone diverges into its own COW store. The table's 'P' flag makes the snapshot persistent (COW metadata survives a reboot); 'N' is transient — in-memory only, cheaper, gone on restart, which is often fine for a genuinely ephemeral sandbox.

# Sketch: a dm-snapshot CoW clone over the golden rootfs.
# 1) Expose the golden image and a small COW store as loop devices.
$ ORIGIN=$(losetup --find --show --read-only golden.ext4)
$ truncate -s 512M cow-clone.img
$ COW=$(losetup --find --show cow-clone.img)

# 2) Read the origin's size in 512-byte sectors.
$ SECTORS=$(blockdev --getsz "$ORIGIN")

# 3) Create a writable snapshot device. Table fields:
#    0 <sectors> snapshot <origin> <cow> P <chunksize-in-sectors>
#    'P' = persistent (COW metadata survives a reboot); 'N' = transient.
#    128 sectors = 64 KiB chunk (the copy-out granularity on first write).
$ echo "0 $SECTORS snapshot $ORIGIN $COW P 128" \
    | dmsetup create clone-01

# 4) Boot the microVM off /dev/mapper/clone-01 — writes land in the COW store,
#    the golden image stays pristine and shared by every other clone.
$ ls -l /dev/mapper/clone-01

# Teardown is explicit and ordered: snapshot first, then loop devices.
$ dmsetup remove clone-01 && losetup -d "$COW" "$ORIGIN"
$ rm -f cow-clone.img

The chunk size is where dm-snapshot gets interesting, and it's a genuine tradeoff with no free lunch. On the first write to any chunk, dm-snapshot copies the entire chunk into the COW store, even if the guest touched a single byte. A large chunk means fewer, larger copy-outs and less exception-table metadata to track — but more write amplification: touch one byte in a fresh chunk and you copy the whole chunk. A small chunk means finer copies and less wasted I/O per write — but more metadata, more exception-table lookups on the hot path, and more per-chunk overhead. Reflink has no equivalent decision: it copies at the filesystem's own block granularity, no tunable. So dm-snapshot hands you a dial reflink doesn't have — power if your write pattern is known, a footgun if it isn't.

There's one more difference that's often the deciding factor at high density: page-cache sharing. With reflink, two clones that share an extent are, from the kernel's point of view, the same physical blocks of the same storage — reads of shared regions can hit the same page-cache pages, so a golden-image block read into cache once benefits every clone still sharing it. That dedup matters enormously when hundreds of microVMs on one host boot from the same golden rootfs and read mostly the same unchanged blocks. With dm-snapshot, each snapshot is a distinct block device, so shared origin blocks tend to be cached per-device rather than deduplicated across clones — the same golden data can end up resident in cache multiple times. The origin's own cache helps, but you don't get reflink's clean cross-clone file-level dedup for free.

Head to head, dimension by dimension

• Clone speed — reflink: one cp --reflink, O(metadata), a few ms, returns instantly. dm-snapshot: create a COW store + one dmsetup create; also fast, but it's a device setup, not a file op.
• Write-path overhead — reflink: copies at the filesystem's block granularity, no tunable, predictable. dm-snapshot: copies a full chunk on first touch (write amplification), tunable via chunk size — powerful if your write pattern is known, a footgun if not.
• Page-cache sharing — reflink: shared extents can dedup in the page cache, so hundreds of clones reading the same golden blocks cache them once. dm-snapshot: each snapshot is a distinct block device, so shared origin data tends to be cached per-clone, wasting memory at density.
• Filesystem requirement — reflink: needs XFS (reflink=1) or Btrfs; fails on ext4. dm-snapshot: works over ANY block device regardless of filesystem — even none.
• Cleanup — reflink: rm the clone; the filesystem reference-counts and frees shared blocks. dm-snapshot: explicit, ordered teardown (dmsetup remove, then release loop/COW store) — more moving parts to leak if you crash mid-clone.
• Granularity — reflink: file-level (clone a rootfs.ext4 file). dm-snapshot: block-device-level (clone a raw device, filesystem-agnostic).
• Best fit — reflink: the default when you control the host filesystem and want maximum simplicity + page-cache dedup at density. dm-snapshot: when reflink isn't available (ext4 host, raw block device, LVM setups) or you need block-level control over the CoW granularity.

How this plays out in PandaStack

PandaStack restores a baked Firecracker snapshot on every create — no warm pool — and the rootfs clone is one small stage of a create that lands at a 179ms p50. On the reflink path, that clone step is roughly 4ms: O(metadata), so it doesn't grow with image size and disappears into the noise. Reflink is the default because the host filesystem is under our control (XFS with reflink on) and the page-cache dedup across same-template sandboxes is a real density win; dm-snapshot is supported and gated by PANDASTACK_DMSNAP (on by default unless set to 0) as the block-layer path for setups where reflink isn't the right fit. Either way, the guest sees an identical contract: a private, writable disk that shares its unwritten blocks with the golden template and diverges only as the sandbox writes. Neither is "better" in the abstract — they're the same copy-on-write idea told at different layers, and the layer is the whole decision. Own the filesystem and want the simplest clone/cleanup plus cache dedup? Reflink. Filesystem-agnostic, on raw devices or ext4, or need to tune the CoW granularity? dm-snapshot. PandaStack's core is open source under Apache-2.0, so you can measure this yourself: point the rootfs directory at XFS and time a create on reflink, flip PANDASTACK_DMSNAP and compare, then put it on ext4 and watch the fallback copy blow the budget. The reflink mechanics — and how the same principle extends to memory pages via MAP_PRIVATE to make a fork cheap — are in /blog/copy-on-write-rootfs and /blog/snapshot-and-fork-explained.

Reflink lies to the filesystem; dm-snapshot lies to the block device. Both let you clone gigabytes for the cost of a little bookkeeping — the only question is which layer you'd rather do the bookkeeping in.