overlayfs Inside the Guest: Read-Only Rootfs, Writable Upper Layer

There are two places to put the copy-on-write when you want a microVM that writes to its filesystem without ever mutating the template it booted from. Inside the guest, with overlayfs: mount the template read-only, stack a writable layer on top, let the kernel present the union as one ordinary directory tree. Or underneath the guest, on the host, with a reflink clone or a dm-snapshot device — so the guest sees a plain writable disk and never learns anything is shared. Containers took the first road; overlay2 is Docker and containerd's default storage driver, and every image layer you have ever pulled is a lowerdir. PandaStack took the second. This post is about the first road: how overlayfs assembles a root filesystem, what copy_up costs when you get it wrong, the boot-ordering problem that catches everyone once, and the sharp edges under an elegant three-line mount command.

lowerdir, upperdir, workdir, merged

overlayfs is a union filesystem: it takes directory trees that already exist on other filesystems and presents a single merged view of them. It owns no storage. Everything it shows you lives in a real directory on a real filesystem underneath — which is why it is cheap, and why debugging it means going behind its back and inspecting the layers directly.

• lowerdir — the read-only layer(s). Your pristine template, mounted ro. Pass several colon-separated and the leftmost wins when a path exists in more than one. overlayfs never writes here, and is happy if the underlying filesystem is genuinely read-only.
• upperdir — the writable layer. Every create, modify, delete and rename that happens through the merged view lands here as a real file, a real directory, or a whiteout marker. Empty the upperdir and the merged view snaps back to pristine.
• workdir — private scratch space for staging copy_up and rename so they appear atomic. It must be an empty directory on the SAME filesystem as upperdir. It is not part of the merged view: do not write to it, do not back it up, do not put it on a different mount.
• merged — the mountpoint where the union appears, and the only path your userland should ever touch. Reading a path that exists only in lower reads through to lower; writing to it triggers a copy_up into upper first.

# The whole mechanism, in one mount.
# upperdir and workdir MUST be on the same filesystem, and workdir must be empty.
$ mkdir -p /mnt/lower /mnt/rw /mnt/merged
$ mount -o ro /dev/vda /mnt/lower
$ mount -t tmpfs -o size=50%,mode=0755 tmpfs /mnt/rw
$ mkdir -p /mnt/rw/upper /mnt/rw/work

$ mount -t overlay overlay \
    -o lowerdir=/mnt/lower,upperdir=/mnt/rw/upper,workdir=/mnt/rw/work \
    /mnt/merged

# Multiple lower layers: leftmost has priority. This is how image layers stack.
$ mount -t overlay overlay \
    -o lowerdir=/layers/app:/layers/runtime:/layers/base,upperdir=/mnt/rw/upper,workdir=/mnt/rw/work \
    /mnt/merged

# Omit upperdir and workdir entirely and you get a read-only union of the
# lower layers — no writable layer at all.
$ mount -t overlay overlay -o lowerdir=/layers/app:/layers/base /mnt/ro-merged

What copy_up actually costs

Here is the single most important fact about overlayfs, the one that turns a tidy design into a production incident: when you modify a file that lives only in lowerdir, overlayfs copies the entire file into upperdir before your write proceeds. Not the block you touched. Not the page. The file — data, mode, ownership, timestamps, xattrs. This is copy_up, and it is synchronous with your write.

# lower holds a big file; upper starts empty.
$ ls -l /mnt/lower/blob.bin
-rw-r--r-- 1 root root 2147483648 Aug 23 09:14 /mnt/lower/blob.bin
$ du -sh /mnt/rw/upper
0	/mnt/rw/upper

# Append exactly one byte through the merged view.
$ printf x >> /mnt/merged/blob.bin

# overlayfs copied all 2 GiB up before that one byte could land.
$ du -sh /mnt/rw/upper
2.1G	/mnt/rw/upper

# It is not only data writes. Without metacopy=on, a pure metadata change
# copies the whole file too:
$ chmod 600 /mnt/merged/other-blob.bin        # full copy_up
$ touch     /mnt/merged/third-blob.bin        # full copy_up
$ setfattr -n user.tag -v 1 /mnt/merged/x     # full copy_up

# Even opening for write and writing nothing is enough:
$ exec 3>>/mnt/merged/fourth-blob.bin; exec 3>&-   # full copy_up

# Deletes don't copy — they leave a whiteout: a 0/0 character device.
$ rm /mnt/merged/from-lower.txt
$ ls -l /mnt/rw/upper/from-lower.txt
c--------- 1 root root 0, 0 Aug 23 09:15 /mnt/rw/upper/from-lower.txt

The consequences land in predictable places. A database file, a preallocated log, an ML checkpoint in the template — any of those, touched once, is copied in full. If your upper is a tmpfs sized at half of guest RAM, one recursive chmod across a template full of large files can fill it and hand you ENOSPC on a filesystem that looked empty a second ago. Because copy_up is whole-file, cost follows the template's file layout, not the workload's write volume.

The microVM pattern: read-only virtio-blk plus a writable upper

The reason people reach for overlayfs in a microVM is not really performance. It is that the template image becomes physically immutable: attach it as a read-only virtio-blk device and no bug, no rogue process and no dd typo inside a guest can corrupt the image every other guest is booting from. Firecracker makes that a one-field decision on the drive.

{
  "boot-source": {
    "kernel_image_path": "/var/lib/microvm/vmlinux-5.10",
    "initrd_path": "/var/lib/microvm/initramfs.cpio.gz",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off ro overlay_root=ram"
  },
  "drives": [
    {
      "drive_id": "rootfs",
      "path_on_host": "/var/lib/microvm/templates/base/rootfs.ext4",
      "is_root_device": true,
      "is_read_only": true
    },
    {
      "drive_id": "scratch",
      "path_on_host": "/var/lib/microvm/vms/abc123/upper.ext4",
      "is_root_device": false,
      "is_read_only": false
    }
  ]
}

Because is_read_only is enforced by the VMM, one host file can back many microVMs with no risk of one scribbling on another: one image, N guests, zero per-guest disk provisioning. Reset is equally satisfying — nothing to roll back, because nothing was ever changed. Throw away the upper layer and the next boot is byte-identical to the template.

Where the upper layer lives is the real design decision, and it is a straight trade. A tmpfs upper is free to create, resets on every boot, and needs no per-VM host file — but it is RAM, it counts against the guest's memory budget, and it dies with the VM. A second small block device (the scratch drive above) keeps writes off the memory budget and survives a reboot — but now you garbage-collect one host file per guest, exactly the per-VM disk you were avoiding. Sparse images soften that, and a fixed-size scratch disk bounds a runaway guest with no quota machinery.

The ordering gotcha: you cannot do this from fstab

Here is the part that bites everyone exactly once. The filesystem you want to overlay is the root filesystem. By the time /etc/fstab is processed, init is already running from that root — the root is mounted, and it is too late. You cannot overlay a filesystem you are standing on.

So assembly has to happen in an earlier world: an initramfs, whose /init runs as PID 1 out of a RAM-backed root, does the mounts, and then hands over with switch_root. That is a shell script, not a config file, and it is the real deliverable of any "read-only rootfs with an overlay" project.

#!/bin/sh
# /init — PID 1 inside the initramfs, before the real root exists.
# Requires overlayfs (CONFIG_OVERLAY_FS=y, or the module bundled in here) and a
# mount(8) that understands -t overlay (busybox mount does).
set -e

mount -t proc     none /proc
mount -t sysfs    none /sys
mount -t devtmpfs none /dev

mkdir -p /mnt/lower /mnt/rw /mnt/merged

# LOWER: the pristine template, read-only virtio-blk.
mount -o ro /dev/vda /mnt/lower

# UPPER: a writable scratch disk if one was attached, else RAM.
# upperdir and workdir must be siblings on the SAME filesystem.
if [ -b /dev/vdb ]; then
  mount /dev/vdb /mnt/rw
else
  mount -t tmpfs -o size=50%,mode=0755 tmpfs /mnt/rw
fi
mkdir -p /mnt/rw/upper /mnt/rw/work

mount -t overlay overlay \
  -o lowerdir=/mnt/lower,upperdir=/mnt/rw/upper,workdir=/mnt/rw/work \
  /mnt/merged

# Carry the early pseudo-filesystems across instead of remounting them.
mkdir -p /mnt/merged/proc /mnt/merged/sys /mnt/merged/dev
mount --move /proc /mnt/merged/proc
mount --move /sys  /mnt/merged/sys
mount --move /dev  /mnt/merged/dev

# switch_root drops the initramfs contents and execs the real init as PID 1.
# It must be exec'd: PID 1 has to BE the new init, not a child of this script.
exec switch_root /mnt/merged /sbin/init

Three details there are load-bearing. exec matters because PID 1 must become the real init — fork it and you have a shell script supervising your operating system, reaping nothing and forwarding no signals. mount --move matters because remounting /proc and /sys after the pivot works right up until something holds a reference into the old ones. And the overlayfs driver must already be in the initramfs, built in or insmod'd before the mount; discovering that it is a module living on the root filesystem you have not mounted yet is a memorable five minutes.

In-guest overlay vs host-side copy-on-write

overlayfs, reflink and dm-snapshot are all copy-on-write. The interesting question is never whether it is CoW — it is which layer does the sharing, because the layer decides granularity and granularity decides cost.

• In-guest overlayfs — where CoW happens: inside the guest kernel, above the filesystem. Granularity: whole file (copy_up), unless metacopy=on defers the data. Reset cost: near zero — umount and hand the guest a fresh empty upper. Caveat: one touched byte in a big file copies the whole file, and assembly is initramfs work shipped inside every image.
• XFS reflink clone of the rootfs — where CoW happens: on the host, in the host filesystem, below the guest entirely. Granularity: filesystem block, a few KiB. Reset cost: rm the clone and cp --reflink a new one, both O(metadata). Caveat: the host filesystem must support reflinks (XFS with reflink=1, or Btrfs); on ext4 cp --reflink fails and you fall back to a full multi-GB copy.
• dm-snapshot — where CoW happens: on the host, at the block layer, below any filesystem. Granularity: the configured chunk size, the tunable reflink does not have. Reset cost: dmsetup remove plus discarding the COW store — cheap, but an ordered teardown with more to leak. Caveat: write amplification if the chunk is oversized, and each snapshot is a separate device, so shared origin blocks cache per-clone rather than deduplicating.
• Read-only virtio-blk image + writable second disk — where CoW happens: nowhere, technically. There is no copy; the split is by device and the guest decides what goes where (overlayfs, a symlinked /var, a bind mount). Granularity: whatever the guest chose. Reset cost: discard or reformat the scratch disk. Caveat: the immutability is VMM-enforced and real, but you now garbage-collect one host file per guest.
• Full copy of the image — where CoW happens: nowhere at all. Granularity: the entire image, eagerly. Reset cost: another full copy. Caveat: seconds of I/O and a second physical copy per guest — correct, dead simple, no kernel features required, and completely disqualifying on a create path measured in milliseconds.

Read down that list and the choice mostly makes itself. Want the template physically unwritable, and long-lived guests that reset in place without the host lifting a finger? In-guest overlayfs. Want cheap clones of a multi-gigabyte image with block-granular divergence and a guest that needs no cooperation? Host-side CoW. Both is legitimate — one read-only host image shared across guests, each running its own overlay — and plenty of appliance designs do exactly that. The tax is two CoW layers whose failure modes look nothing alike.

The sharp edges

Whiteouts and opaque directories

overlayfs cannot delete a file in the read-only lower layer, so it fakes it: it writes a character device with major 0, minor 0 into the upper layer at that path, and the merged view reads that as "this does not exist." Deleting a whole directory that exists in lower uses an xattr instead — trusted.overlay.opaque="y" on the upper directory, meaning "do not merge with lower; what you see here is all there is."

That is why naively archiving an upper layer betrays you. A tar run without device-node and xattr handling — or unpacked by a user lacking the privileges to create character devices and set trusted.* attributes — quietly drops the deletions, and files you carefully removed reappear from the lower layer on the other side. It is precisely why the OCI image spec encodes whiteouts as ordinary .wh.<name> files that any unprivileged tar can round-trip. If you ship upper layers between hosts, do the same.

redirect_dir, metacopy, and their caveats

Two mount options exist to make the expensive cases cheaper. metacopy=on lets a metadata-only change — chmod, chown, a timestamp, an xattr — copy up only the inode metadata, leaving the data in lower behind a reference, so the data copy waits until someone writes data. redirect_dir=on records a directory rename as a trusted.overlay.redirect xattr instead of copying the whole subtree up. Both are enormous wins for the workloads that hurt most.

Both also come with real caveats. They depend on kernel version and build configuration and can be off by default or gated behind module parameters, so a mount that works on your laptop may be quietly slower — or refuse — on an older kernel. They record state in trusted.overlay.* xattrs, so an upper written with these features on is not cleanly interpretable by a mount with them off. And the security caveat is the one to internalise: these xattrs tell overlayfs to go look somewhere else in the lower layer for data. If the upper layer can come from an untrusted source — a user-supplied image, an unpacked archive — crafted attributes become a way to point the merged view at files it should not expose. That is why following them is not unconditionally enabled.

Inode numbers, d_ino, and file watchers

overlayfs presents files whose real inodes live on other filesystems, and the seams show. The inode number a file reports can change after copy_up — it is a different inode on a different filesystem now — and readdir has historically been able to return a d_ino that does not match a subsequent stat's st_ino. Anything identifying files by (device, inode) notices: tar and rsync hardlink detection, du deduplication, find -samefile, build caches. The xino mount option (xino=on / xino=auto) fixes most of it by encoding the layer index into the inode number's high bits, at the cost of needing spare bits — which is why auto degrades on filesystems with 32-bit inode numbers.

The related annoyance is file watching. An inotify watch is attached to an inode. Watch a file that still lives in lower, let something write to it, and the write lands on a brand-new upper inode your watch knows nothing about — so the watcher sees nothing while the file visibly changes. If you have ever had a hot-reloading dev server that worked on the host and went deaf inside a container, this is a strong suspect. Watch directories, not files, where you can.

Stacking, SELinux, and the snapshot that freezes everything

Stacking overlays on overlays is the next thing people try, and it is worse than it looks. Using an overlay's merged directory as another overlay's lowerdir is permitted only in limited configurations depending on kernel version; using one as an upperdir is not supported at all. Even where it works you have multiplied every property above: copy_up can cascade, inode identity gets stranger, and "which of five layers is this file from?" degrades fast. Wanting a third level usually means moving the sharing down to the block layer.

SELinux is its own project. Labels travel with a file through copy_up like any other xattr, and overlayfs checks permissions against both the mounter's credentials and the accessing task's — subtler than it sounds when layers came from different places with different labels. Container runtimes handle it by supplying an explicit label for the merged mount. Hand-roll overlay roots on an enforcing system and expect the failures to look like inexplicable EACCES rather than anything mentioning SELinux.

Finally, the microVM-specific one: a snapshot freezes the upper layer along with everything else. If your upper is a tmpfs, its contents are guest RAM, so a Firecracker memory snapshot captures them — convenient, since a restored guest keeps its writes, but "reset by discarding the upper" no longer survives a restore and the memory file is now as large as everything the guest wrote. If the upper is a second block device, the snapshot must capture that disk consistently with the memory, or you restore a guest whose page cache and disk disagree about reality. Neither is a reason to avoid overlayfs — they are reasons to decide deliberately whether your writable layer is machine state or trash.

How PandaStack does it — and why differently

PandaStack runs no overlay inside the guest. Each sandbox gets a copy-on-write clone of the template's ext4 rootfs made on the host — an XFS reflink by default, dm-snapshot as the block-layer alternative — and boots off that clone as an ordinary read-write disk. Inside, / is plain ext4 on /dev/vda. There is no lowerdir, no whiteout, no initramfs assembly step, and nothing in the guest that has to know it is running on shared storage. Run findmnt in a fresh sandbox with sbx.exec("findmnt -no FSTYPE,SOURCE /").stdout and you get "ext4 /dev/vda" — grep the mount table for overlay and you get nothing.

The reasons are the ones the comparison already implies. Granularity: a sandbox appending to a large file copies one filesystem block, not the file — the exact case where in-guest overlayfs is worst. Guest neutrality: templates are ordinary images with ordinary init, so a user bringing their own rootfs need not also bring an initramfs that knows our layering scheme. And snapshot coherence: because every create restores a baked Firecracker snapshot rather than cold-booting, the disk clone and the memory restore are two halves of one operation. The reflink step is a few milliseconds inside a create that lands at a 179ms p50 and a 203ms p99, with the snapshot restore step itself around 49ms. A template's first cold boot, before a snapshot exists, takes about 3 seconds; after that nobody boots anything. A same-host fork — a second CoW clone of a live machine's disk plus a copy-on-write map of its memory — runs in 400–750ms, cross-host in 1.2–3.5s.

None of that makes in-guest overlayfs wrong. It makes it a different trade. If the template must be physically unwritable at the device level, if guests must reset themselves without host involvement, or if you already ship an initramfs — overlayfs is a good answer, and it is one mount command. If guests write to big files, or you want block-granular divergence and images that are plain bootable disks, put the copy-on-write below the guest. The one thing not to do is pick from the architecture diagram: take your real template, run your real workload, du the upper layer afterwards, and time the clone both ways. The layer that copies less wins, and which one that is depends entirely on files you already have.

overlayfs shares files; reflink and dm-snapshot share blocks. Both are copy-on-write, and both are cheap right until you touch something big — the only question is how much of it gets copied when you do.