Immutable microVM Rootfs with dm-verity

I'm Ajay; I build PandaStack, a Firecracker microVM platform, so read this as opinionated rather than neutral. The opinion is this: mounting a root filesystem read-only proves nothing. The ro flag is a promise the guest makes to itself, enforced by the same kernel that would happily be told otherwise, over an image file that anything on the host with root can rewrite while the guest is asleep. dm-verity is a different category of thing. It is a Merkle tree over the block device, and the kernel checks a block's hash on every read before handing the data to anyone. Modify a byte of the image and the read fails — not eventually, not on the next audit, on that read.

What a read-only mount actually promises

Read-only is genuinely useful and I use it constantly. It stops a package manager from mutating a template, keeps a build honest about where its outputs go, and makes accidental writes fail loudly instead of silently drifting a machine away from its image. All of that is about accidents, and against accidents it works well.

Against anything else, list what it does not cover and the picture changes fast. A process with CAP_SYS_ADMIN inside the guest can remount rw. Anything that can open the block device can write past the filesystem entirely — the ro flag lives on the mount, not on /dev/vda. The image file on the host is an ordinary file, and every write to it happens outside the guest's field of view. And storage does what storage does: a flipped bit in a rootfs on one host out of forty is not an attack, it is a Tuesday, and a read-only mount will serve you the flipped bit with total composure.

A read-only mount protects the image from the guest. It offers the guest nothing at all about whether the image is the one you built.

How dm-verity works: a Merkle tree under the filesystem

dm-verity is a device-mapper target that sits below the filesystem and above the real block device. You give it a data device, a hash device, and a root hash. It presents a virtual read-only device; every 4096-byte block read through it is hashed and checked against a leaf in a precomputed tree, whose interior nodes chain upward until they reach the single root hash you supplied. If the check fails, the read returns EIO. There is no configuration in which it returns the data anyway.

Three properties matter for how you design around it. It is lazy: verification happens per block on read, so opening a verity device is instant regardless of image size and you pay only for what you actually touch — which on a microVM booting a 2 GiB template is a small fraction of it. It is read-only by construction: there is no way to write through a verity device, because a write would invalidate a tree that was computed offline. And the entire trust of the system collapses to one 32-byte number, which means the interesting engineering is not in the hashing, it is in how that number reaches the kernel.

# The template rootfs, exactly as it will land on every host.
# Nothing about verity modifies the image -- it hashes what is already there.
ls -l rootfs.ext4
# -rw-r--r-- 1 root root 2147483648 rootfs.ext4        # 2 GiB = 524288 blocks

# Build the Merkle tree into a separate file. Defaults: sha256, 4096-byte
# data and hash blocks, and a RANDOM salt mixed into every leaf.
veritysetup format rootfs.ext4 rootfs.hash
# VERITY header information for rootfs.hash
# UUID:              1f0e0c8e-6b7a-4f61-9a3d-2c5b8e0d7a44
# Hash type:         1
# Data blocks:       524288
# Data block size:   4096
# Hash block size:   4096
# Hash algorithm:    sha256
# Salt:              9f3c1d0a...7d
# Root hash:         8b1a5c93...e2      <-- 32 bytes that stand for 2 GiB

# Tree overhead is about 1/128 of the data: one 32-byte digest per 4096-byte
# block, 128 digests per hash block, plus the levels above. ~0.8%.
du -h rootfs.hash        # 17M

# GOTCHA: that random salt means two runs of "format" over a byte-identical
# image produce two different root hashes. If you want reproducible builds --
# and for an image supply chain you do -- pin it. Either derive it from the
# content, or use --salt=- for no salt at all.
ROOT_HASH=$(veritysetup format --salt=- rootfs.ext4 rootfs.hash |
            awk '/^Root hash:/ {print $3}')
echo "$ROOT_HASH" > rootfs.roothash

# Offline verification of the whole device -- for CI, not for boot, because
# unlike the runtime path this one reads and hashes every block.
veritysetup verify rootfs.ext4 rootfs.hash "$ROOT_HASH" && echo "image is intact"

# Shipping two files is annoying. Append the tree to the image instead and
# hand the same file to both roles:
#   veritysetup format --hash-offset=2147483648 rootfs.ext4 rootfs.ext4

The root hash has to arrive out of band

This is the part people skip and it is the only part that decides whether any of it means anything. If the root hash travels next to the image, in the image, or in a metadata file fetched from the same bucket over the same channel, then an attacker who can rewrite the image can rewrite the hash, and you have built an elaborate checksum of whatever you were given. Verity does not authenticate; it binds. It proves the bytes you are reading are the bytes some root hash describes. Whether that root hash is yours is a separate problem you have to solve separately.

The practical answers, in increasing order of seriousness. Put the root hash on the kernel command line, so it is fixed by the host's configuration rather than by the artifact — good against a tampered image, useless if the attacker owns the host. Sign the root hash and verify the signature against a key in the kernel keyring, which is what dm-verity's root hash signature support is for. Or go all the way and put the root hash into a measured boot chain, so the number is attested rather than merely configured. Most fleets land on "signed at build, pinned in host config", and that is a defensible place to stop as long as you say out loud that the host is inside your trust boundary.

Opening the device, then breaking it on purpose

You should run the corruption test once before you trust any of this in production, because the failure mode is more interesting than the success case and it is not what most people guess.

# The runtime path, without the kernel involved yet.
veritysetup open rootfs.ext4 vroot rootfs.hash "$ROOT_HASH"
mount -o ro /dev/mapper/vroot /mnt/root
sha256sum /mnt/root/usr/bin/python3.12       # reads fine, every block checked

# Now be the adversary: one byte, deep inside the image, after formatting.
umount /mnt/root && veritysetup close vroot
printf '\x41' | dd of=rootfs.ext4 bs=1 seek=1099511627 conv=notrunc

# Re-open with the SAME root hash. Note what does NOT happen: this succeeds.
# Verity is lazy. It has verified nothing yet, because you have read nothing.
veritysetup open rootfs.ext4 vroot rootfs.hash "$ROOT_HASH"
mount -o ro /dev/mapper/vroot /mnt/root      # also succeeds -- superblock is clean

cat /mnt/root/path/to/the/file/in/that/block
# cat: /mnt/root/path/...: Input/output error      <-- EIO, not bad data

dmesg | tail -1
# device-mapper: verity: 254:0: data block 268435 is corrupted

# Failure policy is a mount-time decision, and "EIO" is often the WRONG one:
# an init script that ignores a read error will happily continue on a
# half-verified system. Fail the machine instead.
veritysetup open rootfs.ext4 vroot rootfs.hash "$ROOT_HASH" \
  --panic-on-corruption        # kernel panic; with panic=1 the VM dies and
                               # your platform's restart policy takes over
# Alternatives: --restart-on-corruption (reboot), --ignore-corruption (log
# only -- for forensics, never for production), --ignore-zero-blocks (skip
# hashing zero-filled blocks, useful for sparse images).

Two lessons in that transcript. First, opening a corrupted device succeeds, and so does mounting it, because verity checks blocks you read and you have not read the broken one yet. If you want an eager answer you run veritysetup verify, and you pay for reading the whole image — which is exactly the cost the lazy design exists to avoid. Second, the default EIO is a filesystem error, and filesystem errors get retried, logged and swallowed by software written by people who assumed a disk was flaky rather than that an image was forged. On a disposable microVM there is no reason to be subtle: panic on corruption, let the machine die, and let the platform report a failed create. A VM that refuses to exist is a much clearer signal than a VM serving I/O errors from one directory.

Wiring it into a Firecracker microVM

A microVM has no BIOS, no bootloader and no GRUB, which for once makes life simpler: the kernel command line is the entire boot configuration, nothing appends to it, and there is no interactive prompt to fix a typo in. Give the guest two drives — the image and the hash tree — and let the kernel assemble the verity device before it mounts root, using dm-mod.create.

{
  "boot-source": {
    "kernel_image_path": "/var/lib/pandastack/kernels/vmlinux-5.10",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off ro rootfstype=ext4 root=/dev/dm-0 dm-mod.create=\"vroot,,,ro,0 4194304 verity 1 /dev/vda /dev/vdb 4096 4096 524288 1 sha256 8b1a5c93...e2 9f3c1d0a...7d 1 panic_on_corruption\""
  },
  "drives": [
    {
      "drive_id": "rootfs",
      "path_on_host": "/var/lib/pandastack/img/rootfs.ext4",
      "is_root_device": false,
      "is_read_only": true
    },
    {
      "drive_id": "verityhash",
      "path_on_host": "/var/lib/pandastack/img/rootfs.hash",
      "is_root_device": false,
      "is_read_only": true
    }
  ]
}

Reading the table line left to right: start sector 0, 4194304 sectors (2 GiB at 512 bytes each), target verity, format version 1, data device /dev/vda, hash device /dev/vdb, 4096-byte data and hash blocks, 524288 data blocks, hash tree starting at hash-device block 1 because block 0 is the superblock, sha256, the root hash, the salt, then one optional argument: panic_on_corruption. root= names /dev/dm-0 rather than /dev/vda, because the thing you want mounted is the verified mapping, not the disk underneath it.

Three things will bite you, in roughly this order. The guest kernel needs CONFIG_DM_VERITY and CONFIG_DM_INIT built in, not as modules — a module cannot help you mount the root filesystem it lives on, and the minimal kernel configs that circulate in the microVM world generally have neither, so plan on building a kernel. Both drives should have is_root_device false with root= spelled out yourself, or Firecracker's own root= handling and your dm-mod.create will disagree about what /dev/dm-0 is. And the quoting in dm-mod.create is load-bearing: the table contains spaces, so the value must be quoted on the command line, which means escaped quotes in the JSON, which means somebody will lose an hour to a shell that ate them.

The alternative is a small initramfs that runs veritysetup open and then switch_root. It costs a boot stage and some RAM, and it buys you real error handling: you can log something human, retry a fetch, or drop to a debug shell instead of panicking with a kernel message. If your image comes from anywhere dynamic — an object store, a signed manifest, a per-tenant selection — do it in the initramfs. If the image is fixed at host provisioning time, dm-mod.create is fewer moving parts.

A verified rootfs is one you cannot write to

This follows from the mechanics but it reshapes the whole guest, so it is worth saying plainly: there is no such thing as a writable dm-verity device. The tree was computed offline over specific bytes; a write would make the tree wrong, and dm-verity's answer to that is not to update the tree, it is to refuse. So every byte your workload produces — logs, temp files, package installs, the application's own state — has to live somewhere else.

• An overlayfs upper layer on a second disk or a tmpfs, with the verified device as lowerdir. The guest sees an ordinary writable tree, the writes land in the upper, and the base stays verified. The cost is overlayfs's own sharp edges: a whole-file copy_up on first write, whiteouts for deletes, and a boot ordering problem you get wrong exactly once.
• A durable volume mounted at the paths that need persistence — /var/lib/postgresql, /data, /app/uploads — with everything else genuinely immutable. This is the cleanest model when you know what the workload writes, and it makes the answer to "what state does this machine have" a list rather than a shrug.
• tmpfs for /tmp, /run and /var/log, sized deliberately. Fast, disappears with the machine, and it turns "a runaway process filled the disk" into "a runaway process filled its own RAM budget and died", which is a much better incident.
• Nothing at all, for a true appliance. If the guest is one static binary answering requests, a verified read-only root with a tmpfs for /run is a complete and very pleasant design. Most workloads are not this, but the ones that are should take the offer.

The second-order consequence is operational rather than technical. You can no longer fix a machine by installing something on it. Every change becomes a rebuild of the image, a new root hash, and a redeploy — which is the point, and also the reason this fails in organisations that were not already building images in CI. dm-verity does not make your fleet immutable; it makes it impossible to pretend your fleet is immutable when it is not.

The honest tension with copy-on-write

Here is where I have to be careful, because this is the paragraph a vendor would fudge. PandaStack's create path does not boot a verified device. Every create reflinks the template rootfs into a private writable clone and restores a baked Firecracker snapshot on top of it — that is the whole reason a create is around 179ms p50 rather than the roughly 3 seconds a cold boot costs. The guest wakes into a filesystem that is writable by design, because the product is a sandbox that runs your code, and code writes files.

So what would dm-verity actually protect in a system shaped like this? The template image supply chain: the artifact that lands on the host. Verity gives you a cryptographic statement about rootfs.ext4 as it sits on disk on host number 37, checkable at any moment, on every block, for free-ish. That is genuinely valuable and it is where the value is. What it does not do is follow the CoW clone. The instant the guest gets a writable reflink of that image, verity is guarding the base, not the mutations — and the mutations are where the interesting things happen.

There is a second wrinkle specific to snapshot-restore platforms, and I have not seen it discussed much. A restored guest barely reads its rootfs at boot, because it is not booting: it resumes mid-life from a memory image. Most of the state you would want to trust — the running kernel, the page cache, the loaded binaries, the process table — comes out of vm.mem, not out of the block device dm-verity is verifying. On our streaming path that memory image is paged in on demand from object storage. A rootfs integrity story that ignores the memory image is answering a smaller question than it appears to.

Where it genuinely earns its keep

Three situations, and they are more common than the security framing suggests.

First, fleets that pull template images from object storage. A control plane publishes a new image, dozens or hundreds of hosts fetch it independently, and each one then serves it to guests for weeks. That is a long window in which the artifact sits on somebody's disk, and a checksum verified once at download time says nothing about the state of the file six hours later. Verity moves the check from "once, at fetch" to "continuously, on every read", which is the difference between a claim about the past and a property of the present.

Second, the rsync problem. You push a rootfs to forty hosts. Thirty-nine are fine. One has a bit-flip — bad RAM during the copy, a truncated transfer that resumed strangely, a disk that is failing quietly. The symptom is not an error, it is one host where 3% of sandboxes fail in a way nobody can reproduce, because the corrupted block only matters when something reads that particular file. I have chased this class of bug and it eats days. Verity converts it into an immediate, specific, loud failure that names the block number. That is not a security win, it is a debugging win, and honestly it is the one I would cite first.

Third, regulated environments that need image-integrity evidence. When an auditor asks how you know the image running in production is the image that came out of your pipeline, "we checked the SHA when we uploaded it" is a process answer. "The kernel refuses to serve a block that does not chain to this signed root hash, and here is the boot configuration that pins it" is a technical control, and it maps onto integrity-monitoring requirements without you having to write an essay. The root hash also makes a very tidy thing to record in a provenance attestation — it names the exact bytes, not a build.

fs-verity: the same idea, aimed at files instead of devices

fs-verity solves an adjacent problem and gets conflated with dm-verity constantly. It is per-file, it lives inside the filesystem (ext4, f2fs, btrfs), and crucially it works on a filesystem that stays writable. You enable it on a specific file; the kernel builds a Merkle tree for that file, makes the file immutable from then on, and verifies pages against the tree on every read. The rest of the filesystem carries on as normal.

# Per-file integrity on a filesystem that is still perfectly writable.
fsverity enable /usr/local/bin/pandastack-init
fsverity measure /usr/local/bin/pandastack-init
# sha256:3a5f0c...91  /usr/local/bin/pandastack-init

# The file is now immutable. Not "read-only by permission" -- the kernel
# refuses the write, as root, with the file owned by you.
echo evil >> /usr/local/bin/pandastack-init
# bash: /usr/local/bin/pandastack-init: Operation not permitted

# ...while everything around it behaves like an ordinary filesystem:
apt-get install -y jq          # fine
touch /usr/local/bin/whatever  # fine, and completely unverified

# The measurement is what you pin. Compare it against an expected value at
# launch, or use the kernel's builtin signature support so the check happens
# against a key in the keyring rather than in your init script.

The right way to choose between them: dm-verity is for "this whole image is the image I built", fs-verity is for "this specific binary is the binary I signed". If your guest needs a writable root but you want the interpreter, the agent binary and a handful of libraries to be tamper-evident, fs-verity fits where dm-verity cannot go. Its blind spot is symmetrical and important — it protects the files you enrolled, and says nothing about files added later, files you forgot, deletions, or a directory swapped out from under you. A per-file mechanism cannot make a statement about a filesystem.

Four approaches: what each detects, what each misses

Softest to hardest, with the honest failure column filled in. Kernel feature availability varies by distro and kernel config, so verify the specifics against your own kernel before you design around them.

• Plain read-only mount — Detects: accidental writes through the mount, which is a real and useful category. Misses: essentially everything adversarial. A remount, a raw write to the block device, any host-side edit of the image file, and every form of silent corruption pass straight through and are served as data. It is a promise the guest makes to itself, and the guest is not the party you were worried about.
• dm-verity — Detects: any modification to any block of the image, malicious or accidental, at the moment it is read, assuming the root hash reached the kernel through a channel the attacker does not control. Misses: everything outside the verified device — the kernel, the boot args, the writable layer stacked on top, the memory image of a restored snapshot. Costs: the device is read-only forever, the tree adds about 0.8%, and changing anything means rebuilding the image.
• fs-verity — Detects: modification of the specific files you enrolled, per page, on a filesystem that remains writable. Files become immutable once enabled, so it is tamper-evident and tamper-resistant at the same time. Misses: files you did not enroll, files created afterwards, deletions, renames, and any structural change to the filesystem. Integrity for binaries, not for an image.
• Sign the image, verify once, then trust the host — Detects: a tampered or corrupted artifact at download time, which catches the large majority of real incidents. Misses: everything after the check — a bit-flip that appears hours later, a write by any process with root, a swap of the file between verification and use, and slow storage decay. Cheap, boring, and what most fleets actually run, including ours.

What PandaStack actually does today

We do not run dm-verity in the guest. I want that stated flatly rather than buried, because the corpus of vendor security writing is mostly built out of things that are technically true about a feature flag somebody once tested.

What we do run is the fourth row of that comparison, done carefully. Template seeds are published per generation to object storage with a SHA256 manifest and a CURRENT pointer, so a host that syncs a seed checks the artifacts it downloaded against a manifest, and a re-bake produces a new generation rather than mutating one in place — which means a template change self-invalidates instead of silently drifting. Content-addressed generations also make the "one host has a stale image" failure a visible mismatch rather than a mystery. Transport is TLS. Streamed memory chunks are fetched over that same channel and are not individually Merkle-verified, which is a real gap and the obvious next thing to build: a per-chunk hash list next to the memory image would do for vm.mem roughly what the verity tree does for a rootfs.

Where I would reach for dm-verity, if you are building this yourself: on the host, over the template images, with the root hash pinned in host configuration and signed at build time. That gets you the rsync-bit-flip detection and the audit answer, which are the two benefits that actually pay for themselves, without pretending the sandbox's writable clone is immutable. And if you need integrity inside a guest that has to stay writable — an agent binary, an interpreter, a policy enforcement hook — enroll those specific files in fs-verity and pin their measurements. That combination is honest about what each layer covers, which is more than most immutable-infrastructure diagrams manage.

The summary

Read-only is a mount flag and mount flags are opinions. dm-verity is a Merkle tree the kernel consults before serving you a block, which turns tampering and bit rot from things you might notice later into an EIO — or, if you configure it sensibly, a dead VM and a failed create. Wiring it into a Firecracker guest is two drives, a dm-mod.create table on the boot args, a kernel with DM_VERITY and DM_INIT built in, and a decision about where the writes go, because a verified device is by definition one you cannot write to.

Then be honest about the boundary. Verity protects an artifact, not a running system: not the writable layer, not the memory image of a snapshot, not the host that named the root hash. On a copy-on-write, snapshot-restore platform like ours it belongs upstream of the guest, guarding the image that lands on every host — and fs-verity belongs inside the guest, guarding the specific binaries that must not change. Anyone selling you either as "immutable infrastructure" is selling the diagram, not the mechanism.