virtiofs vs virtio-blk: How Files Actually Get Into a MicroVM

The question arrives in some form every couple of weeks: can I just mount a host directory into the sandbox? People ask it because it is how Docker works, and because the alternative sounds absurd — build a whole disk image to move a 4 KB file? There are two real answers in the microVM world, they are not variations on one theme, and picking between them determines things that look completely unrelated to storage: whether you can clone a running machine in under a second, how much host code parses attacker-controlled input, and what a multi-tenant compromise actually reaches.

I'm Ajay. I build PandaStack, a Firecracker platform where every sandbox create is a snapshot restore at 179ms p50, so I am not neutral here — our rootfs is a block image and I will explain exactly why. But the interesting part of this comparison is not which one I picked. It is that the two designs put the filesystem on opposite sides of the trust boundary, and almost every downstream property follows from that one decision.

What the guest actually sees

With virtio-blk, the host hands the guest a block device — /dev/vda — and nothing else. Behind it, on the host, is a file: an ext4 image, a loop device, a device-mapper target, an LVM volume. The host does not know or care what is inside it. The guest kernel probes the device, finds a partition table or a bare filesystem, mounts it, and from that point every open, every stat, every readdir is resolved by the guest's own ext4 driver reading blocks. The requests crossing the boundary are read 512 sectors at offset N and write this buffer at offset M. That is the entire vocabulary.

With virtiofs, the host hands the guest a directory. On the guest side a filesystem driver mounts it; on the host side a daemon — virtiofsd — is running, holding that directory open. The requests crossing the boundary are FUSE messages: LOOKUP this name in that parent inode, GETATTR, OPEN, READ from this file handle, SETXATTR, RENAME. The guest is not reading blocks. It is asking a host process to perform filesystem operations on its behalf, and the host process answers using the host's own filesystem, page cache, inode numbers and permission model.

So the two are not competing implementations of the same idea. One is a disk; one is a network filesystem that happens to be very fast because the network is a shared memory ring. NFS is the closer cousin to virtiofs than virtio-blk ever was, minus the network and plus a lot of engineering to make the metadata operations cheap.

The real question: who owns the filesystem

Draw the trust boundary and ask which side the filesystem implementation sits on. With virtio-blk it is entirely inside the guest. A malicious guest that corrupts its own ext4 has corrupted a file on the host and nothing more; the host never parses it. Path traversal is meaningless because there are no paths on the wire. Permissions are the guest's business. If the guest wants to fill the disk with garbage, the ceiling is the image size.

With virtiofs the implementation is on the host, in virtiofsd, and the guest supplies the input. Now the host is the one resolving names, following (or refusing to follow) symlinks, mapping guest uid/gid onto host uid/gid, deciding whether "../.." escapes the shared root. virtiofsd does this carefully — it pins the shared directory, works from file descriptors rather than path strings, uses resolve-beneath openat flags — and that care is exactly the point. It has to be careful because it is a parser standing on the boundary. A block device has no equivalent, because there is nothing to parse.

virtio-blk asks the host to move bytes at an offset. virtiofs asks the host to interpret a filesystem request from an untrusted guest. The second is strictly more powerful and strictly more attack surface.

That framing also explains something people find odd: Firecracker does not have virtio-fs at all. Its device model is block, net, vsock, rng, balloon, a serial console and a one-button keyboard controller. There is no filesystem passthrough, no PCI, no graphics, no shared memory region for the DAX window virtiofs uses for its best performance mode. That is a design tenet, not a backlog item — the host attack surface is roughly the sum of the device emulators, and a full filesystem server is a large addition to that sum. The practical consequence shows up in Kata Containers: on QEMU or Cloud Hypervisor it can share the container rootfs over virtiofs, and on Firecracker it cannot, so it uses the devicemapper snapshotter and gives each container a block device instead.

# The virtio-blk side of the boundary, in full. Firecracker's API is JSON over
# a Unix socket; a disk is a file path plus three booleans.
curl --unix-socket /run/fc.sock -X PUT 'http://localhost/drives/rootfs' \
  -H 'Content-Type: application/json' -d '{
    "drive_id": "rootfs",
    "path_on_host": "/var/lib/pandastack/vms/abc123/clone.ext4",
    "is_root_device": true,
    "is_read_only": false,
    "cache_type": "Unsafe"
  }'

# There is no host-side daemon in that picture. Firecracker opens the file,
# services virtqueue descriptors with pread/pwrite, and that is the whole
# storage stack. Nothing on the host ever looks at a path the guest chose.

# The virtiofs side, on a VMM that supports it (QEMU / Cloud Hypervisor):
virtiofsd \
  --socket-path=/run/vfs.sock \
  --shared-dir=/srv/models \
  --cache=auto \
  --sandbox=namespace \
  --xattr &

qemu-system-x86_64 ... \
  -chardev socket,id=vfs0,path=/run/vfs.sock \
  -device vhost-user-fs-pci,queue-size=1024,chardev=vfs0,tag=models \
  -object memory-backend-memfd,id=mem,size=4G,share=on -numa node,memdev=mem

# Then, in the guest:  mount -t virtiofs models /mnt/models
#
# Count the moving parts. A long-lived host process. A shared memory backend
# (required -- the vhost-user protocol needs the guest's RAM mappable). A tag
# negotiated between two config files. And a host directory that is now part
# of the guest's runtime contract.

The fork argument: copy-on-write needs a block image

Here is the property I care about most, and the reason our rootfs is a local block image rather than a shared directory. A block image is one file, so cloning it is a single operation the filesystem or the kernel can make O(metadata) instead of O(bytes).

On a reflink-capable filesystem — XFS with reflink, btrfs, ext4 with reflink — the FICLONE ioctl produces a second file that shares every extent with the first and diverges only where somebody writes. It is one ioctl, it is sub-millisecond, and it does not care whether the image is 2 GB or 10 GB. Where reflink is not available we build a device-mapper snapshot instead: a shared read-only loop device over the template's disk, a sparse per-sandbox copy-on-write file, and a dm target that stitches them together at 4 KB chunk granularity. The per-sandbox file starts at zero bytes allocated and grows only with writes. Same shape, different kernel primitive.

That is what makes create cheap. In the 179ms budget for a sandbox create, the rootfs clone is roughly 4ms. It is also what makes fork work: the agent pauses the parent so its block I/O settles, takes a consistent copy of its disk image, and clones that one file to each child. Same-host lands in 400–750ms, cross-host in 1.2–3.5s, and the difference between those two numbers is network rather than technique. Worth stating plainly because we got it wrong in our own docs once: that fork captures on-disk state, not RAM, so an unflushed write in the parent is not in the child.

Now try to do that with virtiofs. The guest's storage is a host directory. To give a forked child its own copy you have to copy — or reflink — every file in it, one at a time, and then you have two directories whose relationship the kernel does not track. There is no single object to clone. A thousand-file node_modules is a thousand operations instead of one, and a running fork tree becomes a directory tree you are manually diffing. The O(metadata) property is not a virtio-blk optimization; it exists because the unit of storage is a single file.

# Cloning a block image: one syscall, size-independent.
# (FICLONE = _IOW(0x94, 9, int) = 0x40049409 -- this is the actual ioctl our
#  agent issues before falling back to copy_file_range and then plain copy.)
cp --reflink=always /var/lib/pandastack/templates/base/clone.ext4 \
                    /var/lib/pandastack/vms/abc123/clone.ext4
# ~4ms for a multi-GB image. du says the child costs almost nothing until
# the guest writes; df agrees, because the extents are shared.

# No reflink on this filesystem? Same idea, device-mapper instead. One shared
# read-only origin per template, one sparse CoW file per sandbox.
losetup --read-only --find --show /var/lib/pandastack/templates/base/clone.ext4
truncate -s 1G /var/lib/pandastack/vms/abc123/cow.img   # 0 bytes allocated
losetup --find --show /var/lib/pandastack/vms/abc123/cow.img
sectors=$(blockdev --getsz /dev/loop0)
echo "0 $sectors snapshot /dev/loop0 /dev/loop1 P 8" | \
  dmsetup create pdssnap-abc123          # 8 sectors = 4 KB chunks = ext4 block

# Now the equivalent for a shared host directory. There isn't one.
# The best you can do is walk it:
cp -a --reflink=auto /srv/app /srv/app-fork-abc123   # O(files), not O(1)
# ...and you now own the divergence bookkeeping the kernel was doing for free.

The thing everyone conflates: we stream memory, not the disk

This confusion comes up constantly, usually phrased as "but you already stream the filesystem from object storage, so why not virtiofs?" We do not. We stream memory.

When a sandbox restores from a baked snapshot, the guest RAM image can be several gigabytes, and downloading it before the VM can start would put a multi-GB fetch on the critical path of a 179ms create. So we do not download it. The agent registers a userfaultfd handler and hands Firecracker the file descriptor plus the region mappings. The guest touches a page, the kernel raises a fault, the handler maps that fault to an offset in the snapshot file, fetches a 4 MiB chunk with an HTTP Range GET from object storage, and installs it. A baked header records which chunks are entirely zero so those are filled without a fetch at all, a prefetch trace replays the hot chunk set in the background, and a shared on-disk chunk cache keyed by the object's hash means the first restore on a host pays the network and every later one reads local disk.

The rootfs does none of that. It stays a local file, always, because copy-on-write needs a local block device: reflink is a filesystem operation on a local inode, and dm-snapshot needs a block device to layer over. So the artifact model has two very different layers — memory paged in on demand from object storage, disk cloned locally in O(metadata) — and people collapse them into one "streaming" story that does not exist.

For honesty's sake: we did build a demand-paged rootfs. It serves the template's disk over an NBD device backed by ranged reads from object storage, sitting underneath the same dm-snapshot layer so everything above it is unchanged. It is elegant. It also corrupted guest rootfs images in production in June, its gating end-to-end test has still never passed, and the flag that enables it is now pinned off fleet-wide through configuration management rather than left to a boot script, because we found one host that had drifted into having it on. Streaming a disk is genuinely harder than streaming memory: a page fault can be retried, a block read that returns wrong bytes becomes a filesystem that lies to you.

When virtiofs is the right answer

It would be dishonest to write this as a one-sided comparison. There are workloads where a shared directory is not a compromise, it is the correct design, and forcing them into block images is the wrong call.

The strongest case is a large read-only dataset shared by many guests. Forty gigabytes of model weights, or a corpus, or a monorepo checkout that every build needs. With block devices, each guest gets an image; even with reflink you are managing per-guest images and per-guest page cache, and the copies are only free while nobody writes. With virtiofs, one host directory serves all of them, and when the DAX window is in play the guests map the host page cache directly, so one physical copy of those weights backs every VM on the box. That is a real, large win that block devices cannot match. Verify your VMM's current DAX status before you plan capacity around it — it has been the least settled part of the stack for a long time.

The second case is the developer loop. You want to edit a file on your laptop and have the process inside the VM see it on the next save, with no sync step, no rebuild, no upload. That is precisely what a shared directory is for, and it is why every desktop VM tool ships some form of it. If your VM is a long-lived development environment owned by one trusted person, a shared mount is a much better experience than any file-copy API, including ours.

A third, narrower case: when the host genuinely needs to read what the guest wrote, as files, immediately. With a block image the host sees an opaque ext4 blob and has to mount it — which means the host kernel parsing a filesystem the guest controlled, a thing with its own ugly CVE history. If your architecture requires host-side access to guest output, a shared directory is at least explicit about the coupling instead of pretending it isn't there.

When it is a trap

• Anything you want to fork or snapshot. The mount's state is split between the guest, the host daemon and the host filesystem, and only the first of those is in a VM snapshot. You can restore the machine; you cannot restore its storage into an independent copy.
• Multi-tenant workloads. A shared host directory is a shared mutable surface. Two guests on the same mount can see each other's writes, race each other, exhaust inodes, and fill the host filesystem — which is not a per-guest quota unless you built one.
• Anything where the host filesystem's semantics leak. Inode numbers, hard links, xattrs, case sensitivity, mmap coherency, uid/gid mapping. These now form part of the guest's runtime contract, and they change when you move to a different host filesystem.
• Density under crash. A block image is either consistent or it is not, and fsck is a known procedure. A half-written shared directory is a partial state across many host files with no journal spanning them.
• Cold-start budgets that assume a host daemon is already running. virtiofsd is a process per VM (or per mount) that has to start, connect and negotiate before the guest can mount. That is fine at 30-second boots and awkward at 179ms.

The multi-tenant point deserves its own sentence because it is the one people wave away. If you hand untrusted code a mount of a host directory, you have moved the isolation question from "can the guest escape the VM" — a hypervisor question with a small, audited surface — to "is the host-side filesystem server correct about every path, symlink, and permission decision the guest can ask it to make". Both can be answered well. Only one of them stays answered when you add a feature.

What we do instead: block devices all the way down

So the design is block at every layer, and the interesting part is what replaces each virtiofs use case.

Sharing a read-only base across many guests: the dm-snapshot path already does this. One read-only loop device over the template's disk is shared by every concurrent create of that template — one copy in host page cache, N guests reading it, each with its own sparse CoW file on top. The sharing is at block granularity rather than file granularity, which is less flexible than virtiofs and completely invisible to the guest, which is the point.

Durable state: a named volume is a per-workspace ext4 image that appears as /dev/vdb, /dev/vdc and so on. Volumes are namespaced per workspace, so no cross-tenant path exists to begin with. Cache mode is where this gets interesting — Firecracker defaults a drive to "Unsafe", which does not honour guest flushes, so a guest fsync returns success while the data sits in the host page cache. We keep the rootfs on Unsafe deliberately, since it is copy-on-write and thrown away with the sandbox, and we force volumes to Writeback, because a managed Postgres keeps its data directory there and a WAL fsync that lies is a data-loss bug wearing a performance costume. That flag is also a snapshot property: the drive PATCH API carries only drive_id, path_on_host and a rate limiter, so changing cache mode means re-baking the template, not restarting a VM.

Moving files in and out: an authenticated API, not a mount. Reads and writes go over the control channel into the guest, so the file transfer is a request with an identity attached rather than an ambient directory both sides can touch. It is less convenient than a shared folder. It is also the reason there is no host path anywhere in a sandbox's threat model.

import os
from pandastack import Sandbox
# export PANDASTACK_API_KEY=pds_...
assert os.environ.get("PANDASTACK_API_KEY")

# Files cross the boundary as authenticated API calls, not as a shared mount.
sbx = Sandbox.create(template="base")
sbx.filesystem.write("/app/main.py", "print('hello from a block device')\n")
sbx.filesystem.upload("./requirements.txt", "/app/requirements.txt")

print(sbx.exec("python3 /app/main.py").stdout)
for entry in sbx.filesystem.listdir("/app"):
    print(entry.path, entry.size)

# Durable state is a block device too: a named ext4 image that shows up as
# /dev/vdb inside the guest and outlives the sandbox that mounted it.
sbx2 = Sandbox.create(
    template="base",
    volumes=[{"name": "build-cache", "read_only": False}],
)
print(sbx2.exec("lsblk -o NAME,SIZE,MOUNTPOINT").stdout)

# The fork below is O(metadata) BECAUSE the disk is a single image: pause the
# parent, clone one file, boot the child. No directory walk anywhere in it.
# Note the semantics: this captures ON-DISK state, so sync before you fork.
sbx.exec("sync")
child = sbx.fork()
print(child.id, "forked from", sbx.id)

for s in (child, sbx2, sbx):
    s.kill()

A decision rule you can actually apply

Ask three questions in order, and stop at the first one that answers.

1. Will you clone, fork, snapshot or migrate this machine? If yes, the storage must be a block image. Everything else is a per-file copy problem you will end up writing yourself, badly.
2. Is the workload untrusted or multi-tenant? If yes, do not put a host-side filesystem server on the boundary. The block device's poverty of expression is a security property.
3. Is it a big read-only dataset, or a single trusted developer's edit loop? Then virtiofs, and take the DAX page-cache sharing if your VMM offers it — this is the case where it is not a compromise at all.

Most platform-shaped workloads answer yes at question one or two. Most workstation-shaped and inference-shaped workloads reach question three. The mistake is not choosing either technology; it is choosing one because your tooling defaulted to it, then discovering at question one that the thing you most wanted — cheap clones — was foreclosed months ago.

The summary

virtio-blk moves blocks. The filesystem lives in the guest, the host sees one opaque file, and because storage is one file it clones in a single ioctl — which is what buys a 4ms rootfs clone inside a 179ms create and a 400–750ms same-host fork. virtiofs moves filesystem operations. The filesystem lives on the host, in a daemon parsing guest-controlled requests, and in exchange you get live sharing, one physical copy of a big read-only dataset, and an edit loop with no sync step.

Firecracker only offers the first one, and after building on it for a while I think that constraint pushed us somewhere better than free choice would have. It forced the shared-read-only case to be solved at block level, forced durable state into explicit devices with explicit cache semantics, and left the host with no code path that interprets a filename the guest picked.

And keep the layers straight, because this is where the conversation usually goes wrong: memory streams on demand from object storage, in 4 MiB chunks, through a page-fault handler. The disk stays local, always, because copy-on-write needs a local block device. Two mechanisms, two artifacts, one word — "streaming" — that hides the difference.