tmpfs in a microVM: The Filesystem That Eats Your RAM

Every Linux box ships with a directory that quietly lies about where your data lives. You write a file to `/tmp`, `df` reports a filesystem with a size and a used column, `ls` shows the file, and everything behaves like storage. It isn't storage. On most modern distributions `/tmp` is a tmpfs, so the bytes sit in the same physical memory your application allocates from — and nobody notices on a normal server, because a normal server has 64 GB of RAM and a build that writes 400 MB of scratch.

I'm Ajay; I built PandaStack, a Firecracker microVM platform, and the ratio flips completely once your machine is a microVM. Guest memory is a fixed allocation decided at boot — and in the snapshot-restore world, decided at bake time. There is no thin provisioning and no "we'll add a disk later." Every byte in tmpfs is a byte the workload doesn't get, and — the part that surprises people — every byte in tmpfs at snapshot time is copied into the snapshot's memory file and restored into every clone you make from it. This post is the mechanics: what tmpfs is, how it differs from ramfs and a ramdisk, why it beats virtio-blk for scratch, and how to size and clean it.

What tmpfs actually is (and what it isn't)

tmpfs is a filesystem implemented on top of the kernel's shared-memory subsystem, shmem. A file in tmpfs is a set of page-cache pages plus an inode in kernel memory, and that's the whole implementation — there is no backing block device, no journal, no writeback path, no `fsync()` that means anything. When you unlink the file, the pages go straight back to the allocator. When the machine reboots, the filesystem simply ceases to have ever existed.

Two properties define it. It has a size limit enforced at allocation time, defaulting to half of total RAM when mounted without a `size=` option. And its pages are swappable: given swap, the kernel can evict them under pressure exactly as it would anonymous memory. The bytes appear in `/proc/meminfo` under `Shmem`, counted as page cache even though — critically — the kernel cannot drop them, because there is nowhere to drop them to.

A stock Linux userspace is more memory-backed than most people realise. `/dev/shm` is tmpfs — where `shm_open()` and POSIX shared memory land, and what Python multiprocessing, Chromium, and much ML tooling use for big shared buffers. `/run` is tmpfs at a systemd default of 10% of RAM. `/tmp` is tmpfs on systemd distributions via `tmp.mount`, typically 50%. `/dev` is devtmpfs, a close cousin. `/sys/fs/cgroup` is memory-resident too, though be precise: under cgroup v1 it was a tmpfs with cgroupfs mounts hanging off it; under v2 it's `cgroup2fs`, a kernfs filesystem living in unswappable kernel memory rather than shmem.

ramfs and ramdisks: the two things tmpfs is not

ramfs is tmpfs's ancestor and meaningfully more dangerous. It has no size limit — you can pass `size=` and the option is ignored — and its pages never swap and are never reclaimed. Filling a ramfs mount doesn't give you `ENOSPC`; it gives you an out-of-memory kill, and on a swapless microVM it gives you one fast. The kernel keeps ramfs because early boot needs a filesystem that can't itself require memory management, not because you should mount it for scratch. If you find `ramfs` in a config file someone wrote in 2011, it's almost always a typo for `tmpfs` that nobody caught because the mount worked.

A ramdisk is a third thing: `/dev/ram*` from the `brd` driver is a fixed-size RAM-backed block device that you put a real filesystem on. That buys you two layers of caching, a journal doing real work for no reason, and space that isn't returned when you delete files, because the block device has no idea a file was deleted. For scratch in a VM it is strictly worse than tmpfs on every axis.

The microVM twist: those bytes are guest RAM

A Firecracker microVM's memory is a fixed allocation from its machine configuration at boot: the VMM maps a region of that size and the guest kernel sees exactly that much. You cannot grow it at runtime the way you'd grow a volume. Firecracker does expose a virtio-balloon device, but a balloon lets the guest hand memory *back* — it never lets the guest exceed its configured size, and virtio-mem hotplug depends on guest kernel support you may not have. At snapshot restore, vCPU count and memory size are properties of the snapshot, not negotiable at create time.

Take a 2 GiB guest, an ordinary size for a sandbox running a build. Default sizing gives `/tmp` a 1 GiB ceiling and `/dev/shm` a 1 GiB ceiling, and those ceilings overlap — both draw on the same 2 GiB. Write 900 MB of intermediates into `/tmp` and you haven't touched the rootfs at all; you've taken 900 MB from the compiler that is about to ask for it, and the build dies in the least informative way available.

# The symptom, in a 2 GiB guest. Note the two mutually contradictory
# stories the tools tell you.

$ python3 -c "open('/tmp/blob','wb').write(b'x' * 900 * 1024 * 1024)"
OSError: [Errno 28] No space left on device

$ df -h /tmp /dev/shm /
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.0G  1.0G     0K 100% /tmp        <-- "full"
tmpfs           1.0G     0  1.0G   0% /dev/shm
/dev/vda        9.8G  1.9G  7.4G  21% /            <-- disk is fine

$ du -sh /var /usr /opt
412M    /var
1.1G    /usr
8.0K    /opt

# ...and here is where the missing memory went. Shmem is the tmpfs bill.
$ grep -E 'MemTotal|MemAvailable|Shmem:' /proc/meminfo
MemTotal:        2027384 kB
MemAvailable:     118204 kB
Shmem:           1048576 kB

Two failure modes come out of this and they present very differently. Hit the tmpfs `size=` limit first and you get a clean `ENOSPC` — "No space left on device" on a device that does not exist, which sends people to check disk quotas and LVM for twenty minutes. If the limit is generous and you hit total memory first, the OOM killer arrives instead — and it does not kill whatever filled `/tmp`, because that process has probably exited. It kills the largest RSS, which is your application. Worse, the tmpfs contents survive the kill, so the restart walks straight back into the same wall. An OOM loop with an idle CPU and an empty disk is the signature.

The snapshot dimension: tmpfs lives inside your memory file

This part is specific to snapshot-based platforms, and I think it's genuinely underappreciated. A Firecracker snapshot is two artifacts: a state file describing devices and vCPUs, and a memory file that *is* the guest's RAM — not a summary of it, the pages. Whatever is in tmpfs at capture time is, by definition, in that memory file.

Three consequences follow, in ascending order of how much they should worry you. The memory file gets bigger, costing upload time, storage, and — if you page memory in on demand from object storage the way we do — more chunks to fetch on restore. The contents are then restored byte-for-byte into every VM created from that snapshot, so a half-finished `npm` cache in `/tmp` at bake time becomes a half-finished `npm` cache in ten thousand sandboxes. And the one that turns an inefficiency into an incident: a secret sitting in tmpfs at bake time is shared by every clone. A token in `/run`, a service-account key someone `curl`ed to `/tmp` during template setup, a session cookie in `/dev/shm` — identical across every VM forever. Same category as baking SSH host keys into an image, and harder to spot, because tmpfs is supposed to be where things go to disappear.

So the hygiene pass before baking a template is worth automating. The order matters more than it looks, and one step is subtler than it appears.

#!/usr/bin/env bash
# Run INSIDE the guest, immediately before capturing a template snapshot.
set -euo pipefail

# 1) Delete the scratch first. drop_caches will not do this for you --
#    tmpfs pages are shmem, so the kernel has nowhere to evict them to and
#    will hold them resident until the files are unlinked.
find /tmp     -mindepth 1 -delete 2>/dev/null || true
find /var/tmp -mindepth 1 -delete 2>/dev/null || true
find /dev/shm -mindepth 1 -delete 2>/dev/null || true

# 2) Per-instance junk in /run that a clone must not inherit. Anything with
#    an identity in the name deserves a second look here.
rm -f /run/*.pid /run/*.sock 2>/dev/null || true
rm -f /root/.bash_history /home/*/.bash_history 2>/dev/null || true

# 3) Now the page cache, which IS reclaimable: clean pages read from the
#    rootfs are re-readable from the block device, so paying to carry them
#    in the memory file is pure waste.
#
#    Caveat worth knowing: freeing a page does not zero it. Whether this
#    actually shrinks the snapshot depends on your pipeline being able to
#    tell a free page from a used one -- zero-page elision on capture, or
#    balloon free-page reporting, are the two mechanisms that make it pay.
sync
echo 3 > /proc/sys/vm/drop_caches

# 4) Blunt instrument: zero the freed pages so they elide/compress away
#    instead of riding along as stale garbage. This only reaches as far as
#    the tmpfs size limit allows, so it is a mitigation, not a guarantee.
#    The guarantee is not putting the secret there in the first place.
dd if=/dev/zero of=/tmp/zero bs=1M status=none 2>/dev/null || true
rm -f /tmp/zero
sync

Why tmpfs is still a very good idea in a microVM

Having spent four sections on the hazards, let me argue the other side honestly, because for short-lived sandboxes tmpfs is one of the highest-leverage single-line changes available.

A write to the rootfs does not stop at the guest kernel. It lands in the guest page cache, then at writeback or `fsync` becomes a virtio-blk request: the guest fills a descriptor in a shared ring, kicks the queue with an MMIO write that traps out to the VMM, Firecracker's device thread does a `pwrite` against the host backing file, and completion returns as an interrupt. That's a VM exit and a host syscall per batch, plus whatever the host filesystem does underneath — and on a copy-on-write rootfs the first write to a shared extent must allocate first. None of this is slow in absolute terms. It is just not free, and build tooling calls `fsync` a lot.

A write to tmpfs is a `memcpy` into a page-cache page: no device, no ring, no exit, no host syscall. `fsync` on tmpfs is a no-op that returns success, which is philosophically upsetting and operationally excellent. There's no journal either, so metadata-heavy work — creating and deleting tens of thousands of tiny files, exactly what `npm ci`, `cargo build`, `tsc`, and a pytest run all do — stops paying journal and writeback costs entirely. Point `TMPDIR` at tmpfs for a package install or a transcode with big intermediates and you remove a whole device layer from the hot path.

The trade is precise, and worth one sentence: you are swapping a resource you can overcommit for one you cannot. Hosts thin-provision disk, share extents, and let ten guests each believe they have 10 GB. RAM is a hard reservation — held whether used or not, and on a snapshot platform it's also what you pay to capture, store, and stream. tmpfs is fast because it is expensive.

The four places scratch can live

• tmpfs — Speed: memcpy-fast, no VM exits, `fsync` is a no-op. Capacity ceiling: the `size=` option, capped by guest RAM (default half). Snapshot impact: everything in it is captured in the memory file and cloned into every restore. Failure mode when full: clean `ENOSPC` on a filesystem with no device — or an OOM kill, if you sized it too generously.
• ramfs — Speed: identical to tmpfs; same mechanism minus the accounting. Capacity ceiling: none — `size=` is accepted and ignored. Snapshot impact: same as tmpfs, with no limit to bound it. Failure mode when full: no `ENOSPC` at all — the kernel allocates until the OOM killer intervenes. There is no case where ramfs is the right answer for scratch.
• Rootfs on virtio-blk — Speed: page-cached for reads and buffered writes, then a device round trip at writeback or `fsync`, plus journal work and copy-on-write allocation on first write to a shared extent. Capacity ceiling: the disk image — much larger than RAM, and overcommittable on the host. Snapshot impact: minimal on the memory file, though a disk-capturing snapshot carries whatever you left behind. Failure mode when full: ordinary `ENOSPC` everybody already knows how to debug.
• Attached durable volume — Speed: the same virtio-blk path plus whatever the storage layer costs; slowest of the four, and the only one that guarantees anything. Capacity ceiling: whatever you provisioned, resizable. Snapshot impact: none on guest memory — it lives outside the VM's RAM, which is the point. Failure mode when full: `ENOSPC`, with your data still there afterwards. Anything that must survive a restart, a hibernate, or a host move belongs here.

Configuring it in practice

Start by finding out what you actually have, because the defaults were chosen for a laptop, not for your guest.

# Everything memory-backed in this guest, with its ceiling and usage.
findmnt -t tmpfs -o TARGET,SIZE,USED,AVAIL,OPTIONS

# The two that actually bite in a sandbox.
df -h /tmp /dev/shm

# Resize live -- no unmount, no data loss. Shrinking below current usage is
# allowed: existing files stay, new allocations fail with ENOSPC.
mount -o remount,size=512M,nr_inodes=100k /tmp
mount -o remount,size=256M /dev/shm

# nr_inodes matters independently of size: a million 1-byte files exhaust
# inodes long before bytes, and the error you get is still ENOSPC.

# Persistent, with hardening flags for a guest running untrusted code.
# /etc/fstab
# tmpfs /tmp tmpfs rw,nosuid,nodev,noexec,size=512M,nr_inodes=100k,mode=1777 0 0

# On a systemd distro /tmp is owned by tmp.mount -- override it rather than
# fight it: /etc/systemd/system/tmp.mount.d/size.conf
# [Mount]
# Options=mode=1777,strictatime,nosuid,nodev,noexec,size=512M,nr_inodes=100k

# Per-service private /tmp and /var/tmp, via a mount namespace:
# [Service]
# PrivateTmp=yes

`size=` takes bytes with suffixes or a percentage (`size=25%`) of total RAM — so it moves if you rebake the template at a different memory size, which is either convenient or a landmine depending on whether you remembered. `nr_inodes` is the one people forget: tmpfs allocates an inode per file from a separate budget, so a tool creating a hundred thousand tiny lock files exhausts inodes while `df -h` still shows plenty of space. `df -i` is the tie-breaker.

The hardening triple — `nosuid,nodev,noexec` — is worth mounting with when the guest runs code you didn't write. `nosuid` and `nodev` are real: they close off setuid binaries dropped into a world-writable directory and device nodes conjured there. `noexec` deserves an honest caveat, because it is routinely oversold. It stops `execve()` on a file in that mount. It does not stop `python3 /tmp/thing.py`, because the interpreter is merely *reading* a file it has every right to read. It doesn't stop a shell sourcing a script, an ELF loaded by invoking the dynamic linker directly, or a payload that never touches the filesystem because it lives in a `memfd`. Mount `noexec` — it costs nothing and raises the price of lazy attacks — but don't write it into a threat model as a boundary. The boundary is the hypervisor.

Sizing the split, and the zram question

The sizing problem is: given a fixed pool of guest RAM, how much goes to the workload and how much to scratch? The honest method is measurement, in this order.

1. Measure peak scratch, not average: run the real workload with `TMPDIR` on a generously-sized tmpfs and sample `df /tmp` on a one-second loop, or watch `Shmem` in `/proc/meminfo`. Installers spike hard at unpack time and then release; the peak is what you must fit.
2. Measure peak process RSS separately, on a run where scratch went to disk so the two don't contaminate each other — `/usr/bin/time -v` reports maximum resident set size for free.
3. Add the two, then add slack for the guest kernel and a working page cache — not a rounding error on a small guest.
4. Set `size=` explicitly to the measured peak plus margin. The 50% default is not a decision, it's the absence of one — and an explicit ceiling converts a fatal OOM into a survivable `ENOSPC` with a stack trace pointing at the culprit.
5. Re-measure when dependencies change. A ceiling tuned against March's lockfile is a time bomb with a slow fuse.

Which brings us to zram, and I want to be fair to it. zram is a compressed block device living in RAM, usually configured as swap. Give a guest zram-backed swap and tmpfs pages become evictable: under pressure the kernel swaps them out, meaning it compresses them and stores them back in RAM. You get more effective scratch out of the same physical memory, and for compressible content — source trees, JSON, JS bundles, logs — the ratio can be genuinely good.

The other reading is that you're spending CPU to pretend you have memory you don't have. Every fault on a swapped-out page is a decompression on the guest's own vCPUs — the same ones running the build. If the workload is CPU-bound, or the content is already compressed (tarballs, images, model weights), you pay the CPU and get almost nothing back. My rule: zram earns its place when the workload is memory-bound with compressible scratch and the alternative is failing outright, and it's a bad trade when you're CPU-bound or the real answer is a larger guest. One snapshot note — a zram device is RAM too, so compressed swap contents land in the memory file as well.

On PandaStack there's a constraint that shapes all of this: the guest's memory size is a property of the baked template snapshot, not a per-sandbox knob. Firecracker cannot change vCPU count or memory size at snapshot restore, so a create request can't ask for more RAM than the snapshot was baked with — "just give it more memory" means rebaking a template. That makes the tmpfs ceiling a template-design decision, which is no bad thing: you make it once, deliberately, with measurements, instead of rediscovering it per environment at 2 a.m.

from pandastack import Sandbox

def mem_available_mb(sbx) -> int:
    r = sbx.exec("awk '/MemAvailable/ {print int($2/1024)}' /proc/meminfo",
                 timeout_seconds=15)
    return int(r.stdout.strip())

def tmpfs_used_mb(sbx) -> int:
    r = sbx.exec("df -m --output=used /tmp | tail -1", timeout_seconds=15)
    return int(r.stdout.strip())

with Sandbox.create(template="base", ttl_seconds=900) as sbx:
    # Pin the ceiling instead of inheriting "half of RAM", so a runaway
    # build hits ENOSPC with a traceback rather than the OOM killer.
    sbx.exec("mount -o remount,size=768M,nr_inodes=200k /tmp",
             timeout_seconds=15)

    sbx.exec("git clone --depth 1 https://github.com/example/app /srv/app",
             timeout_seconds=120)

    before = mem_available_mb(sbx)
    build = sbx.exec(
        "cd /srv/app && TMPDIR=/tmp npm ci --no-audit --no-fund",
        timeout_seconds=600,
    )
    after = mem_available_mb(sbx)

    print("exit:", build.exit_code)
    print(f"MemAvailable: {before} MB -> {after} MB")
    print("tmpfs held:", tmpfs_used_mb(sbx), "MB")

    # The point of the delta: scratch space came out of the same pool the
    # build's own heap draws from. Disk usage barely moved. If you plan to
    # snapshot this sandbox, clear /tmp first -- otherwise that scratch is
    # about to become a permanent resident of the memory file.
    sbx.exec("find /tmp -mindepth 1 -delete", timeout_seconds=30)

When this is overkill

Most services should not be doing any of this. A long-lived web app, an API server, a queue worker — modest scratch, a lifetime measured in weeks — should put scratch on the disk and move on. The disk is overcommittable, its failure mode is one your whole team already knows how to debug, and the write path you'd optimise away is a rounding error next to what the service spends on the network.

There's a subtler version of "overkill" worth naming. If your scratch working set already fits comfortably in the guest's page cache, the rootfs is *already* giving you memory-speed writes — you only pay the device cost at writeback and `fsync`. Moving that data to tmpfs converts reclaimable page cache into unreclaimable shmem and buys you little. The wins concentrate where the workload is metadata-heavy and `fsync`-happy: package installs, compilers with big intermediate trees, test suites creating thousands of temp files. If you can't measure a build getting faster, you've just made your OOM risk worse for a slide.

And the obvious one: if the data has to survive anything — a restart, a hibernate, a host migration — tmpfs is wrong by construction, not by degree. Put it on a durable volume. The rule I'd offer is that tmpfs earns its keep for short-lived, metadata-heavy, throwaway work in a guest you sized on purpose. Everywhere else the disk is fine, and "the disk is fine" is an underrated engineering conclusion.