Swap and zram Inside a Firecracker MicroVM: What Actually Happens

Someone always asks it right after an out-of-memory incident: "can we just add swap inside the microVM?" It sounds like free headroom. It isn't, and inside a Firecracker guest it isn't even the thing you think it is. One structural fact gets forgotten the moment you log into a guest and it feels like a normal Linux box: the guest's RAM is not RAM. It's a file on the host, mapped into the Firecracker process's address space. So when the guest kernel decides to "page memory out to disk," it's writing host-file-backed memory into a virtual block device that is itself a host file — memory moving from one host file to another, with two page caches and two eviction algorithms arguing about it. I'm Ajay, I built PandaStack, which runs every sandbox, database, and hosted app as its own Firecracker microVM, so I've watched this go wrong at both levels. Here are the mechanics: why guest memory is a file, how double paging thrashes, why zram is usually the better answer, what it genuinely costs, and how it all interacts with snapshots, the balloon, and demand-paged restore.

Guest RAM is a file on the host

When Firecracker boots a microVM, it creates one large mapping in its own address space and hands the guest a view of it. The guest kernel builds page tables over that region and manages it exactly as if it were physical DRAM, because from inside there's no way to tell the difference. What the guest calls "physical address 0x4000" is an offset into a host mapping owned by a userspace process.

On snapshot restore this becomes explicitly file-shaped. A Firecracker snapshot is roughly a state file describing devices and vCPUs, plus a memory file that is a byte-for-byte image of guest RAM at capture. Restoring maps that file into the process — classically MAP_PRIVATE, so the guest gets a private, copy-on-write view: reads come straight from the file's shared page cache, and only a write copies that page for this VM alone. That's what makes restore-based creates cheap. Nothing is copied up front, pages fault in lazily as the guest touches them, and many microVMs restored from one template snapshot share the untouched pages instead of each holding a duplicate.

That last clause is where the trouble starts. The guest kernel has a complete memory manager — LRU lists, reclaim, writeback, a swap subsystem, an OOM killer — all written assuming its page frames are real, that RAM is fast and disk is slow, and that it alone decides which is which. Inside a microVM none of that holds: the host has its own memory manager making its own decisions about the same bytes, and neither can see the other.

Two-level memory and the double-paging problem

Now add guest swap: attach a virtio-blk device backed by a host file or volume, put a swap area on it, swapon inside the guest. The guest kernel is delighted — it believes it has an overflow tier, so it will happily let workloads allocate past guest RAM, evicting cold anonymous pages to "disk" under pressure.

Follow one page through that eviction. The guest decides page P is cold and swaps it out; guest-side that's a write to virtio-blk. Host-side, that write lands in a file, and unless it was opened O_DIRECT it goes through the host page cache first — so the host now holds a copy of P while the guest frees its original frame. The bytes occupy host memory twice during the transition, and the operation meant to relieve memory pressure just created some. Later the guest touches P again and reads it back, which is either a host page-cache hit (fast, but it means the host held the page in RAM the whole time, so you evicted nothing) or a real disk read (slow, and now a guest process stalls on I/O).

That's double paging — the two-level memory problem: two independent memory managers stacked, each with its own LRU and its own idea of what's hot, neither able to see the other. The pathologies follow directly.

• The host can evict what the guest thinks is hot. The host sees the memory file as one big mapping and applies its own reclaim policy, so a page the guest would never swap can be dropped anyway — the guest then faults on memory it believed was resident, an invisible stall it cannot account for.
• The guest can swap out something the host is already caching. You spend I/O and CPU to move a page from one part of host memory to another part of host memory.
• The host can cache what the guest just evicted. Swap-out writes land in host page cache, so the freshly evicted page stays resident — the guest lost it, the host kept the bytes, nobody freed anything.
• Writeback amplification. Guest-side flushing of swap triggers host-side flushing of the file: one logical eviction becomes two rounds of I/O, two dirty-page trackers, and two throttling mechanisms that interfere.
• Feedback into the same pool. On an overcommitted host, page cache from guest swapping raises host pressure, which raises host reclaim, which evicts guest memory, which causes more guest faults. The mechanism meant to relieve pressure becomes a source of it.

That's not an argument that swap is always wrong — it's an argument that swap is a latency decision disguised as a capacity decision. You're trading predictable, loud failure for unpredictable, quiet slowness. On a laptop that's a great trade; you'd rather your editor stutter than die. On a microVM serving requests against a timeout budget, or building an image, or running an agent's tool call, it's usually terrible, because a process that's orders of magnitude slower has already failed — it just hasn't told anyone yet.

Why zram usually beats a swap-backed block device

zram is the same idea with the disk removed. It creates a block device inside the guest backed by guest RAM, compressing everything written to it and decompressing on the way out. Point swap at it and "swapping out" means "compress this page and keep it, smaller, in RAM." No virtio-blk write, no host file, no host page cache copy, no disk I/O — the page never leaves the guest's memory region.

For a microVM that removes the entire double-paging pathology in one move. The compressed pool lives inside the guest's own memory, which is inside the one host mapping the host already tracks. No second host file for the same bytes, no second page cache, no second writeback path — the host's accounting stays exactly what it was: one mapping, one resident-set number. You've replaced an I/O problem with a CPU problem, and a fast compressor is dramatically cheaper than a fault to storage: a compute operation versus a device round trip. I won't quote a ratio, because it depends entirely on your data — which is the next section.

• Where evicted pages go — zram (in-guest compressed): stay in guest RAM, compressed. Swap file on virtio-blk: written out to a host file through a virtual block device.
• Host-side cost — zram: none beyond the guest's existing memory mapping; the host still sees one region. Swap file on virtio-blk: a second host file, plus host page cache holding copies of pages the guest just evicted.
• Double paging — zram: structurally impossible; there is no second storage tier for the host to also manage. Swap file on virtio-blk: the default outcome — two LRUs, two writeback paths, mutual interference.
• Cost you pay — zram: guest CPU cycles compressing and decompressing on the fault path. Swap file on virtio-blk: I/O latency, host page cache pressure, and writeback amplification.
• Failure mode under real pressure — zram: the pool stops compressing usefully and you reach OOM relatively quickly and legibly. Swap file on virtio-blk: a long thrash where everything is slow and nothing is clearly broken.
• Effective capacity gained — zram: bounded by how compressible your data is; incompressible data gains you nothing. Swap file on virtio-blk: bounded by the size of the file, but the capacity is a lie if you're actually touching those pages.
• Snapshot behavior — zram: the compressed pool is guest RAM, so it's captured in the memory snapshot and restores with the VM. Swap file on virtio-blk: swapped-out pages live in the disk image, so they restore only if that volume is captured and reattached.
• Best fit — zram: bursty peaks over compressible data (JSON, text, logs, source, HTML, sparse heaps). Swap file on virtio-blk: large, genuinely cold working sets on a VM you'd rather slow down than kill — rare in a microVM.

What zram actually costs (it is not free)

Two costs, and the second is the one people miss.

The first is CPU, spent inside the guest on the fault path. Every page into the pool gets compressed; every page out gets decompressed synchronously while a guest process waits. In a VM sized at a couple of vCPUs, that CPU competes with the workload that caused the memory pressure in the first place — if you're already CPU-bound, zram under heavy churn is paying compute you don't have to buy memory. Compressor choice matters: lz4 and zstd sit at different points on the speed-versus-ratio curve, and lz4 is the usual default for latency-sensitive work because you'd rather decompress fast than store small.

The second cost is conceptual: compressed pages are still guest RAM. zram conjures nothing. It takes pages costing N bytes and stores them at a fraction of N inside the same fixed guest memory the host is backing — your total budget is unchanged, you're just storing part of it more densely in exchange for CPU on access. A guest with a genuinely large, genuinely incompressible working set gains almost nothing: you fill the pool, the compressor fails to shrink anything meaningful, and you reach the guest OOM killer anyway, having burned CPU getting there. zram buys headroom against bursts of compressible data. It does not buy you a bigger machine.

Configuring zram inside the guest

The setup is small enough to fit in a snapshot-bake script or an init unit. Load the module with one device, pick an algorithm, size the device, make it swap at a priority above any disk swap, then tune reclaim so the kernel is actually willing to use it.

#!/usr/bin/env bash
# Configure zram as the guest's only swap device. Run inside the microVM.
set -euo pipefail

# One zram device. (num_devices=1 keeps /sys tidy and predictable.)
modprobe zram num_devices=1

# Pick a compressor. lz4 = fastest decompress, best for latency-sensitive
# work. zstd = better ratio, more CPU. Check what your kernel offers:
cat /sys/block/zram0/comp_algorithm   # available algos, [current] in brackets
echo lz4 > /sys/block/zram0/comp_algorithm

# Size the BACKING STORE, i.e. how much *uncompressed* data the pool may hold.
# This is not extra memory: compressed pages still live in guest RAM.
# Modest is correct. Half of guest RAM is a sane starting point.
MEM_KB=$(awk '/MemTotal/{print $2}' /proc/meminfo)
echo $(( MEM_KB * 1024 / 2 )) > /sys/block/zram0/disksize

mkswap /dev/zram0
# Higher priority than any disk swap, so the kernel reaches for RAM first.
swapon --priority 100 /dev/zram0

# Compressed swap is cheap relative to disk, so let reclaim actually use it.
# (The old 'swappiness=10' folklore is tuned for spinning rust, not zram.)
sysctl -w vm.swappiness=100
sysctl -w vm.page-cluster=0        # no readahead: zram is random-access RAM

swapon --show
free -m

Two of those lines deserve a note. `vm.swappiness=100` looks alarming if you learned swap tuning on hard drives, where the advice was to push it toward zero. That advice was about avoiding disk latency. With zram there is no disk, so biasing reclaim *toward* the compressed pool and away from dropping page cache is usually what you want. And `vm.page-cluster=0` disables swap readahead: readahead exists to amortize seek costs on real storage, and against a RAM-backed device it just wastes CPU decompressing pages nobody asked for.

Once it's live, the two files worth watching are `/proc/meminfo` and `/sys/block/zram0/mm_stat`. The second is the honest one — it tells you how much data went in, how much space it actually occupies, and how much total memory the pool is consuming including its own overhead.

#!/usr/bin/env bash
# Is zram earning its keep? Read the two sources of truth.
set -euo pipefail

# Guest-wide view: SwapTotal/SwapFree are the zram device.
grep -E 'MemTotal|MemAvailable|SwapTotal|SwapFree' /proc/meminfo

# mm_stat columns (kernel docs order):
#   orig_data_size  compr_data_size  mem_used_total  mem_limit
#   mem_used_max    same_pages       pages_compacted  huge_pages
read -r orig compr used limit maxused same compacted huge \
  < /sys/block/zram0/mm_stat

printf 'uncompressed stored : %s bytes\n' "$orig"
printf 'compressed size     : %s bytes\n' "$compr"
printf 'total RAM consumed  : %s bytes (incl. allocator overhead)\n' "$used"
printf 'same-page dedup     : %s pages\n' "$same"

# The number that matters is orig vs used. If those are close, your data
# is not compressible and zram is buying you CPU cost with no memory back.
# Then the answer is a bigger guest, not a bigger zram device.

Snapshots: where do the swapped pages go?

This is where the two approaches diverge sharply, and where a lot of surprising behavior comes from. A Firecracker snapshot captures guest RAM into a memory file plus the device/vCPU state. It does not, by itself, capture your block devices — those are separate files that have to travel with it.

• With zram: the compressed pool is guest RAM, so it is inside the memory snapshot. Restore and the pool comes back intact, still compressed, still swapped-in-place. The guest wakes up believing exactly what it believed at capture time. It also means your snapshot's memory image includes compressed data — which, being compressed, will not compress again, so a pool that's full at bake time is dead weight in every restore of that snapshot.
• With a swap file on virtio-blk: the swapped-out pages are not in guest RAM, so they are not in the memory snapshot. They're in the disk image. Restore the memory file without the exact matching disk state and the guest will eventually fault on a swap page and read whatever is at that offset now — which is a memory-corruption-grade bug that presents as inexplicable garbage, not a clean error.
• For copy-on-write forks: a fork sharing the parent's memory copy-on-write inherits a zram pool coherently, because it's just memory. But if both forks share a swap-backed block device, both guests believe they own that swap area exclusively and will write to the same offsets. Two kernels swapping into one swap area is not a supported configuration; it's a data-corruption generator.

The practical rule: if a microVM is going to be snapshotted, restored, or forked — which on a snapshot-native platform is all of them, every time — swap-on-block-device requires you to reason very carefully about which disk state travels with which memory state, and forks make it outright unsafe unless each fork gets its own copy-on-write swap volume. zram sidesteps every bit of that by never leaving RAM.

The balloon device and host-side overcommit

The virtio-balloon device is the host's cooperative channel for asking an idle guest to hand memory back: the guest's balloon driver inflates, pinning free guest pages and telling the host they won't be used, so the host can reclaim that physical memory for a guest that needs it. It is the correct mechanism for redistributing memory between VMs, and it is also on a collision course with guest swap.

Think about what happens when both are active. The balloon inflates to reclaim slack. That reduces the memory the guest has to work with. The guest, now under pressure, responds by swapping — evicting pages to make room for the balloon that is taking room away. If swap is disk-backed, that eviction writes into host page cache, consuming exactly the host memory the balloon was inflating to free. The balloon and the swap subsystem end up in a loop, each responding to pressure the other created, and the net effect can be *more* host memory used, not less, plus a pile of I/O nobody wanted.

The balloon's `deflate_on_oom` flag is the designed escape hatch: when an over-reclaimed guest starts running out of memory, the balloon gives pages back rather than letting the guest OOM killer fire. That's a good setting, and it's also a clearer statement of intent than swap ever is — it says the *host* should decide who gets memory, not the guest's local reclaim heuristics. If you have a balloon, you already have an elastic memory mechanism with global visibility. Adding guest swap underneath it adds a second, blind one.

There's also the overcommit angle. Platforms pack more VMs onto a host than the sum of their configured guest RAM, betting that combined working sets fit in physical memory. Guest swap actively worsens that bet in two ways: it encourages workloads to hold larger nominal footprints (because the guest thinks it has headroom), and it generates host page cache that competes with the very physical memory the bet depends on. zram is neutral to slightly helpful here — the guest's footprint stays bounded by its configured RAM, and denser storage inside that bound means a smaller share of the mapping is actually touched.

Swap versus UFFD demand-paged restore

There's one more interaction that's specific to how fast microVM platforms start VMs, and it's the one that surprised me most in practice. Demand-paged restore — the userfaultfd path, where the guest's memory file isn't fully present at restore time and pages are fetched on fault, potentially from object storage — depends entirely on the guest touching a *small* subset of its memory in the first moments of life. That's why a snapshot restore can complete in ~49ms and land a create at a 179ms p50: nobody is moving gigabytes, because the guest only needs a few megabytes to get going.

Now restore a guest that immediately swaps in a large working set. Every one of those swap-ins is a touch. Under UFFD each touch of an absent page is a fault the handler must satisfy by fetching a chunk — so a guest that wakes up and drags its whole heap back from swap converts lazy paging into eager paging, one expensive fault at a time, in the worst possible order. You built a system whose entire premise is "only fetch what you actually use," and then handed it a workload whose first act is to declare that it uses everything.

zram is much better behaved here for a subtle reason: a compressed pool that was captured in the snapshot is part of the memory image, so swapping a page back in is a read of memory the restore path already has to account for, and the prefetch/hot-set machinery can learn it like any other hot region. It's still work. But it's local work on a known region, not a scattered fetch storm against a remote object.

Practical guidance: right-size, fail fast, compress the burst

Everything above collapses into a short decision procedure.

1. Right-size the guest first. This is the boring answer and it is almost always the correct one. Measure the real peak resident working set — not the average, not the configured size, the peak — and give the guest enough RAM to hold it with margin. Swap of any kind is a mitigation for having gotten this wrong, and mitigations for sizing errors are much worse than just fixing the size.
2. Default to no swap at all. A microVM with no swap fails fast and legibly: the guest OOM killer picks a victim, SIGKILLs it, you get exit 137 with empty stderr and a named process in dmesg, and the VM stays up so you can go read that dmesg. That is a *good* failure. It's immediate, attributable, and doesn't take your latency with it on the way down.
3. If you want a safety margin, use zram — never a swap file on virtio-blk. In a microVM, disk-backed swap buys you double paging, host page cache pressure, snapshot-coherence hazards, and fork unsafety, in exchange for latency you can't predict. zram gives you a bounded, in-RAM, snapshot-coherent buffer for the same purpose.
4. Size the zram device modestly and prove it's working. Half of guest RAM is a reasonable start. Then read mm_stat under real load: if orig_data_size and mem_used_total are converging, your data isn't compressible and you're paying CPU for nothing — that's a signal to grow the guest, not the pool.
5. Match zram to workloads that are bursty and compressible. Text-shaped data — JSON payloads, logs, source trees, HTML, sparse or pointer-heavy heaps — compresses well, and a workload whose memory spikes briefly and then recedes is exactly what a compressed buffer is for. Media, already-compressed archives, encrypted blobs, and dense numeric arrays are not; for those, zram is a CPU tax with a memory-shaped label on it.
6. Set an explicit memory limit on the process that's actually growing. A cgroup limit or a runtime flag (JVM heap, Node's --max-old-space-size, a worker's own cap) turns "the VM degraded mysteriously" into "this process hit its ceiling and said so." Bounded failure inside the guest beats unbounded pressure every time.

Swap is not more memory. It is permission to be slow instead of permission to fail — and in a microVM, being slow is usually just failing with extra steps and a worse error message.

How PandaStack approaches guest memory

PandaStack's position on this is basically "right-size the template and make replacement cheap," which is less a philosophy than a consequence of how snapshot-restore works. A microVM's RAM is baked into its snapshot — Firecracker can't resize guest memory at restore — so memory is a property of the template you pick, not a knob on each create. When our app-hosting base template kept OOM-ing on Next and Vite and tsc builds, the fix wasn't to bolt swap into the guest and let those builds thrash; it was to re-bake the template at 4 GiB. Sizing errors get fixed by sizing.

The reason that's affordable is that a lost VM is cheap to replace. A create is a snapshot restore — roughly a 49ms restore step landing at a 179ms p50 and about 203ms p99, against the ~3s of a first-ever cold boot — so a guest that OOMs, fails loudly, and gets re-created is an inconvenience rather than an outage. Same-host forks are 400–750ms sharing the parent's memory copy-on-write, cross-host 1.2–3.5s. Per-VM isolation isn't the limiter either: the networking layer pre-allocates 16,384 /30 subnets per agent, so memory is the real binding constraint on density, which is precisely why we'd rather guests hold a bounded, honest footprint than a swap-inflated one. The one place we accept a slow, careful create is managed PostgreSQL at 30–90s, because that's the workload you can't treat as disposable — and it's also the workload where you least want a swap death spiral standing between a query and its answer.

If you want to add zram inside your own sandboxes, nothing stops you — it's a normal guest kernel feature, and baking it into a template snapshot means every restore comes up with the pool already configured. Here's the shape of it from the SDK, including the part most people skip: measuring whether it helped.

from pandastack import Sandbox

# Configure zram inside a guest, then prove whether it's earning its keep.
with Sandbox.create(template="code-interpreter", ttl_seconds=600) as sbx:
    setup = (
        "modprobe zram num_devices=1 && "
        "echo lz4 > /sys/block/zram0/comp_algorithm && "
        # Half of guest RAM. Compressed pages are STILL guest RAM.
        "echo $(( $(awk '/MemTotal/{print $2}' /proc/meminfo) * 512 )) "
        "  > /sys/block/zram0/disksize && "
        "mkswap /dev/zram0 && swapon --priority 100 /dev/zram0 && "
        "sysctl -w vm.swappiness=100 vm.page-cluster=0"
    )
    r = sbx.exec(setup, timeout_seconds=60)
    assert r.exit_code == 0, r.stderr

    # Run the workload that actually spikes. Then read the honest numbers.
    sbx.exec("python3 /work/ingest.py --batch big.json", timeout_seconds=600)

    stats = sbx.exec(
        "grep -E 'SwapTotal|SwapFree' /proc/meminfo; "
        "awk '{print \"orig=\" $1 \" used=\" $3}' /sys/block/zram0/mm_stat"
    )
    print(stats.stdout)
    # orig >> used  -> compressible, zram is buying real headroom.
    # orig ~= used  -> incompressible; you're paying CPU for nothing.
    #                  Move to a template with more baked RAM instead.

The last comment is the whole post in three lines. zram is a real tool with a real, narrow job: absorbing bursts of compressible memory inside a guest that is otherwise correctly sized. It is not a substitute for enough RAM, and disk-backed swap inside a microVM is mostly a way to convert a clear failure into an expensive mystery. PandaStack's core is open source under Apache-2.0, so you can run the control-plane API and per-host agent on your own KVM hosts and watch the memory behave — or misbehave — yourself. For the two-reaper picture of what happens when memory does run out, see the OOM killer writeup (/blog/firecracker-oom-killer-guest-memory).