Guest page cache: why it bloats microVM snapshots

When you snapshot a Firecracker microVM, the memory image (`vm.mem`) is a byte-for-byte copy of the guest's RAM at that instant. That's the whole point — restore maps those bytes back and the guest wakes up exactly where it left off. But it also means the snapshot inherits everything the guest happened to have in memory, including a large pile of file data the Linux kernel cached from disk while the machine was running. That data is trivially reconstructable — it's still sitting on the rootfs the guest can read at any time — yet it ships inside the memory image anyway. This post is about why that happens, why it hurts density and streaming restore, and the one-line fix: drop the guest page cache before you bake.

What the guest page cache actually is

Linux never reads a file straight into your program and forgets it. Every time the guest reads a file — a shared library, a Python package, a config, anything on the rootfs — the kernel keeps those pages in the page cache, on the theory that something will read them again soon. That's not a copy off to the side; the page cache is real, live guest RAM. Boot a machine, `pip install` a stack, import a few modules, and you've filled hundreds of megabytes of memory with clean copies of files the guest already has on disk.

That's excellent behavior for a long-running machine — cache hits are why the second read is fast. But a template snapshot isn't a long-running machine. It's a frozen starting point you're going to restore thousands of times. And at freeze time, the page cache is indistinguishable from any other RAM: it's just pages, so it all goes into `vm.mem`.

Why this bloats the snapshot

Think about what's actually in a baked template's memory image. Some of it is genuinely live and irreplaceable — process heaps, the kernel's own structures, anything you can't get back from disk. But a big fraction is stuff you don't need to carry at all:

• Cached copies of shared libraries and interpreter binaries — already on the rootfs, re-readable on demand.
• Cached pages of installed packages read during setup (every import that touched a `.so` or `.py`).
• Clean file-backed pages from config, locale data, and assets the boot process happened to open.
• Reclaimable slab — dentry and inode caches the kernel built up walking the filesystem.
• Zero pages — memory that was allocated but never written, holding nothing at all.

None of that is working set. It's the memory equivalent of packing a moving box with photocopies of books that are already on the shelf you're moving to. A bigger `vm.mem` is slower to write at bake time, slower to move around, and — the part that actually stings — slower to bring back on restore.

The fix: drop caches before you bake

Linux gives you a direct lever for this. `sync` flushes dirty pages to disk so nothing is lost, then writing to `/proc/sys/vm/drop_caches` tells the kernel to release reclaimable memory. The value `3` means "drop the page cache and the reclaimable slab (dentries + inodes)." Run it inside the guest, right before you take the snapshot, and the memory image collapses toward just the live working set.

# Run inside the guest, immediately before snapshotting.
# sync first so no dirty page is lost; then release clean cache + slab.
sync
echo 3 | sudo tee /proc/sys/vm/drop_caches

# 1 = page cache only, 2 = slab only, 3 = both. Use 3 for a bake.
# This frees clean, file-backed pages the guest can re-read from its rootfs.

In a PandaStack-style bake pipeline you'd do this over the same exec channel you use for everything else in the guest — create the sandbox, get it to the state you want to freeze, drop the cache, then snapshot. With the Python SDK it's a single `exec` before the snapshot call:

from pandastack import Sandbox

# Bring a sandbox to the state you want to freeze, then shrink its RAM
# footprint before snapshotting.
sbx = Sandbox.create(template="base", persistent=True)
try:
    # ... run setup: install deps, warm the app, whatever the template needs ...

    # Release clean page cache + reclaimable slab so the snapshot only
    # captures the live working set, not re-readable file data.
    drop = sbx.exec("sync; echo 3 | sudo tee /proc/sys/vm/drop_caches",
                    timeout_seconds=30)
    assert drop.exit_code == 0, drop.stderr

    # Now take the snapshot — its memory image is smaller and denser.
    snap = sbx.snapshot()
    print("baked:", snap.id)
finally:
    sbx.kill()

The tradeoff: first touch re-faults

This isn't free, and it's worth being honest about the cost. When you drop the page cache, the guest genuinely forgets those file pages. After restore, the first process that reads one of those files takes a page fault and the kernel re-reads it from the rootfs — a little slower on that first touch than if the page had still been warm in memory. You've traded a smaller, denser snapshot for a handful of cold reads early in the restored VM's life.

For a template you restore once and run for hours, that trade might not be worth it — the cache would have paid for itself. For a template you restore thousands of times, or stream on demand, it's a clear win: you were going to re-read most of that data from disk anyway, so paying to carry it in the memory image is pure overhead.

Why this matters more when you stream vm.mem

On PandaStack, restore doesn't always start by downloading the whole memory image. With userfaultfd (UFFD) memory streaming, the guest's `vm.mem` is paged in on demand from object storage: the guest touches a page, the kernel raises a fault, and a handler fetches just that chunk over a range request. This is what lets an agent boot without waiting for a multi-gigabyte download first — the restore itself is around 49ms and a full create lands at a p50 of 179ms (p99 ~203ms), versus roughly 3 seconds for a cold boot with no snapshot.

Streaming changes the economics of snapshot size completely. Every chunk in `vm.mem` is a potential fetch. If a big fraction of the image is cached file data, you're either paying to fetch chunks the guest will re-read from its local rootfs anyway, or those chunks sit there inflating the image for no reason. Dropping the cache before the bake means fewer non-zero chunks to fetch on restore — a smaller, working-set-only image is directly fewer round-trips to storage when the guest wakes up.

This composes with zero-page elision. A baked header records which chunks are actually non-zero; chunks that are all zeros are never fetched at all — they're just zero-filled locally. Dropping the page cache and reclaimable slab turns a lot of file-backed pages back into memory the guest simply won't touch, so between the two techniques the restore path fetches close to only the pages that carry real, live state.

Adjacent techniques: balloons and MADV_DONTNEED

Dropping caches is the bluntest instrument, but it's not the only one for reclaiming guest memory before a freeze. A virtio-balloon device lets the host ask the guest to give memory back: the balloon "inflates," the guest hands those pages to the driver, and the hypervisor can treat them as free — a snapshot taken with the balloon inflated captures fewer live pages. At a finer grain, a process (or the kernel on its behalf) can call `madvise(MADV_DONTNEED)` on a range to tell the kernel those pages aren't needed, letting them be reclaimed rather than preserved.

All three are the same idea from different angles: a snapshot faithfully captures whatever the guest is holding, so the highest-leverage optimization is making the guest hold less at the moment you press the button. `drop_caches` targets clean file cache and slab; balloons target free-able anonymous memory; `MADV_DONTNEED` targets specific ranges an application knows it's done with. For a template bake, the cache drop is the cheapest and highest-yield of the set — one command, no device model, no application changes.

The underlying joke is a good one to keep in mind: a snapshot full of cached files is you shipping photocopies of a book that's already on the destination shelf. The disk is right there in the restored VM. Drop the cache, let the guest re-read from disk on first touch, and spend your snapshot bytes — and your storage round-trips — on the state you actually can't get back any other way.