Memory Oversubscription in MicroVM Fleets, Explained

Add up the configured RAM of every guest on a busy microVM host and you will often find a number larger than the host's physical memory. On the base template a PandaStack sandbox is configured with 4 GiB; pack enough of them and the sum sails past what the machine actually has installed. This is not a mistake or a leak — it's memory oversubscription, and it's deliberate. The whole thing works because most guests don't touch most of their pages most of the time, so the host can safely promise more RAM than it owns. This post is the focused explainer on that one idea: the mechanisms that make oversubscription safe-ish — lazy demand paging, copy-on-write page sharing, the balloon device, same-page merging, and swap as the backstop — and then the harder half, the risk, because oversubscription is a bet, and the bet can lose. It sits alongside our posts on overcommit mechanics, density economics, and page-cache sharing; this one is specifically about how far you can push the promise, and how not to push it off a cliff.

Oversubscription is airline overbooking for RAM

The cleanest mental model is the one airlines already live by. A plane has a fixed number of seats, but the airline sells more tickets than seats, wagering that a predictable fraction of passengers won't show. Most days that's free money — the plane fills, nobody's bumped, and the airline booked revenue against seats that would otherwise have flown empty. The bet only fails on the rare day everyone shows up, and then someone gets bumped at the gate.

Memory oversubscription is the same bet with RAM as the seats and guests as the passengers. A guest 'buys a ticket' for 4 GiB when you configure it, but it usually flies with a carry-on — a few hundred megabytes of actual working set. The host sells more 4 GiB tickets than it has 4 GiB seats, betting that guests, like passengers, are mostly idle relative to their reservation. It's profitable precisely because that bet usually holds. And it has the same failure mode: when every guest tries to occupy its full configured seat at once, the host runs out of physical memory and someone gets bumped — except the gate agent here is the Linux OOM killer, and being bumped means a VM gets terminated.

Mechanism 1: lazy demand paging — a page costs nothing until it's touched

The foundation of the whole trick is that configuring a guest with 4 GiB does not hand it 4 GiB of physical memory. It promises that up to 4 GiB of guest-physical addresses will resolve to something when the guest touches them. The host allocates a real physical page only at the moment the guest first reads or writes a given guest page. Until then the address is a promise the kernel hasn't had to keep, and the resident set — what the host is genuinely backing with RAM — tracks only the pages the guest has actually referenced.

Snapshot-restore makes this even more pronounced. Because PandaStack restores every sandbox from a baked snapshot rather than cold-booting, a restored guest doesn't eagerly slurp its entire memory image into RAM up front. With UFFD memory streaming, the guest's memory is paged in lazily: the guest touches an address, the kernel raises a userfaultfd event, a handler fetches the relevant 4 MiB chunk from object storage over an HTTP Range GET, installs it, and the guest continues. Pages the guest never touches are never fetched and never occupy host RAM. There's a further refinement — zero-elision — where a baked header records which chunks are non-zero, so a fault in an all-zero region is satisfied by zero-filling in place with no fetch and no stored bytes. A freshly-restored guest's address space is mostly zeroes, so a large share of its 'memory' costs nothing to serve.

For oversubscription, the consequence is direct: a guest doesn't draw down the host's physical RAM in proportion to its configured size. It draws down in proportion to its working set. That is what lets the summed configured RAM exceed physical RAM in the first place — the sum is a sum of ceilings, and the host is only ever paying for floors. The demand-paging internals are in /blog/userfaultfd-explained and /docs/internals/streaming-restore.

Mechanism 2: copy-on-write shared pages — one physical copy, many readers

Lazy paging keeps untouched pages from being resident. Copy-on-write keeps identical pages from being duplicated. When many guests restore or fork from the same template, their memory images start byte-for-byte identical — same guest kernel, same booted userland, same warmed runtime. PandaStack maps that shared image copy-on-write (MAP_PRIVATE over the shared snapshot file), which means every guest reads the common pages out of a single shared physical copy. The kernel breaks the sharing only when a guest writes: a write faults, the kernel allocates one fresh 4 KiB page, copies the original into it, and re-points just that guest's page table at the private copy. Everyone else keeps reading the original.

This matters for the oversubscription bet in a specific way: a shared page is not a promise the host has to keep separately for each guest. It's one physical page answering for all of them at once. So the more your guests share, the less real RAM your oversold promises actually draw down, and the safer the overbooking gets. A hundred guests restored from one template don't threaten to demand a hundred private copies of their common image — they demand one, plus whatever each has individually dirtied. The same copy-on-write idea backs the rootfs disk via XFS reflink; here it's applied to RAM pages. Forking leans on it hardest: a same-host fork lands in 400–750ms by mapping the parent's memory copy-on-write, while a cross-host fork runs 1.2–3.5s because the bytes have to travel first.

Mechanism 3: the virtio-balloon — reclaiming RAM a guest already got

Lazy paging and copy-on-write keep memory from being allocated. The virtio-balloon is the lever for taking back memory a guest already got but no longer needs. It's a paravirtual device: a driver inside the guest that the host can 'inflate.' On inflate, the guest driver allocates guest pages and hands them back to the host, telling the guest kernel this memory is spoken for — and the host is then free to reclaim those physical pages and give them to a guest that needs them now. Deflate the balloon and the guest gets them back.

The balloon exists because oversubscription is a statistical bet, and every bet wants a hedge. A guest that briefly ballooned up to run a heavy job and then went idle is holding physical pages it isn't using — dead weight against the host's memory budget. Inflating its balloon claws that RAM back cooperatively, without killing anyone, and redistributes it to a guest under pressure. It's not a guarantee: it depends on a working guest driver and a guest that has genuinely free memory to surrender, so it's a pressure-relief valve, not a magic wand. But it converts 'this guest over-reserved' from a permanent loss into a recoverable one, which is exactly the tool you want when you've promised more RAM than you have. The balloon vs. hotplug trade-offs live in /blog/firecracker-memory-hotplug-vs-ballooning.

Mechanisms 4 and 5: same-page merging and swap as the backstop

KSM: dedup you scan for, at a price

Copy-on-write from a shared snapshot gives you sharing for free because the pages started identical. KSM (kernel same-page merging) solves the harder, general version: find identical pages that were produced independently and merge them after the fact. A background kernel thread (ksmd) scans pages you've marked mergeable, hashes them, and collapses matches into one write-protected shared page — copy-on-write forking them back apart on write. It recovers memory that structural CoW can't, because it merges across guests with no common origin.

The trade-offs are why it's a deliberate choice, not a default. First, CPU: ksmd burns cycles continuously scanning whether or not there's much to merge — a standing tax in exchange for reclaimed RAM. Second, and more serious for a multi-tenant platform, security: merging identical pages across guests opens a timing side channel. A write to a merged page faults measurably slower than a write to an unshared one, so a hostile guest can time writes to infer whether another tenant holds a page with specific content — the basis of published cross-VM memory-deduplication attacks. A security-first platform that already gets large structural sharing from same-template restores may reasonably decline to pay either cost.

Swap: the backstop that trades latency for survival

Swap is the last line before the OOM killer. When the host is genuinely short of physical pages, it can page cold memory out to disk to free RAM for hot pages. For an oversubscribed host, a modest amount of swap is a shock absorber: a brief, correlated spike that would otherwise trigger the OOM killer can instead be absorbed by pushing the coldest pages to disk, buying time for the balloon to reclaim or for the spike to pass. The catch is that swap is orders of magnitude slower than RAM, so leaning on it hard turns a memory-pressure event into a latency event — swap thrash, where the host spends its time shuffling pages instead of running guests. Swap is the right backstop for absorbing rare, short spikes; it is a terrible primary strategy for a host that's chronically oversubscribed. It buys survival, not performance.

The five stances, side by side

Line up the ways a host can relate to its memory promises — from refusing to oversubscribe at all, up through each mechanism — against how each one saves memory and how each one fails:

• No oversubscription — How it saves memory: it doesn't; every guest's full configured RAM is reserved up front, so summed configured never exceeds physical. Failure mode: no memory-pressure failure — but density is poor and idle guests waste RAM they'll never touch. The safe, expensive baseline.
• Lazy demand paging + CoW — How it saves memory: guests are backed only for pages they actually touch, and identical pages across same-template guests share one physical copy. Failure mode: correlated working-set growth — if many guests fault in and dirty large regions at the same moment, the resident total climbs toward the configured sum and the slack evaporates. The core of the bet.
• Balloon reclaim — How it saves memory: the host inflates idle guests' balloons to claw back RAM they over-reserved and hands it to guests under pressure. Failure mode: cooperative and slow — it needs a working guest driver and genuinely-free guest memory; a guest that's actually using its RAM has nothing to give back, so the balloon can't rescue a true simultaneous peak. A hedge, not a guarantee.
• Swap backstop — How it saves memory: pages the coldest memory to disk to free physical RAM for hot pages during a spike. Failure mode: swap thrash — lean on it beyond brief spikes and the host spends its cycles shuffling pages to disk, collapsing throughput. Survival at the cost of latency.
• OOM killer (no backstop left) — How it saves memory: it doesn't save it, it reclaims it violently by terminating a process. Failure mode: a live guest is killed to keep the host alive. This is the airline bumping a passenger at the gate — the thing every mechanism above exists to avoid.

The hard part: bounding the bet so it doesn't lose

Everything above is machinery for tilting the odds. None of it repeals arithmetic: if enough guests genuinely need their full configured RAM at the same instant, no amount of cleverness conjures physical memory that isn't there. Oversubscription pays off precisely to the degree that guests' memory demand is uncorrelated and mostly below their ceiling. A fleet of independent, bursty, mostly-idle sandboxes is the ideal case — the peaks rarely align, so the host rides the average. A fleet that all runs the same memory-hungry job on the same cron tick is the adversarial case — the peaks align perfectly, and the average is a lie. So the discipline is entirely about bounding how much you oversubscribe and reserving room to survive the bad case.

1. Track actual RSS against configured, continuously. The oversubscription ratio you can safely run is a property of your real workload, not a guess. Watch the resident set of your VMM processes versus their configured sizes and let the measured gap set your ceiling.
2. Set a deliberate overcommit ratio with headroom. Don't oversubscribe to the theoretical maximum. Reserve a slice of host RAM that no guest promise is allowed to draw from, so a spike has somewhere to land before it hits swap or the OOM killer.
3. Keep the balloon and swap as graded fallbacks, not the plan. Use the balloon to reclaim over-reserved RAM cooperatively and a modest swap file to absorb brief spikes — but size the fleet so neither is load-bearing under normal operation.
4. Schedule by real free memory, not configured memory. This is the single most important lever: place new guests based on what the host actually has free right now, not on the sum of what its existing guests were configured to use.

That last point is where PandaStack's design does the work for you. The scheduler scores agents on free capacity — free CPU and free memory, updated on a heartbeat — not on how much configured RAM the guests already there added up to. An agent that's oversubscribed on paper but has ample genuinely-free memory scores well; one that's near its real ceiling scores poorly and gets skipped, and any agent whose heartbeat has gone stale is excluded entirely. Scheduling on the resident reality rather than the configured promise is what keeps the fleet-level bet honest: you never pile a new guest onto a host that's already close to calling its own promises in.

You can watch the exact gap the scheduler cares about. Configured size is the guest's ceiling; resident size is what the host is truly backing right now — and the difference across a host is your live oversubscription slack:

# Per-guest: configured RAM is the ceiling; RSS is what the host actually backs.
# The gap between them is the room oversubscription lives in.
$ ps -o rss,command -C firecracker --no-headers | awk '{print $1/1024 " MiB  " $2}'
312 MiB  /usr/bin/firecracker --api-sock ...   # a 4 GiB-configured guest, mostly idle

# Host-wide: MemAvailable is the honest free-memory number a scheduler should
# place against. Committed_AS over MemTotal is the oversubscription itself --
# fine while guests stay idle, dangerous the instant they don't.
$ grep -E 'MemTotal|MemAvailable|Committed_AS' /proc/meminfo
MemTotal:       32900120 kB
MemAvailable:   21044880 kB   # <-- schedule against THIS, not summed configured RAM
Committed_AS:   48120064 kB   # promised well past MemTotal: the overbooking, quantified

# Pseudo-scheduler: pick the host by real free capacity, never by configured sum.
#   score = 0.6 * free_cpu + 0.3 * free_mem_gb   (+ a streaming-restore bonus)
# A host oversubscribed on paper but with real headroom still scores well;
# one near its true ceiling scores low and is skipped.

Oversubscription is airline overbooking for RAM: profitable because passengers are usually no-shows, disastrous the day they all board at once. The mechanisms don't make the plane bigger — they just make the no-show rate higher and give you a way to reseat people before the gate agent starts bumping.

The takeaway: a bet you engineer, not a law you break

Memory oversubscription lets a microVM host carry guests whose summed configured RAM exceeds its physical memory, and it's safe-ish for concrete, stackable reasons. Lazy demand paging means a guest is backed only for the pages it actually touches — and under snapshot-restore with UFFD streaming, untouched pages aren't even fetched. Copy-on-write means identical guests from one template share a single physical copy of their common image, so the promise costs the host once, not once per guest. The balloon reclaims RAM a guest over-reserved. Same-page merging and swap sit further out as costlier fallbacks. Each mechanism widens the gap between what you promised and what you're paying, and oversubscription lives in that gap.

The discipline is never forgetting it's a wager. Track resident against configured, set an oversubscription ratio with real headroom, keep the balloon and swap as graded hedges rather than the plan, and — above all — schedule by free memory, not configured memory. PandaStack's scheduler already does that last part by scoring on free capacity per agent, and its snapshot-restore-plus-CoW-plus-streaming design keeps restored guests from eagerly consuming their full configured RAM, so the bet starts from a favorable position. But the OOM killer is real and correlated load is real, and the only oversubscription ratio you should trust is the one you measured on your own load. PandaStack's core is open source under Apache-2.0, so you can watch the RSS-versus-configured gap on your own hosts. For the overcommit accounting, see /blog/overcommit-microvm-density; for the sharing mechanics, /blog/page-cache-sharing-microvm-density; and for the economics all of it pays for, /blog/microvm-density-economics.