The OOM Killer and Guest Memory in Firecracker

Ask what happens when a Firecracker microVM runs out of memory and you'll get two different, both-correct answers depending on which memory you mean. There are two reapers, at two levels, and they have very different blast radii. Inside the microVM, the guest kernel runs its own OOM killer: exhaust the RAM you configured for the guest and a process inside the VM gets a SIGKILL, while the VM itself keeps running. One level up, the host kernel sees the whole Firecracker VMM as just another process — and if the host is under memory pressure, the host OOM killer can reap that entire process, taking the whole microVM down at once. This post walks both reapers: how guest memory is configured, what each OOM killer actually does, how overcommit and lazy page-in turn a quiet host into a hungry one, and how to size and treat your sandboxes so that when a reaper does come, you survive it.

Two levels of memory, two OOM killers

The single fact that untangles all of this: a microVM's guest RAM is a fixed number you hand Firecracker at boot, and that number is a boundary the guest kernel enforces on itself. But the physical pages backing that RAM live in the host's address space, inside the Firecracker process, and the host kernel enforces its own boundaries on that process. So there are two independent memory systems, each with its own out-of-memory story.

• Inside the guest — the guest kernel manages mem_size_mib worth of RAM. When guest processes allocate more than that, the guest's OOM killer picks a victim process (highest oom_score) and SIGKILLs it. The kernel, the other processes, and the VM all survive. This is the ordinary Linux OOM killer you already know, just running inside a VM.
• On the host — the Firecracker process (the VMM plus its vCPU threads) is, to the host kernel, one process holding a large anonymous memory mapping. If the host runs low on physical RAM, the host OOM killer can select the Firecracker process as its victim and SIGKILL it. That kills the VMM, which kills the guest — every process inside, the guest kernel, the whole sandbox — in one shot.

How guest memory is configured: mem_size_mib

A guest gets exactly the RAM you tell Firecracker to give it, set once via the machine-config API before the VM boots. There is no dynamic resize: the guest sees this much memory for its whole life (barring the balloon device reclaiming slack, which we'll get to). This is the ceiling the guest kernel polices.

# PUT /machine-config over Firecracker's Unix-socket API,
# before InstanceStart. mem_size_mib is the guest's total RAM.
curl --unix-socket /run/fc.sock -X PUT 'http://localhost/machine-config' \
  -d '{
    "vcpu_count": 2,
    "mem_size_mib": 512,
    "smt": false,
    "track_dirty_pages": true
  }'

# The guest now believes it has 512 MiB, full stop. Allocate past
# that inside the guest and the *guest* OOM killer fires — the VM
# survives, one guest process gets SIGKILLed.

Note the value above is a configuration example, not a benchmark — pick the number to fit your workload. The important property is that it's a hard boundary from the guest's side: the guest kernel will never let its processes collectively hold more than mem_size_mib of resident memory. When they try, the guest OOM killer intervenes. What the guest cannot see or influence is whether the host actually has that physical RAM free at any given moment — which is the crack the host OOM killer lives in.

The guest OOM killer: one process dies, the VM lives

When a workload inside the microVM allocates past the guest's configured RAM, you're in ordinary Linux OOM territory — the guest kernel's OOM killer scans the guest's processes, scores each by memory footprint and oom_score_adj, and SIGKILLs the fattest offender to reclaim its pages. The classic symptom is a build or a data job that swells until it trips the killer. Here's the shape of it, driven from the PandaStack SDK so you can see the guest reaper fire in isolation:

from pandastack import Sandbox

# The template's RAM is baked into its snapshot (say ~512 MiB here).
# We deliberately allocate well past it *inside the guest*.
with Sandbox.create(template="code-interpreter") as sbx:
    # Python that keeps appending ~50 MiB chunks until the guest
    # kernel's OOM killer reaps THIS process. The VM stays up.
    hog = (
        "buf = []\n"
        "while True:\n"
        "    buf.append(bytearray(50 * 1024 * 1024))  # 50 MiB\n"
    )
    r = sbx.exec(f"python3 -c '{hog}'")
    print(r.exit_code)   # non-zero: process killed (often -9 / 137)
    print(r.stderr)      # frequently empty — SIGKILL leaves no traceback

    # Crucial: the sandbox itself is still alive and usable.
    alive = sbx.exec("echo still-here && free -m | awk '/Mem:/{print $2}'")
    print(alive.stdout)  # "still-here\n<total MiB>" — VM survived the reaping

The tell is the exit code and the silence. A process reaped by SIGKILL doesn't get to run a handler, flush a log, or print a traceback — it's terminated mid-instruction, which is why the guest process often exits with 137 (128 + 9) and empty stderr. If you see a worker vanish with no error, no stack trace, and a 137, the guest OOM killer is the prime suspect. Check the guest kernel ring buffer (dmesg) for the "Out of memory: Killed process" line naming the victim. The reassuring part: your sandbox is still there. You can exec into it, read logs, retry — because the guest OOM killer only removed a process, not the machine.

The host OOM killer: the whole sandbox dies at once

Now the reaper with the wider blade. The host kernel doesn't know or care that the Firecracker process is a hypervisor — it's a process holding a big anonymous mapping, same as any other. When the host's physical RAM runs short and there's nothing left to reclaim, the host OOM killer runs, scores every process on the box, and SIGKILLs one to free memory. If it picks a Firecracker process, that VMM dies instantly, and with it the guest kernel and every process inside the microVM. There is no in-guest OOM event, no graceful degradation, no chance for the guest to shed load — the whole sandbox simply ceases to exist between one instant and the next.

This is the failure mode that surprises people, because from inside the guest nothing was wrong: the workload might have been using half its configured RAM, well-behaved, nowhere near its own limit. It died because a neighbor got greedy and the host had oversubscribed memory. The host OOM killer is impartial and blunt — it optimizes for freeing the most memory to save the box, not for fairness to any one tenant. It's a SIGKILL delivered by a kernel that does not negotiate, does not ask, and does not warn.

• Guest OOM killer — victim: one process inside the VM. Blast radius: that process only; the guest kernel and VM survive. Trigger: guest processes exceed mem_size_mib. Recoverable in-place: yes — exec back in and retry.
• Host OOM killer — victim: the whole Firecracker VMM process. Blast radius: the entire microVM — guest kernel and every process, gone at once. Trigger: host physical RAM exhausted (usually from memory overcommit under load). Recoverable in-place: no — the sandbox is dead; you re-create it.

Why RSS creeps: overcommit and lazy page-in

The host OOM killer rarely fires because one VM asked for too much up front — it fires because a host's real memory usage crept past what's physical while nobody was watching. Two mechanics drive that creep, and both are features, not bugs.

The first is overcommit. Most microVMs boot, do a little work, and go idle holding far less RAM than they were configured for. So a platform packs more VMs onto a host than the sum of their mem_size_mib — betting that their combined working set (what they're actually touching at any instant) fits in physical RAM, even though the sum of their configured maximums doesn't. The bet usually pays. When it doesn't — when too many guests get busy and touch too many pages at once — the host runs out of physical RAM to back the promises it made, and the OOM killer collects.

The second is lazy page-in, and it's especially relevant to snapshot-restore. When Firecracker restores a snapshot, the guest's memory isn't all read into RAM up front — pages are backed lazily and faulted in on demand as the guest touches them (with UFFD streaming, faulted straight from object storage). So a freshly restored VM starts with a small resident set that grows as the guest works. The host's RSS for that process climbs over the first seconds or minutes of the guest's life, not at restore time. Which means a host can look comfortable right after a burst of creates and then tighten as those guests page in their working sets — the memory pressure arrives late, after the VMs are already running.

The balloon: reclaiming slack before the reaper does

There's a cooperative middle path between "let the host fill up" and "kill a VM": the virtio-balloon device. It gives the host a channel to ask an idle guest to hand some of its RAM back voluntarily. The guest's balloon driver inflates — pinning free guest pages and telling the host it won't use them — so the host can reclaim that physical memory and spend it on a guest that needs it, deflating later to give it back. Used well, ballooning drains the slack from quiet guests so the host never reaches the pressure point where the OOM killer has to choose a victim at all.

The catch is that ballooning is cooperative — it depends on the guest driver complying, so it's a soft optimization, not an enforced cap. And there's a direct interaction with the guest OOM killer worth knowing: the deflate_on_oom flag. With it on, if an over-reclaimed guest starts running out of memory, the balloon automatically deflates to give pages back before the guest's OOM killer fires — the balloon yields instead of your workload dying. It's the difference between the host quietly returning memory and a guest process getting SIGKILLed. For the full mechanics of inflate/deflate, stats, and the honest cooperative caveats, see the virtio-balloon writeup (/blog/firecracker-balloon-device).

Practical guidance: size, cap, and treat VMs as disposable

You can't talk the reapers out of their job, so the strategy is to make each kind of kill either unlikely or survivable. Three moves cover most of it.

• Size guest RAM to the workload. mem_size_mib too small and the guest OOM killer reaps your processes on ordinary spikes; too large and you either waste RAM or inflate the overcommit risk on the host. Right-size to the real peak working set — measure it, don't guess. When a template's builds OOM at one size, the fix is to bake a larger snapshot, not to hope. (PandaStack's base app template is baked at 4 GiB precisely because Next/Vite/tsc builds OOM'd at less.)
• Cap the VMM on the host with a cgroup. A memory.max on each Firecracker process bounds how much any single VM can consume, so one runaway guest can't drag the whole host into global memory pressure. A cgroup-local OOM kill takes out one VM deterministically instead of letting the global host OOM killer pick an arbitrary victim — you contain the blast radius to the offender.
• Treat every VM as disposable. Assume any sandbox can vanish at any instant — host OOM, host reboot, host replacement — and design so that it's recoverable rather than catastrophic. Keep durable state off the ephemeral rootfs (on a persistent volume or an external store), make work idempotent and retryable, and let a killed VM be re-created cheaply rather than mourned. If losing a VM means losing data, you've built the fragility in yourself.

How PandaStack handles both reapers

PandaStack leans hard on "disposable" being cheap. Every sandbox, managed database, and hosted app is its own Firecracker microVM, and RAM is the real binding constraint on how many pack onto a host (the networking layer pre-allocates 16,384 /30 subnets per agent, so per-VM network isolation is never the bottleneck — memory is). So the platform sizes memory per template rather than per request: an app's RAM is a property of its baked snapshot, and the agent honors the baked topology on create instead of trusting a per-create size that snapshot-restore couldn't change anyway. That's how a build-heavy template gets 4 GiB while a lightweight one runs leaner — you pick the tier by picking the template.

And because a create is a snapshot restore — roughly a 49ms restore landing at a 179ms p50 and about 203ms p99, versus the ~3s of a first cold boot — a sandbox lost to a host OOM kill is genuinely cheap to replace, not a long recovery. The whole system is built on that: sandboxes are treated as disposable, durable data lives on persistent volumes rather than ephemeral rootfs, and forks stay cheap too (a same-host fork is 400–750ms sharing the parent's memory copy-on-write; a cross-host fork is 1.2–3.5s once artifacts move). The one place we deliberately slow down and protect state is managed PostgreSQL, where create takes 30–90s to bootstrap and the data sits on a durable volume, precisely because a database is the thing you can't treat as disposable. PandaStack's core is open source under Apache-2.0, so you can run the control-plane API and per-host agent on your own KVM hosts, set your own cgroup caps, and watch both reapers yourself. For the cooperative reclaim layer that keeps the host reaper away, start at the virtio-balloon writeup (/blog/firecracker-balloon-device); for how vCPUs and RAM are baked into the snapshot, see the vCPU scheduling model (/blog/firecracker-vcpu-scheduling-model).