microVM Memory: Balloon vs Hotplug vs Re-Provision

"How do I give my microVM more RAM?" is one of the most common questions I get, and the honest answer surprises people: with Firecracker, you mostly don't — not to a running guest, anyway. On a full VM or a QEMU box you'd reach for memory hotplug and grow the guest live. Firecracker deliberately doesn't have that. What it does have is a virtio-balloon, which does the opposite direction of what people expect: it reclaims idle RAM, it doesn't grant more. And on a snapshot-restore platform there's a third fact that trips everyone up — the guest's RAM is frozen at boot/snapshot time and you cannot change it at restore. So there are really three levers here, they solve three different problems, and confusing them is why the RAM question feels harder than it should. Let me lay all three out.

Three levers, three different problems

When someone says "change the VM's memory," they could mean one of three genuinely different operations, and the confusion is that all three get filed under "memory management." It's worth naming them before we get into which ones Firecracker actually offers:

• Hotplug — add (or remove) physical memory to a running guest so the guest sees more total RAM than it booted with, live, no reboot. This is the "give it more" operation. It's common on full VMs and QEMU; Firecracker does not have it.
• Ballooning — inside a guest with a fixed maximum size, inflate a balloon to hand idle pages back to the host (or deflate to take them back). This is the "reclaim the slack" operation. It never raises the guest's ceiling — it just lets the host reclaim what a guest isn't using. Firecracker supports this.
• Re-provision — stop caring about resizing this VM and instead boot (or restore) a differently-sized one. This is the "compost it and grow a new one" operation, and on a microVM platform it's usually the right answer.

Why Firecracker has no memory hotplug (on purpose)

Memory hotplug isn't a feature Firecracker forgot to build — it's a feature it chose not to build, and the reasoning is the whole Firecracker philosophy in miniature. Firecracker's pitch is a minimal device model: a tiny, aggressively-audited set of emulated devices (a couple of virtio devices, a serial console, a handful of others) and nothing else. Every device you add is more code between the guest and the host, more attack surface, more that has to be correct for the security boundary to hold. Hotplug — wiring up ACPI memory devices, the machinery to online and offline memory regions in a running guest, the host-side plumbing to grow the guest's physical address space live — is a lot of complicated code for a capability that a serverless-microVM model doesn't really need.

And it doesn't need it because of how these platforms scale. A full VM is a pet: it's long-lived, expensive to create, and you resize it in place because throwing it away and rebuilding is painful. A microVM is cattle — cheap to create, disposable, gone in a heartbeat. When the workload needs more memory, you don't perform surgery on the running guest; you boot a bigger one. Hotplug optimizes for a world where the VM is precious and the create is slow. Firecracker optimizes for the opposite world, where the create is so cheap that resizing-by-replacement is simply easier and safer than resizing-in-place. Cutting hotplug keeps the device model small, and a small device model is a small attack surface — which is the entire reason you reached for a microVM instead of a container. (For the exact device list and what is and isn't supported, verify against the Firecracker docs — the surface is intentionally minimal and does shift between versions.)

You don't resize a microVM. You compost it and grow a new one.

What the balloon does — and the direction people get wrong

Firecracker does ship a virtio-balloon device, and because it's the one memory knob that exists, people reach for it expecting it to be the "give my guest more RAM" button. It is not. The balloon only ever operates below the guest's fixed maximum, and its natural job is reclaiming idle memory, not granting new memory.

The mechanism: a small balloon driver inside the guest can inflate — allocating and pinning free guest pages and telling the host "I'm not using these, you can have them back." The host reclaims that physical RAM and spends it on a busier guest. When the guest needs its memory again, the host deflates the balloon and the driver frees the pages back into the guest's allocator. Notice what never happens: the guest's total RAM ceiling never goes up. A guest that booted with 2 GiB always has a 2 GiB ceiling. The balloon just moves the free-memory boundary inside that fixed envelope — inflating hands slack to the host, deflating (up to the original size) takes it back. Deflating a fully-deflated balloon does not give the guest 3 GiB; there's no more where that came from.

That direction — reclaim, not grant — is exactly why the balloon matters for density. Most microVMs boot, do a little work, and go quiet holding far less than they reserved. By inflating idle guests' balloons the host reclaims that slack and overcommits: it can run more guests than it has physical RAM, as long as the combined working set of everything actually touching memory at any instant fits. It's the same bet an OS makes letting processes allocate more virtual memory than physical RAM — not everyone spends at once. (For a deep dive on the balloon's three knobs and its cooperative-honor-system caveats, see /blog/firecracker-balloon-device — this post is about the higher-level choice between the three levers.)

The snapshot-restore catch: RAM is baked in

Here's the fact that surprises people building on a snapshot-restore platform, and it's the load-bearing one: a Firecracker guest's memory size and vCPU count are properties of the snapshot, not of the restore call. When you snapshot a running microVM, its memory image is exactly the size of the guest's RAM, and the machine configuration (memory size, vCPU count) is frozen into the snapshot state. When you restore, Firecracker maps that memory image back and resumes — it cannot resize the guest as part of restoring it. You get a guest with exactly the RAM and vCPUs it had when the snapshot was taken. There is no "restore this 2 GiB snapshot as a 4 GiB guest" flag, because the memory image simply isn't that big and the resume path isn't a boot.

The practical consequence for anyone building on top: an app's memory is governed by the baked snapshot's configuration, not by whatever size you pass in the create request. If your platform restores every sandbox from a template snapshot, the template decides the RAM. You can send `memory_mb: 8192` in a create body all day, but if the snapshot being restored was baked at 2 GiB, you get a 2 GiB guest — the restore path overrides the request to match what the snapshot actually contains. This isn't a bug to work around; it's the physics of restoring a memory image. To change the size, you change the thing you're restoring.

So how do I actually give my microVM more RAM?

You pick a template whose baked snapshot already has the memory you need, and if you need a different size, you destroy the VM and create a new one from a differently-sized template. Resizing is re-provisioning. That sounds heavy until you remember that on a snapshot-restore platform a create is cheap — a restore lands around 49ms, with an end-to-end create at roughly 179ms p50 and 203ms p99; only the very first cold boot of a template is around 3s. Throwing away a 2 GiB VM and booting a 4 GiB one is a sub-second operation, not the multi-minute reprovision a full-VM world would charge you. So "resize" collapses into "destroy and re-create against the bigger template," which is both simpler and safer than plumbing live memory into a running guest.

from pandastack import Sandbox

# You do NOT resize a running microVM. Its RAM is baked into the template's
# snapshot, so you pick the template whose baked size matches the workload.
#
# Say "base" is baked at 4 GiB and "code-interpreter" at 2 GiB (these are the
# template's baked sizes, not something you pass per-request). The memory you
# get is decided by the SNAPSHOT, not by the create call.

# Need ~2 GiB? Restore the small template.
small = Sandbox.create(template="code-interpreter", ttl_seconds=300)

# Need more headroom (e.g. a Next/Vite/tsc build that OOMs at 2 GiB)?
# You don't grow `small`. You compost it and grow a bigger one.
small.destroy()
big = Sandbox.create(template="base", ttl_seconds=300)   # baked at 4 GiB

# Passing memory_mb here would be ignored where it disagrees with the baked
# snapshot: the restore path forces cpu/mem to match what the snapshot holds.
# To change the size for real, bake a template snapshot at the size you want.
result = big.exec("npm run build")
print(result.exit_code)
big.destroy()   # the whole microVM goes away — no live-resize surgery anywhere

If none of your existing templates has the size you need, the real fix is upstream: bake a template snapshot at the memory size you want, and restore from that. That's a one-time bake, after which every create off that template comes up at the right size in restore time. This is exactly how first-party templates get their sizes — for example, a general apps template baked at 4 GiB / 2 vCPU specifically because Next/Vite/tsc builds OOM'd at 2 GiB. The memory decision lives at bake time, deliberately, so it's a property of the immutable artifact rather than a per-request knob that the restore path would have to honor and can't.

Balloon vs. hotplug vs. re-provision, side by side

• Mechanism — Balloon: inflate/deflate a driver inside the guest to hand idle pages back to the host, under a fixed ceiling. Hotplug: add/remove physical memory to a running guest so its total RAM changes live. Re-provision: destroy the VM and create/restore a differently-sized one.
• Direction — Balloon: reclaims idle RAM (and returns it, up to the original size). Hotplug: grants (or removes) RAM, raising/lowering the live ceiling. Re-provision: any size you want, because it's a fresh guest.
• Firecracker support — Balloon: yes, virtio-balloon is a supported device. Hotplug: no, deliberately omitted to keep the device model minimal. Re-provision: always available and, with snapshot-restore, cheap.
• Use case — Balloon: memory overcommit and density across many mostly-idle guests. Hotplug: growing a long-lived pet VM in place (a full-VM/QEMU world). Re-provision: changing a workload's memory size on a disposable-VM platform.
• Cost — Balloon: near-zero, one API call, cooperative. Hotplug: n/a on Firecracker. Re-provision: one destroy + one create — sub-second when the target template snapshot is warm (~49ms restore, ~179ms p50 create), ~3s only on a template's first cold boot.

Read that table and the strategy falls out of it. Balloon is your density lever — run it across the fleet to reclaim slack from idle guests and overcommit safely. Hotplug is the lever you'd want in a pet-VM world and simply don't have here, and you don't miss it because re-provision is the lever that replaces it. And re-provision is cheap enough, thanks to snapshot-restore, that resizing-by-replacement stops being a compromise and becomes the clean default.

When you'd genuinely miss hotplug (and why you usually don't)

To be fair to hotplug: there are workloads where growing a running guest in place is genuinely the right shape. A stateful process that has spent an hour building up an in-memory index, a database with a warm buffer cache you don't want to cold-start, a long-lived session with expensive-to-rebuild state — for those, "kill it and boot a bigger one" throws away work, and hotplug's whole point is to add headroom without losing that state. In the full-VM world where the VM is a pet and the create is slow, hotplug earns its complexity.

But that's precisely the workload shape a serverless-microVM platform is built to avoid. The moves that would make you reach for hotplug — hoarding hours of in-memory state in one irreplaceable guest — are the moves you design out when the VM is disposable and creates are sub-second. If a guest holds state that's expensive to lose, the microVM-native answer is a snapshot: capture the warm state, then restore it into a fresh (and, if you re-baked, larger) guest, rather than growing the original live. And for the copy-on-write cases — hand a warm parent to several children — you fork: a same-host fork lands in roughly 400–750ms sharing the parent's memory copy-on-write, and a cross-host fork is 1.2–3.5s once the artifacts move over the network. Between snapshot, restore, and fork, the state-preservation job that hotplug does for a pet VM is covered without ever adding memory to a running guest — and without the device-model surface that made hotplug a security cost in the first place.

How this plays out in PandaStack

PandaStack is my open-source Firecracker platform, and it leans into all three of these facts rather than fighting them. Every sandbox, managed database, and hosted app is its own Firecracker microVM, created via snapshot-restore, and its memory is governed by the template snapshot it restores from — not by the create request. The create path deliberately overrides the request's cpu/mem to match the baked snapshot, because trying to honor a mismatched size at restore is impossible: the memory image is the size it is. So sizing is a template decision, made at bake time, and "give this workload more RAM" means "target the bigger template," not "resize the running VM."

The balloon rides on top as the density lever — reclaiming idle slack so the platform can pack many mostly-idle sandboxes onto one KVM host, where the binding constraint is memory rather than networking (an agent pre-allocates 16,384 /30 subnets, so per-sandbox network isolation is never the bottleneck). And re-provision is cheap enough — ~49ms restore, ~179ms p50 / ~203ms p99 create, ~3s only on a template's first cold boot — that resizing-by-replacement is the intended workflow, not a workaround. Because the core is Apache-2.0 open source, you can run the control-plane API and per-host agent on your own KVM hosts and watch this exact model play out. The one thing to internalize is the one that trips everyone up: on a snapshot-restore microVM, you size the template, not the request — and if you want more RAM, you compost the VM and grow a new one.