Firecracker Shutdown and Reboot Semantics, Explained

Starting a Firecracker microVM is the part everyone benchmarks. Stopping one is the part everyone gets wrong, then discovers six weeks later when a host running 300 sandboxes has 4,000 network namespaces and no free disk. "Just kill the process" is a technically valid way to stop a microVM, and also the reason your fleet slowly fills with corpses. I'm Ajay; I built PandaStack, which creates and destroys Firecracker microVMs continuously. Here's what actually happens when a microVM dies, what each path leaves behind, and why the dead VM is never the operational problem.

There is no power button

Firecracker is deliberately minimal. It does not emulate ACPI the way a full QEMU-style machine does, so there is no power button to press and no ACPI shutdown event for the guest to receive. That's a design decision, not an omission — Firecracker's security argument rests on a very small device model. The consequence: "politely ask the guest to shut down" must be expressed through a device that does exist.

On x86_64: SendCtrlAltDel through the emulated i8042

Firecracker's API socket exposes a small set of actions via PUT /actions, and the one that matters here is SendCtrlAltDel. It injects the Ctrl+Alt+Del key sequence through the emulated i8042 keyboard controller. By convention, Linux init treats that as a request to reboot in an orderly fashion: systemd (or whatever is PID 1) runs its shutdown transaction, stops units, unmounts filesystems, then asks the kernel to restart. The canonical graceful path on x86_64 is therefore not a power button at all — it's a keyboard interrupt your guest's init has been culturally trained to read as "wrap it up."

Which means the graceful path depends entirely on the guest. If PID 1 doesn't listen for Ctrl+Alt+Del — a minimal init, a custom PID 1, a guest that boots straight into your application — the action is delivered and nothing happens. The VM keeps running. That's the most common surprise here: SendCtrlAltDel succeeds whether or not the guest cares.

On aarch64: the i8042 doesn't exist, so neither does the action

There is no i8042 on aarch64, so Firecracker doesn't emulate one, so SendCtrlAltDel isn't available there. You signal shutdown another way: reach the guest's userspace (a vsock guest agent running `poweroff`, an SSH command, your own control channel), or skip the graceful path and signal the VMM from the host. Develop on an Apple Silicon Mac and deploy to x86_64 servers and this asymmetry bites exactly once, looking like "shutdown works in prod but hangs locally."

reboot=k, and why a Firecracker guest reboot is really an exit

Look at almost any Firecracker kernel command line and you'll see `console=ttyS0 reboot=k panic=1 pci=off`. The `reboot=k` does quiet, load-bearing work: it tells Linux to reboot via the keyboard-controller method — poke the i8042 reset line — instead of ACPI, EFI, or a triple fault. Firecracker traps that reset. And here's the part that trips people up: Firecracker does not reboot the machine. It exits.

There is no "restart the guest" in the Firecracker model. A microVM is a process; a guest reset is terminal for that process. So `reboot` inside the guest is a synonym for `shutdown`, and `panic=1` plus `reboot=k` means "a guest kernel panic becomes a clean VMM exit within a second" — exactly what you want in a fleet, since a panicked guest sitting there forever is a leaked VM you're still paying for. An orchestrator that assumes a guest can reboot in place and come back will instead observe every reboot as a death, and it will be right to.

In Firecracker, "reboot" is not a lifecycle event. It is a suicide note with good manners.

SIGTERM, SIGKILL, and the guest that was never told

Once you give up on the graceful path you're signalling the VMM process, and it's worth being precise about what that means for the guest. Both SIGTERM and SIGKILL end the Firecracker process with no notification to the guest whatsoever: execution proceeds normally and then there is no more inside the VM. Unflushed page cache gone, in-flight virtio-blk requests gone. The difference between the two signals is entirely about the host side.

• SIGTERM — the process can in principle run teardown before exiting: close the API socket, flush metrics, release descriptors in an orderly way. What it never does is let the guest sync.
• SIGKILL — the kernel reaps the process immediately with zero userspace teardown: descriptors closed, KVM VM object destroyed, guest memory unmapped. Always works, leaves the largest pile of debris.
• Neither — substitutes for the graceful path if the guest writes to a durable volume. To the guest filesystem, both are unplugging the machine.

The correct shape is an escalation ladder with a hard timeout at every rung and a reclaim step at the bottom that runs unconditionally: ask nicely, wait a bounded time, escalate, wait again, escalate again, then clean up regardless of which rung did the killing.

#!/usr/bin/env bash
# Stop one Firecracker microVM: graceful -> forceful -> reclaim.
# Deliberately no `set -e`: a failed step must never skip the reclaim.
set -uo pipefail

VMID="${1:?usage: stop-vm.sh <vm-id>}"
RUN=/srv/jailer/firecracker/$VMID/root
API=$RUN/run/firecracker.socket
NS=ns-$VMID
GRACE=10      # seconds for guest init to unmount and halt
TERM_WAIT=5   # seconds for the VMM to exit on SIGTERM

pid=$(cat "$RUN/firecracker.pid" 2>/dev/null || true)
alive() { [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; }

# 1. Ask nicely. x86_64 only: SendCtrlAltDel arrives via the emulated i8042,
#    guest init reads it as "shut down", the resulting reset traps to the VMM,
#    and Firecracker EXITS (it does not reboot). On aarch64 there is no i8042 --
#    tell the guest agent to run `poweroff`, or go straight to SIGTERM.
if [ -S "$API" ]; then
  curl -sS --unix-socket "$API" -X PUT "http://localhost/actions" \
    -H 'Content-Type: application/json' \
    -d '{"action_type":"SendCtrlAltDel"}' >/dev/null || true
fi

# 2. Bounded wait. A guest with no Ctrl+Alt+Del handler ignores step 1
#    entirely and the API still returned success, so never wait forever.
for _ in $(seq "$GRACE"); do alive || break; sleep 1; done

# 3. Escalate. From here the guest gets no warning at all.
if alive; then
  echo "vm $VMID ignored ctrl-alt-del after ${GRACE}s; SIGTERM"
  kill -TERM "$pid" 2>/dev/null || true
  for _ in $(seq "$TERM_WAIT"); do alive || break; sleep 1; done
fi

# 4. Escalate again. This is the power cord.
if alive; then
  echo "vm $VMID still up after SIGTERM; SIGKILL"
  kill -KILL "$pid" 2>/dev/null || true
  sleep 1
fi

# 5. Reclaim. THE ONLY STEP THAT MATTERS AT FLEET SCALE. It runs no matter
#    which rung above actually killed the VM -- including "none of them,
#    it was already dead when we got here".
ip netns pids "$NS" 2>/dev/null | xargs -r kill -KILL 2>/dev/null || true
ip netns del "$NS" 2>/dev/null || true       # takes tap0 + guest veth with it
ip link del "vh-$VMID" 2>/dev/null || true   # host veth, if it outlived the ns
rm -f "$API" "$RUN/run/vsock.sock" 2>/dev/null || true
rm -rf "$RUN/overlay" 2>/dev/null || true    # disposable CoW clone
rmdir /sys/fs/cgroup/firecracker/"$VMID" 2>/dev/null || true
echo "vm $VMID stopped and reclaimed"

The VMM exited. The sandbox did not.

This is the whole point of the post. A Firecracker process dying is cheap, fast, and reliable. The pile of host-side objects created to make that VM exist is none of those things, because almost none of them are owned by the process's lifetime. Firecracker exits, the kernel reclaims its memory and descriptors, and everything else is yours forever:

• Network namespace and tap device — the per-sandbox netns outlives the process. Deleting it takes the tap and guest veth with it, but nobody deletes it for you. The number-one leak.
• Host-side veth and iptables/nftables rules — netns gone but host end or NAT rule left behind: a dangling interface and a rule pointing at nothing.
• Jailer chroot — the per-VM chroot tree with its bind-mounted rootfs and device nodes stays on disk, and those bind mounts can block later cleanup.
• cgroups — the jailer's per-VM cgroup directory does not vanish with the process. Empty cgroups are small; a few thousand a day is a real leak.
• The API socket and the vsock UDS — stale socket files are why the next restore on a recycled ID fails with "address already in use."
• CoW clone files and snapshot artifacts — the reflinked rootfs clone, the memory file, any diff snapshot. These silently eat the disk.
• The control-plane row — a row saying "running" for a VM that died two hours ago is how a scheduler places work onto capacity that doesn't exist.

The discipline that makes this survivable: cleanup must be idempotent and must run from more than one place. From a defer right after the kill. From agent startup, sweeping what the last crash orphaned. From a periodic janitor pass. If any of those three corrupts state by running twice, you don't have cleanup, you have a race. Here's the shape in Go — every step treats "already gone" as success, and the record is marked reclaimed last, so a crash mid-way just means the janitor tries again.

package sandbox

// Reclaim tears down every host-side resource belonging to one microVM.
// It is idempotent by construction: "already gone" is success. Safe to call
// from a defer after kill, from crash recovery at agent startup, and from a
// periodic janitor -- in any order, any number of times, concurrently.
func Reclaim(ctx context.Context, vm VM) error {
	var errs []error
	step := func(name string, fn func() error) {
		if err := fn(); err != nil && !alreadyGone(err) {
			// Record and keep going. One stuck resource must never
			// prevent the other six from being reclaimed.
			errs = append(errs, fmt.Errorf("%s: %w", name, err))
		}
	}

	// 1. The VMM process. Verify /proc/<pid>/exe still points at firecracker
	//    before signalling -- PIDs get reused, and killing the wrong one is
	//    a genuinely exciting way to lose a neighbour's VM.
	step("vmm", func() error { return killIfStillFirecracker(vm.PID) })

	// 2. Network. Deleting the namespace destroys tap0 and the guest veth
	//    with it: one call, several resources. The host veth may survive.
	step("netns", func() error { return netns.Delete(vm.NetNS) })
	step("veth", func() error { return netlink.LinkDelByName(vm.HostVeth) })

	// 3. Unix sockets. Stale socket files are the classic cause of a failed
	//    restore on a recycled slot.
	for _, path := range []string{vm.APISocket, vm.VsockUDS} {
		step("socket", func() error { return os.Remove(path) })
	}

	// 4. Disk. The CoW clone is disposable and gets deleted. A durable
	//    volume is NOT ours to delete -- detach it and leave it alone.
	step("clone", func() error { return os.RemoveAll(vm.CloneDir) })
	step("volume", func() error { return detachVolume(ctx, vm.VolumeID) })

	// 5. Jailer chroot + cgroup. Nothing removes these on process exit.
	step("chroot", func() error { return os.RemoveAll(vm.ChrootDir) })
	step("cgroup", func() error { return os.Remove(vm.CgroupPath) })

	// 6. Only now touch the control plane. Crash before this and the next
	//    janitor pass re-runs Reclaim, which is a no-op the second time.
	step("record", func() error { return vm.Store.MarkReclaimed(ctx, vm.ID) })

	return errors.Join(errs...)
}

// alreadyGone reports whether an error just means the resource was already
// cleaned up: no such file, no such process, no such device.
func alreadyGone(err error) bool {
	return errors.Is(err, os.ErrNotExist) ||
		errors.Is(err, syscall.ESRCH) ||
		errors.Is(err, syscall.ENODEV)
}

Pre-allocating network resources also changes the failure mode in your favour. PandaStack agents pre-build 16,384 /30 subnets with namespaces and taps ready to go, so allocation is a slot check-out and cleanup is "return the slot to the pool" — far easier to make idempotent than "destroy this namespace, unless it's already destroyed, unless something still holds it open." A leak then shows up as a shrinking free list, not a slow accumulation you find at 3 a.m.

Unclean shutdown and the guest filesystem

SIGKILL leaves the guest filesystem exactly as a power cut would: journal uncommitted, page cache unflushed, whatever ext4 had queued simply lost. On a normal server that's a real problem. On a disposable copy-on-write rootfs it mostly isn't, and it's worth being clear why: every sandbox boots from a reflinked or dm-snapshot clone of a baked template image, and that clone dies with the sandbox. You aren't corrupting a filesystem you intend to keep — you're discarding a scratch copy that happens to be inconsistent, which is a distinction without a difference.

Durable volumes are the opposite case. A managed Postgres VM, a persistent app sandbox, anything with a volume attached — those bytes are the product. Before any stop path that isn't graceful, the guest needs to have actually flushed: stop the service, `sync`, ideally unmount or freeze. Postgres and friends are crash-safe and recover from their WAL, so SIGKILL is survivable — but "survivable via crash recovery" and "clean" are different things, and recovery on a large database isn't free. The rule to write on the wall: ephemeral rootfs, kill freely; durable volume, quiesce first.

Reading the corpse: how the VM died

Three very different events all look like "the Firecracker process is gone," and they demand three different responses: the guest asked to shut down (normal), the VMM failed (bug or misconfiguration, page someone), the host OOM killer took it (capacity problem, stop scheduling here).

1. Exit status 0 — the guest requested a reset and Firecracker exited cleanly. A successful SendCtrlAltDel, an in-guest `poweroff`, and a `panic=1` reboot-on-panic look identical from outside. The last is a crash wearing a clean exit code, so check the serial console before calling it healthy.
2. Non-zero exit, no signal — Firecracker itself failed. It defines specific exit codes for configuration and internal errors; check your version's docs rather than guessing, and read the log.
3. Terminated by a signal — the 128+N shell convention applies: 143 is SIGTERM (you or your orchestrator), 137 is SIGKILL, 139 is SIGSEGV and a genuine VMM crash worth reporting.
4. 137 specifically — is ambiguous: either your ladder reached the SIGKILL rung, or the host OOM killer did. Check the kernel log for an oom-kill line and the cgroup's memory.events for an oom_kill bump. If it's the OOM killer, the VM is a symptom and your memory accounting is the bug.
5. A seccomp violation — Firecracker runs under a seccomp-BPF filter and a blocked syscall kills it. After a kernel, library, or Firecracker upgrade, suspect the filter before the guest.

Whatever the cause, the response is identical in one respect: run the same idempotent reclaim. "How did it die" changes what you alert on. It never changes whether you clean up.

Pause and snapshot: the third kind of stop

There's a stop that isn't a death. Pause the VM (vCPUs stop executing), snapshot it (memory file plus device state, alongside the disk), then let the VMM process exit. The microVM is now stopped in every way that matters for cost — no vCPU threads, no resident guest memory, no process — but it is not gone. Restoring it resumes the exact machine, mid-syscall, its processes unaware any time passed. That's a footgun as much as a feature: a restored guest's clock is stale and the connections it thinks it has are long dead.

This is the mechanism behind scale-to-zero. An idle sandbox doesn't need to be deleted and rebuilt, it needs to be parked. Waking it is a snapshot-restore — the same fast path a fresh create uses. On PandaStack that's p50 179ms, p99 around 203ms, with the restore step itself about 49ms, versus roughly 3 seconds for a genuine first-ever cold boot. So "stop this thing" almost never has to mean "destroy this thing."

from pandastack import Sandbox

# Three different things people all call "stopping the VM".
sbx = Sandbox.create(
    template="base",
    ttl_seconds=3600,          # backstop: the platform reclaims it regardless
    metadata={"job": "nightly-report"},
)

# (a) Quiesce. If anything durable is attached, flush it BEFORE you stop.
#     Neither a snapshot nor a SIGKILL will do this for you.
sbx.exec("systemctl stop report.service || true", timeout_seconds=30)
flush = sbx.exec("sync", timeout_seconds=60)
assert flush.exit_code == 0, flush.stderr

# (b) Stopped but resumable. Memory + device state + disk are captured; the
#     VMM process goes away, the machine does not. Waking it is a restore,
#     not a cold boot -- so parking an idle sandbox is nearly free.
snap = sbx.snapshot()
print("parked:", snap)

# (c) Stopped and gone. The sandbox AND every host resource behind it --
#     netns, tap, CoW clone, sockets, chroot, cgroup -- are reclaimed.
sbx.delete()

# In any code path that can raise, let the context manager own the lifetime.
# The teardown runs on the way out whether the block succeeded or blew up,
# which is the same discipline as the deferred Reclaim above -- just yours
# to get right once instead of at every early return.
with Sandbox.create(template="code-interpreter", ttl_seconds=600) as job:
    out = job.exec("python /work/analyze.py", timeout_seconds=300)
    if out.exit_code != 0:
        raise RuntimeError(out.stderr)   # VM is still torn down on the way out

Four ways to stop a microVM, side by side

Here's the honest comparison of the paths. Behaviour varies by Firecracker version, guest init, and architecture, so verify the specifics against the Firecracker docs for the release you actually run before you hard-code an assumption into your orchestrator.

• Does the guest know — SendCtrlAltDel: yes, if PID 1 listens; init runs its own halt path. SIGTERM: no, running one instant and gone the next. SIGKILL: no, identically. Pause + snapshot: no, frozen mid-instruction, never learns it stopped.
• Guest filesystem state — SendCtrlAltDel: clean, unmounted, journal committed. SIGTERM: dirty, as after a power cut. SIGKILL: dirty, identically. Pause + snapshot: consistent with the captured memory, but not flushed unless you synced first.
• How long it takes — SendCtrlAltDel: as long as guest init needs; unbounded, possibly forever if nobody listens. SIGTERM: near-immediate plus teardown. SIGKILL: immediate, no teardown. Pause + snapshot: proportional to the memory image written out.
• Can it come back — SendCtrlAltDel: no. SIGTERM: no. SIGKILL: no. Pause + snapshot: yes, restore resumes the exact machine.
• Host resources reclaimed — SendCtrlAltDel: no. SIGTERM: no. SIGKILL: no. Pause + snapshot: no. This row is the point of the article: none of the four clean up your netns, tap, chroot, cgroup, sockets, or files. Always your job.
• Reach for it when — SendCtrlAltDel: a stateful guest with a durable volume. SIGTERM: routine teardown of a disposable sandbox. SIGKILL: the graceful window expired or the VM is wedged. Pause + snapshot: idle now, wanted later.

When this is all more machinery than you need

Everything above is the cost of the microVM model, and it's a real cost. If your workload is first-party code on infrastructure you control, with no untrusted input and no multi-tenancy, a container gets you a far simpler shutdown story: the runtime owns the namespaces and the writable layer and reaps them on process exit, so "just kill it" genuinely is the whole answer. Don't adopt a hypervisor and inherit a seven-item reclamation checklist to run your own cron job. The microVM approach earns its complexity when the code is untrusted, model-generated, or somebody else's — when per-tenant blast radius is a requirement rather than a nice-to-have.

If you're in that world, the lesson is narrow: killing a microVM is easy and boring, so build the whole system assuming it can happen at any moment for any reason. Give the guest a bounded graceful window because durable data deserves it. Escalate on a timer because guests hang. And put every host resource behind one idempotent reclaim function that runs from a defer, from crash recovery, and from a janitor — because the VM was never going to take you down. The 4,000 network namespaces were.