What Runs as PID 1 Inside a MicroVM (and Why It Matters)

A microVM boot is shorter than you think. The VMM loads a kernel image, hands it a command line, and jumps to it. The kernel brings up its subsystems, finds the root block device, mounts a filesystem on it, and then does exactly one interesting thing: it executes a single program and gives it process ID 1. Everything you think of as "the machine" — your agent, your shell, your build, your web server — is a descendant of that one process. Whatever it does, and more importantly whatever it fails to do, becomes a property of the sandbox that you will later diagnose as something else entirely.

I'm Ajay; I built PandaStack. This post is about what PID 1 actually owes the Linux kernel — the reaping contract, the genuinely weird signal semantics, and the fact that its return is a kernel panic — and then about the choice you have to make in a microVM: `init=/bin/sh`, a tiny custom init, a tini-style reaper, busybox, systemd, or your own guest agent wearing the PID 1 hat. Plus the part almost nobody writes about: what init's job becomes when every sandbox create is a snapshot restore rather than a boot, and the machine wakes up believing it is still bake day.

What PID 1 actually owes the kernel

There are exactly three obligations, and they are not obvious from reading any init's source. Each one, skipped, produces a distinctive failure that looks like a bug somewhere else.

It is the reaper of orphans

When a process exits, the kernel keeps a small husk of it around — the exit status, the resource usage — until its parent calls `wait()` and collects it. That husk is a zombie: a slot in the process table and a PID, nothing else. Normally the parent reaps promptly and you never see one.

But when a parent dies before its children, those children are re-parented, and in a plain namespace that means PID 1. A build script that forks a compiler and exits, a shell that backgrounds a job and returns, a Python process that spawns workers and gets killed — all of them hand their orphans to PID 1. If PID 1 never calls `wait()`, each of those orphans becomes a permanent zombie when it finishes. In a short-lived sandbox nobody notices. In a long-lived one that has serviced ten thousand exec calls, you reach a machine where `fork()` fails — and the error you actually see is npm complaining it cannot spawn a child, which sends you looking at npm.

It has signal semantics nothing else has

This is the one that surprises people. For any ordinary process, a signal with no installed handler gets the kernel's default action — SIGTERM terminates, SIGINT terminates. PID 1 is exempt. The kernel does not apply default actions for signals PID 1 has not explicitly installed a handler for; unhandled signals sent to it from other processes are simply discarded. That's a deliberate safety measure — it stops a stray `kill` from taking the system down by accident — and it is also why a naive PID 1 silently ignores SIGTERM.

The operational symptom: your control plane sends SIGTERM to stop a sandbox politely, nothing happens because the kernel dropped it, and after the grace period — ten seconds, thirty, whatever your default is — it sends SIGKILL. Nobody flushed anything. Nobody finished a transaction. You have written a graceful shutdown path that has never once run, and the only evidence is that stopping things is mysteriously slow.

It must not exit — ever

If PID 1 exits, the kernel panics. Not "the system shuts down" — panics, with a message along the lines of `Attempted to kill init!` including the exit code, which is how you get the genuinely funny incident report: your sandbox did not crash, its init returned zero. Success was fatal.

On a real machine that's a catastrophe. In a microVM it's closer to a shutdown mechanism, provided you configured the kernel to treat a panic as a fast reboot rather than a display of ASCII. That's what `panic=1` and `reboot=k` are doing below, and it's why boot args and init choice have to be designed together rather than copy-pasted from separate blog posts.

Where init= actually gets set

In Firecracker there is no bootloader, no GRUB menu, no initramfs stage to hide logic in. You PUT a boot source over the API socket: a kernel image path and a `boot_args` string. That string is the guest kernel command line, and the last token in it is the entire subject of this post.

# The Firecracker boot source: a kernel, and the string the kernel parses.
# Everything after the image path is the guest kernel command line.
curl --unix-socket /run/firecracker.sock -i \
  -X PUT 'http://localhost/boot-source' \
  -H 'Content-Type: application/json' \
  -d '{
    "kernel_image_path": "/var/lib/pandastack/kernels/vmlinux-5.10",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off i8042.noaux i8042.nomux init=/usr/bin/pandastack-init"
  }'

# console=ttyS0 -- kernel messages go to the serial port, which the VMM hands
#                  you as an ordinary file. Without it, a guest that dies before
#                  your agent starts dies silently and you get to guess.
#
# reboot=k      -- reboot via the keyboard-controller path. A microVM has no
#                  ACPI to negotiate with, so this turns `reboot` inside the
#                  guest into an immediate, clean VM exit instead of a hang.
#
# panic=1       -- on panic, reboot after 1 second rather than sitting on a
#                  panic screen forever. This matters *precisely because PID 1
#                  exiting IS a panic*: with reboot=k it is the difference
#                  between a wedged VM you keep paying for and a process that
#                  exits and frees its slot.
#
# pci=off, i8042.* -- do not probe hardware that is not there. In a snapshot
#                  world you pay boot cost once, at bake time, but it is free.
#
# init=...      -- the one program the kernel execs after mounting the rootfs.
#                  If this path is missing or not executable, the kernel falls
#                  back to its built-in list (/sbin/init, /etc/init, /bin/init,
#                  /bin/sh) and panics if none of them work. Usual causes: a
#                  "static" binary that isn't, or a rootfs for the wrong arch.

Two practical notes. `init=` overrides whatever the rootfs would have chosen, which makes it the best bisecting tool you own. And the whole string is readable from inside the guest at `/proc/cmdline`, which is the fastest way to discover that the image you are debugging is not the image you thought you configured.

The spectrum of choices, honestly

`init=/bin/sh` is where everyone starts, and it is genuinely the right tool for debugging: a root shell on the serial console at the earliest possible moment, before any service could have gone wrong. It is also a terrible production init for all three reasons above at once. A shell does not reap orphans it did not create with a job-control `wait`. It installs no SIGTERM handler. And it exits — a stray EOF on the console panics the kernel. It's the debugging equivalent of removing the doors to see the engine better.

A tiny custom init is where most microVM platforms land. A few hundred lines: install handlers, spawn the workload, loop on `waitpid`, forward signals to the child's process group, exit with the child's status. It starts in tens of milliseconds because there is nothing in it. The cost is that it is code you now own, including the parts you didn't think about the first time.

Tini and dumb-init style reapers are that same idea, already hardened by the container world — tini reaps and forwards signals, and dumb-init's specific contribution is forwarding to the process *group* rather than the single child, which is the exact bug most hand-rolled inits ship with. They're excellent wrappers and they are not service managers: they supervise one thing and are done when it dies. Busybox init sits a step up, reading `/etc/inittab`, respawning entries, reaping correctly — a real init in a very small binary, and reasonable machinery if you need two or three daemons with restart-on-crash.

systemd is the full article: dependency ordering, socket activation, per-unit cgroup accounting, journald, timers, restart policies. All genuinely useful, none of it free. The boot path is a graph of units being resolved, and the reason you chose a microVM was to not have a boot path shaped like that. It also expects a fairly complete environment — `/proc`, `/sys`, cgroup v2, often dbus — which is a set of assumptions rather than a set of features. If people SSH in and run services, systemd earns its keep. If the guest runs one agent process and is then deleted, you're paying a service manager to manage one service.

Choosing systemd for a single-purpose microVM is hiring a stage manager for a one-person show. Nothing goes wrong. It just costs more than the show.

A minimal, correct PID 1

Here is the whole contract in one file. I've written it in Go because the concurrency reads more clearly than the C equivalent, but the syscalls are the same ones you'd make from C and the ordering matters identically.

// A minimal, correct PID 1. Three jobs, ordered by how badly they break
// things when you skip them: reap orphans, handle signals, never return.
package main

import (
    "os"
    "os/exec"
    "os/signal"
    "syscall"
)

func main() {
    // 1. Install handlers BEFORE spawning anything.
    //
    // The kernel does not apply default actions to signals PID 1 has not
    // installed a handler for. A PID 1 with no SIGTERM handler does not die
    // on SIGTERM -- it ignores it. Your orchestrator then waits out its
    // grace period and sends SIGKILL, which is why "graceful shutdown"
    // shows up in your dashboards as a flat, unexplained 30-second stall.
    sigs := make(chan os.Signal, 16)
    signal.Notify(sigs, syscall.SIGTERM, syscall.SIGINT, syscall.SIGCHLD)

    // 2. Spawn the real workload in its OWN process group. A workload that
    // spawns a shell that spawns a build that spawns four compilers is one
    // process group; signalling only the direct child orphans the rest of
    // it, and the rest of it keeps burning CPU you are billed for.
    cmd := exec.Command("/usr/bin/pandastack-agent")
    cmd.Stdin, cmd.Stdout, cmd.Stderr = os.Stdin, os.Stdout, os.Stderr
    cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
    if err := cmd.Start(); err != nil {
        os.Exit(1) // no workload, no reason to exist -- and the kernel agrees
    }
    child := cmd.Process.Pid // == the new process group id, thanks to Setpgid

    // NOTE: we deliberately never call cmd.Wait(). It races the Wait4(-1)
    // below over the same exit status. Pick one owner and stick to it.
    for sig := range sigs {
        if s, ok := sig.(syscall.Signal); ok && s != syscall.SIGCHLD {
            // 3. Forward to the GROUP -- a negative pid means "this process
            // group". That is what reaches the grandchildren.
            _ = syscall.Kill(-child, s)
        }

        // 4. Reap EVERYTHING that is ready, not just our own child.
        //
        // When any process in the guest dies, its children reparent to PID
        // 1. If nobody wait()s on them they sit as zombies: a slot in the
        // process table holding an exit status no one read. One is
        // invisible. Ten thousand, in a sandbox that has served ten
        // thousand execs, is a machine that can no longer fork. WNOHANG
        // means "reap what is ready, do not block" -- we have to get back
        // to the signal loop.
        for {
            var ws syscall.WaitStatus
            pid, err := syscall.Wait4(-1, &ws, syscall.WNOHANG, nil)
            if pid <= 0 || err != nil {
                break // nothing left to reap right now
            }
            if pid == child {
                // 5. Exit DELIBERATELY, with the child's status. PID 1
                // returning at all panics the kernel, so make that panic a
                // decision instead of the consequence of falling off the
                // end of main(). With panic=1 reboot=k on the cmdline this
                // is a clean, fast VM exit your supervisor can observe.
                os.Exit(ws.ExitStatus())
            }
        }
    }
}

Roughly forty lines of substance, and note what is not in there: no service graph, no dependency resolution, no config file. This init has no opinions to be wrong about, and its startup cost is dominated by the dynamic linker — it rounds to nothing against the boot budget I broke down in /blog/microvm-boot-time-anatomy.

Why snapshots change the question entirely

Here is where microVM init design stops resembling container init design. If every create is a fresh boot, init's boot cost is on your critical path. But if every create is a snapshot restore — which is how PandaStack works: a template's first spawn is a real cold boot of about 3 seconds at bake time, and every create after that restores the baked snapshot, roughly 49ms for the restore step and 179ms p50 end to end, about 203ms p99 — then init's boot cost is paid once, in a build job, and never again. You could run systemd and it would cost you nothing per create. That doesn't make the choice irrelevant; it moves it. What matters now is init's *restore* behaviour, and almost no init on earth was written with restore in mind.

Think about what the guest wakes up into. Its clocks have not advanced since the snapshot was taken, so it believes it is still bake day. Its network identity may have been re-patched underneath it — different MAC, different route, new namespace — and any DHCP lease it holds is stale, possibly by weeks. Every TLS session and database connection it had open points at a peer that closed the socket long ago. Every timer it armed believes in a world that ended. And the entropy pool it seeded at boot is now identical across every sandbox restored from that snapshot.

The frozen clock produces the most confusing outages, because it is silent until it isn't. A guest restored from a March snapshot, running in July, checks a TLS certificate's validity window and sees one that hasn't been issued yet — or happily accepts one that expired last month. Nothing logs an error about the clock; you get a wall of certificate-verification failures and go looking at your CA. The fix isn't clever: something on the resume path has to synchronize the clock before the workload gets a chance to care. On x86_64 Firecracker exposes a VMGenID device precisely so a guest can be told "you are a restored copy" — /blog/firecracker-vmgenid-device-explained covers what to hang off that signal, and /blog/firecracker-guest-clock-and-time-drift-explained goes deep on time.

Concretely, what belongs on a resume path: re-sync the wall clock, re-seed the CSPRNG, reconcile any network identity patched underneath you, drop and re-establish long-lived connections rather than letting them fail one request at a time, re-arm timers relative to now, and re-run the readiness handshake so the host knows the restore actually finished. That last one is not optional, which brings us to the agent.

The guest agent: PID 1, or PID 1's first child?

Every sandbox platform has a process inside the guest that the host talks to. PandaStack's is `pandastack-init`, and it speaks over vsock rather than the network — the host reaches it without the guest needing a NIC, a routable port, or a firewall rule, which matters when the workload in that guest is untrusted by construction. I wrote up the protocol design in /blog/firecracker-guest-agent-vsock-protocol; here I only care where it sits in the process tree.

The agent is load-bearing for readiness. The host does not consider a sandbox ready when the restore call returns — it considers it ready when something inside answers. Probing for that answer is how you know a restore genuinely finished rather than merely started, and it's the difference between a create-latency number you can trust and one that describes the VMM's optimism.

Agent as PID 1

The tempting design, and a defensible one: earliest possible start, no ordering problem because there is no ordering, and a trivially simple process tree. But the costs are real. Your agent now owns the reaping contract for every orphan in the machine, and that reap loop has to keep running while the agent is busy streaming a hundred megabytes of build output. It owns the signal contract, so a bug there is a machine that cannot be stopped politely. And the big one: if the agent crashes, PID 1 has exited, the kernel panics, and the sandbox is gone. A null dereference in your file-read path becomes hard machine loss rather than a restarted service. That may be acceptable — a sandbox with no working agent is useless anyway — but it should be a decision, not a discovery.

Agent supervised by a tiny init

The other shape: a small init as PID 1 owning reaping and signals, with the agent as its first and usually only child. Reaping is centralized in code whose only job is reaping, so it's easy to get right and easy to leave alone. Signal handling lives in one place. And the agent becomes restartable — if it dies, init respawns it and the sandbox recovers instead of evaporating. You give up a few milliseconds of startup and gain a supervision story.

One subtlety: a restarted agent has no memory of in-flight work, so any exec that was streaming when it died is now an orphan producing output nobody reads. A supervised agent needs to reconcile on start — enumerate what's running, decide what to adopt and what to kill — or you've traded a clean total failure for a messy partial one. Both designs are legitimate. Having no design, and finding out which one you picked during an incident, is not.

Failure modes you will actually recognize

None of the above is theoretical. Every item here shows up in a bug tracker with a title that names the wrong subsystem.

• Zombie accumulation in a long-lived sandbox — PID 1 is a shell, the sandbox has served exec calls for days, and `fork()` starts failing. The reported bug is "npm install randomly fails" or "cannot allocate memory"; the actual state is a process table full of husks nobody reaped. The tell is a count that only goes up and a restart that fixes it, which sends everyone chasing a memory leak.
• SIGTERM into the void — every stop takes exactly your grace period, on every sandbox, forever. It looks like network latency or a slow control plane. It is PID 1 with no handler installed and your orchestrator waiting for a reply that was never coming.
• The process group that outlived your exec — the call returned 0 and something is still running. A start script backgrounded a server, the shell exited, the server reparented to PID 1, and nothing supervises or reaps it. You find out when the CPU graph refuses to go idle.
• The last log line, missing — a process writing to a pipe gets libc's block buffering, not line buffering. It fills a 4 KB buffer, gets SIGKILLed because SIGTERM was ignored, and the buffer dies with it. The most important output in any debugging session is the last thing before the crash, and that's exactly the part you lose.
• The hang at shutdown — an unreachable network mount, or dirty pages on a device that is going away. `umount` blocks, init waits, and a teardown that should take milliseconds becomes a timeout. You can always pull the power on a VM, but a supervisor that hasn't been told to will wait politely for a long time.
• The panic that says success — PID 1 finished its work and returned cleanly with status zero, and the kernel panicked because init is not allowed to be finished. The log line is `Attempted to kill init!` with an exit code of 0x00000000, the most confusing possible thing to read at 3am.

What this looks like from outside the guest

From the caller's side, all of it shows up as one question: is process cleanup something you have to do, or something that happens to you?

from pandastack import Sandbox

sbx = Sandbox.create(
    template="code-interpreter",
    ttl_seconds=900,
    metadata={"job": "build-1841", "owner": "agent-runner"},
)
try:
    # Every exec starts a process tree inside the guest. That tree gets
    # cleaned up only if something in there is willing to wait() on it.
    out = sbx.exec("cd /work && ./build.sh > build.log 2>&1", timeout_seconds=600)
    if out.exit_code != 0:
        raise RuntimeError(f"build failed ({out.exit_code}): {out.stderr[-2000:]}")

    # The part people get wrong: the exec RETURNED, but a daemonized child of
    # build.sh may still be running. It reparented to PID 1 the instant its
    # parent exited. If PID 1 is a shell, that child now has no supervisor and
    # no reaper -- it just runs, in a machine you believe is idle, on CPU you
    # are billed for. Worth actually looking, once, on your own template.
    stray = sbx.exec("ps -eo pid,ppid,pgid,pcpu,comm | awk '$2 == 1 && $1 != 1'")
    if stray.stdout.strip():
        print("orphans still resident:\n" + stray.stdout)

    log = sbx.filesystem.read("/work/build.log")
finally:
    # The only teardown that is actually guaranteed. Killing the sandbox
    # destroys the VM, and that destroys every process in it -- reaped,
    # zombie, orphaned, wedged on a hung mount, all of it. This is the
    # underrated ergonomic win of a VM boundary: process cleanup stops being
    # a routine you have to write correctly and becomes the absence of a
    # machine. The TTL above is the same guarantee for the case where your
    # own process dies before it reaches this line.
    sbx.kill()

A disposable machine forgives a lot. But "the VM gets deleted eventually" is not a strategy for a sandbox that stays up for hours running an agent, and it does nothing for the zombie count, the ignored SIGTERM, or the log line you lost. The boundary saves you at the end; init determines the quality of everything before the end.

Debugging it from inside the guest

The good news: PID 1 problems are unusually easy to diagnose once you know where to look, because the kernel exposes all of it through `/proc`.

# --- Who is PID 1, really? ---------------------------------------------
cat /proc/1/comm                  # e.g. pandastack-init, systemd, sh, busybox
readlink /proc/1/exe              # the real binary behind that name
tr '\0' ' ' < /proc/1/cmdline     # argv (NUL-separated on disk)

# What did the KERNEL actually parse? Your init= may not be the one you set --
# a stale template, a fallback to /bin/sh, a rootfs from the wrong build.
tr '\0' ' ' < /proc/cmdline

# --- Which signals does PID 1 handle, and which is it dropping? ----------
# SigCgt is a hex bitmask of CAUGHT signals. All zeros means it handles
# nothing at all, which means SIGTERM to this machine does nothing.
grep -E 'Sig(Blk|Ign|Cgt)' /proc/1/status

# --- Zombies: state Z, parent 1, nobody ever called wait() ---------------
ps -eo pid,ppid,stat,comm | awk '$3 ~ /^Z/'
ps -eo stat= | grep -c '^Z'       # a number that only goes up IS the bug
cat /proc/sys/kernel/pid_max      # how much room is left before fork() fails

# --- Orphans still burning CPU after an exec "returned" ------------------
# ppid 1 and not init itself: something reparented and stayed.
ps -eo pid,ppid,pgid,etime,pcpu,comm | awk 'NR==1 || ($2==1 && $1!=1)'

# --- Output you are about to lose ----------------------------------------
ls -l /proc/1/fd/1                # a pipe? then it is block-buffered
stdbuf -oL -eL ./your-command     # line-buffer it, or lose the final words

# --- Mounts that will hang a shutdown ------------------------------------
awk '{print $5, $NF}' /proc/1/mountinfo

# --- The bisect tool -----------------------------------------------------
# Set init=/bin/sh in boot_args and watch the serial console. If the guest is
# healthy here, the problem is in init/services, not the kernel or rootfs.
# It is also a PID 1 with no reaper and no handlers: fine for five minutes,
# never for production.

The one prerequisite is that you can see the serial console, which is what `console=ttyS0` bought you. If a guest dies before the agent starts, the console is the only witness — /blog/firecracker-serial-console-debugging covers wiring it up and what the interesting messages mean.

The five options, across the dimensions that bite

• Orphan reaping — /bin/sh: none in practice, so orphans accumulate as zombies until the PID table gives up. tini-style reaper: correct; it is the entire reason the binary exists. busybox init: correct. systemd: correct, plus per-unit cgroup accounting so you can see whose children they were. Agent as init: correct only if you wrote the reap loop, and it must keep running while the agent is busy elsewhere.
• Signal handling — /bin/sh: no handler, so the kernel discards SIGTERM and every stop is a timeout then a kill. tini-style reaper: forwards signals, and the dumb-init variant forwards to the process group. busybox init: the standard set plus inittab shutdown actions. systemd: full stop semantics with per-unit timeouts and kill modes. Agent as init: yours to get right, including forwarding to the group rather than the child.
• Boot cost — /bin/sh: essentially zero. tini-style reaper: negligible, a tiny static binary. busybox init: small, plus whatever inittab starts. systemd: a unit dependency graph resolved on every boot, which is the shape of thing you chose a microVM to avoid. Agent as init: tens of milliseconds, mostly the dynamic linker.
• Restore and resume behaviour — /bin/sh: none; the shell has no idea it was snapshotted. tini-style reaper: none; it supervises, it does not re-initialize. busybox init: nothing built in, though a respawn entry can fake it. systemd: rich enough to model resume, and heavy enough that its own timers, clock assumptions, and journal go stale across one too. Agent as init: the only option you can actually design around resume — sync the clock, reseed entropy, re-arm timers, re-run the readiness handshake.
• Debuggability — /bin/sh: unbeatable, which is why it is the bisect tool. tini-style reaper: transparent; it barely does anything, so it barely hides anything. busybox init: readable inittab, small surface. systemd: excellent introspection once you know it, and a lot to learn before you do. Agent as init: as good as the logs you remembered to write before the serial console became your only channel.
• What happens when it crashes — /bin/sh: kernel panic; a stray EOF on the console is enough. tini-style reaper: it barely crashes, but it exits when its single child dies, which is a panic unless you planned for it. busybox init: survives child crashes and respawns them. systemd: restarts services per policy. Agent as init: a null dereference in your file-read handler is total machine loss — the strongest argument for putting a tiny init above it instead.

The summary

The kernel's contract with PID 1 is three clauses long. Reap the orphans, because everything in the machine eventually becomes your child. Install handlers for the signals you want to act on, because the kernel will not apply defaults on your behalf and a signal you didn't handle was silently thrown away. And do not return, because returning is a panic — the kernel has no concept of init being finished, so a clean exit with status zero reads to it as the system having lost its mind.

The choice among `/bin/sh`, a tiny custom init, a tini-style reaper, busybox, and systemd is usually framed as latency versus facilities, and on a fresh-boot platform that's right. On a snapshot-restore platform it stops being right, because boot cost is paid once at bake time and never again. What replaces it is a question almost nobody asks of an init: what do you do when you wake up somewhere else, at a time that is not now, with a network identity changed underneath you and timers that all believe in a world that has ended? Boot runs once; resume runs every time. That is the real case for a purpose-built guest init — not that it's faster, since at bake time nobody is watching, but that it's the only one you can design a resume path into.

For the surrounding machinery: the restore path is broken down in /blog/snapshot-restore-boot-path, where the milliseconds actually go is in /blog/microvm-boot-time-anatomy, the control protocol your agent speaks is in /blog/firecracker-guest-agent-vsock-protocol, the clock problem that makes resume paths mandatory is in /blog/firecracker-guest-clock-and-time-drift-explained, and the serial console you'll need when none of this works is in /blog/firecracker-serial-console-debugging.