Upgrading the Daemon That Owns Your Running VMs

Shipping a new version of a stateless API server is a solved problem. Drain connections, start the new process, stop the old one, and the worst case is a handful of retried requests. Every deploy tool on earth does this competently, which is why nobody writes about it any more. Shipping a new version of the per-host daemon that supervises live customer virtual machines is a completely different problem, and the reason it is different is that the daemon's children are not requests. They are other people's running machines, and the default behaviour of your init system is to take them with it.

I'm Ajay; I build PandaStack, a Firecracker microVM platform, which means every one of my Linux hosts runs an agent process that owns some number of live guests — customer sandboxes, hosted apps, managed Postgres instances with real data in them. I ship that agent regularly. The first few times I shipped it, I did not fully understand what `systemctl restart` was going to do, and I found out the way everyone finds out.

This post is the deploy problem nobody writes about: how to restart a process whose children are other people's machines, and how to know afterwards that you actually did.

The trap: your init system kills the whole cgroup

Here is the mechanism, because the mechanism is the entire lesson. When systemd starts a service it places the main process in a cgroup named after the unit. Anything that process forks — directly or several generations down — inherits that cgroup, because cgroup membership is inherited on fork and nothing removes it unless you explicitly move the process out. So if your daemon launches a VMM by calling fork and exec, the VMM lands in `system.slice/your-agent.service` alongside the daemon, and as far as systemd is concerned it is part of the service.

Then you run `systemctl restart`. The default `KillMode` is `control-group`, which means "stop this unit" is implemented as "signal every process in the cgroup." Not the main process. Every process. Your daemon gets a SIGTERM and so does every guest VMM it ever started, and the guests receive absolutely no warning — the VMM process simply stops existing, page cache unflushed, mid-instruction. From inside the guest it is indistinguishable from someone pulling the power cord out of a rack.

The second, sneakier version of the same trap is `TimeoutStopSec`. Suppose you set `KillMode=process` and think you are safe. Your daemon's shutdown handler now has to finish within `TimeoutStopSec` (90 seconds by default). If it hangs — a stuck object-storage call during a final state flush, a database handle that will not close, a mutex held by a goroutine waiting on a socket that is never going to answer — systemd escalates. And the escalation is `SIGKILL` sent with `KillMode`'s original semantics for the final `KillSignal` stage, plus, if `KillMode` is anything other than `process` or you have `Delegate=` misconfigured, a sweep of the cgroup. A daemon that hangs on shutdown once a quarter is an annoyance. A daemon that hangs on shutdown and takes forty guests with it is an incident report.

The unit file below is the shape that survives a restart. None of these settings are exotic; the point is that every one of them is a deliberate departure from a default that would kill your guests.

# /etc/systemd/system/pandastack-agent.service
#
# The daemon supervises Firecracker microVMs that must OUTLIVE it.
# Every setting below exists because the default would kill them.

[Unit]
Description=PandaStack host agent
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/local/bin/pandastack-agent

# THE IMPORTANT LINE. Default is control-group, which SIGTERMs every
# process in the unit's cgroup on stop -- including every VMM we forked.
# `process` signals ONLY the main process and leaves the rest alone.
KillMode=process
KillSignal=SIGTERM

# Belt and braces: even with KillMode=process, do not let the final
# SIGKILL stage sweep anything we forgot to move out of the cgroup.
SendSIGKILL=no

# Give the daemon enough time to flush per-VM state to disk, but not so
# much that a wedged shutdown blocks the deploy forever. If it cannot
# quiesce in 30s it is stuck, and we would rather it die than hang --
# adoption on the next start will sort out whatever it left behind.
TimeoutStopSec=30s

# Restarting must be routine, not dramatic. The guests are unaffected.
Restart=always
RestartSec=2s

# We move VMM processes into their own transient scopes, so systemd must
# let us manage a subtree rather than assuming it owns everything here.
Delegate=yes

# The daemon dying must never be interpreted as "the host is unhealthy"
# by something with the authority to reboot it.
OOMPolicy=continue

[Install]
WantedBy=multi-user.target

`KillMode=process` is necessary and not sufficient. It stops systemd from signalling your guests on a normal stop, but they are still in the service's cgroup, which means they still share its resource accounting, they still show up in `systemctl status`, and any future operator running `systemctl kill --kill-whom=all` gets a very exciting afternoon. The durable fix is to not put them there in the first place.

Reparenting: make the guests genuinely not your children

The design goal is stronger than "survives a restart." It is that the daemon's lifetime and the guests' lifetimes should be unrelated facts about the machine. The daemon should be able to crash, be upgraded, be killed by an OOM event, or be stopped by a confused operator, and the only observable consequence is that nothing is watching the guests for a few seconds.

Two mechanisms get you there. First, launch each VMM into its own transient scope — `systemd-run --scope --unit=vm-<id>` — or into a cgroup you create yourself outside the service tree. The VMM is then a peer of your daemon, not a descendant, and no operation on the service unit can reach it. Second, double-fork or explicitly `setsid` so the VMM is not in the daemon's process group either, because process groups are the other channel through which a signal you meant for one process reaches thirty.

The consequence of doing this correctly is that after a restart your new daemon comes up knowing nothing. It did not start those VMs. It has no file descriptors pointing at them, no in-memory map of them, no parent-child relationship to reap them by. Everything it needs to know has to have been written down.

Per-VM state on disk is the handoff protocol

Treat the on-disk per-VM directory as the interface between the old binary and the new one, because that is exactly what it is. It has to be sufficient to reconstruct full supervision from a cold start, which in practice means:

• The VMM PID, plus something to disambiguate it. PIDs get reused, and after an hour of uptime the PID you recorded may belong to somebody's cron job. Record the process start time from /proc/<pid>/stat, or verify /proc/<pid>/exe still resolves to the VMM binary before you signal anything.
• The API socket path. This is what lets you probe the VMM for liveness and issue commands to it without having started it — the real test of whether a VM is adoptable is whether its control socket answers.
• Network identity: namespace name, tap device, the allocated subnet slot. On PandaStack each host pre-allocates 16,384 /30 subnets, so a slot is a number in a pool — and a slot whose VM you failed to adopt is a slot that leaks.
• Storage: the copy-on-write rootfs clone path, any attached durable volume, the loop or device-mapper devices behind them.
• The baked snapshot metadata this VM was restored from: template name, seed generation, memory backend. This is what tells you whether the new binary can even understand this VM.
• A schema version on the whole record, written first, read first.

Write it atomically — temp file, fsync, rename — because the one time it matters is when the process died halfway through writing it. A truncated state file is worse than a missing one: a missing one means "adopt nothing here," while a truncated one means "parse error" and your reconcile loop has to decide, at 3 a.m., whether a parse error should garbage-collect a customer's database.

The reconcile-on-boot pass

The daemon's startup path should not assume it started anything. It should walk the state directory, probe what it finds, and sort each entry into adopt, reclaim, or quarantine. Crucially, "I don't understand this" is a third outcome and not a synonym for "delete it."

package agent

// Reconcile runs on every start -- first boot, planned upgrade, crash
// recovery -- and makes no assumption about who started the VMs it finds.
// The daemon is a supervisor that ADOPTS, never a parent that assumes.
func (a *Agent) Reconcile(ctx context.Context) (Stats, error) {
	var st Stats

	dirs, err := os.ReadDir(a.stateRoot) // /var/lib/pandastack/vms
	if err != nil {
		return st, fmt.Errorf("read state root: %w", err)
	}

	for _, d := range dirs {
		id := d.Name()
		st.Found++

		// 1. Load the record. A parse failure is NOT permission to delete.
		//    Quarantine it, alert, and let a human look. Deleting state you
		//    cannot read is how you turn a bad release into data loss.
		rec, err := LoadVMRecord(filepath.Join(a.stateRoot, id))
		if err != nil {
			st.Quarantined++
			a.metrics.Quarantined.Inc()
			log.Printf("reconcile vm=%s: unreadable state, quarantining: %v", id, err)
			continue
		}

		// 2. Version gate. A record newer than this binary understands means
		//    we are a ROLLBACK running after a forward migration. Refuse to
		//    touch it rather than reinterpreting fields we do not have.
		if rec.SchemaVersion > CurrentSchemaVersion {
			st.Quarantined++
			log.Printf("reconcile vm=%s: schema v%d > supported v%d; leaving alone",
				id, rec.SchemaVersion, CurrentSchemaVersion)
			continue
		}
		rec = MigrateRecord(rec) // v1 -> v2 -> v3, in memory, forward only

		// 3. Is the process still the process? PIDs are reused; a bare
		//    kill(pid) against a recycled PID is how you murder a neighbour.
		if !processStillIs(rec.PID, rec.StartTimeTicks, a.vmmBinary) {
			st.Dead++
			_ = a.Reclaim(ctx, rec) // idempotent: netns, tap, slot, clone, sockets
			continue
		}

		// 4. The authoritative liveness test is the VMM's own control
		//    socket, not the PID. A live process with a dead API socket is
		//    a wedged VM, which is a different problem from a dead one.
		if err := probeVMMSocket(ctx, rec.APISocket, 2*time.Second); err != nil {
			st.Wedged++
			a.metrics.Wedged.Inc()
			log.Printf("reconcile vm=%s: pid alive, socket dead: %v", id, err)
			continue // do NOT kill it here; let a human or a policy decide
		}

		// 5. Adopt. Re-register in memory, re-mark the network slot as taken
		//    so the allocator never hands it out twice, and restart the
		//    per-VM watchers this binary owns.
		a.network.MarkSlotInUse(rec.NATIDSlot)
		a.registry.Put(rec)
		a.startWatchers(ctx, rec)
		st.Adopted++
	}

	// 6. The other direction: rows the control plane thinks are HERE and
	//    running, that we did not find on disk. Report them; never delete
	//    them from here. A local disk problem must not become a global
	//    "customer's VM no longer exists".
	st.UnaccountedRemote = a.reportMissing(ctx, a.registry.IDs())

	a.metrics.Adopted.Set(float64(st.Adopted))
	a.metrics.ReconcileCompletedAt.SetToCurrentTime()
	return st, nil
}

Two things in there are worth stating explicitly. The liveness test is the VMM's control socket, not the PID — a process can be alive and completely unable to serve its guest, and those two states need different responses. And step 6 goes only one way: the host reports what it cannot find, and something with a fleet-wide view decides what that means. A host that deletes control-plane rows because its own disk looks wrong is a host that can amplify a local failure into a global one. That is the same ownership discipline that keeps a rolling deploy from reaping a peer's VMs.

State compatibility: the new binary must read the old binary's mind

Adoption creates an obligation people underestimate. Every upgrade is now a data migration, because the state on disk was written by the version you are replacing, and it is read by the version you are installing. You have built a serialization format with two independent implementations running against it, which is the classic setup for a compatibility bug.

The rules are not complicated, they are just easy to skip when you are shipping a one-line fix:

1. Version the record explicitly. An integer field, read before anything else, present from version one. Retrofitting a version field onto an unversioned format is a genuinely unpleasant afternoon.
2. Only ever add optional fields within a version. Removing a field, renaming it, or changing its meaning is a version bump, and a version bump means the new binary carries migration code for the old shape.
3. The new binary reads N-1 and N. The old binary must at minimum not choke on N — which usually means it ignores unknown fields rather than erroring on them. That property is what makes rollback survivable.
4. Never change the state schema and the boot path in the same release. If something breaks you want to know which of the two did it, and more importantly you want to be able to roll back one without the other.
5. Write the migration as a pure function from old struct to new struct, and unit-test it against real captured records from production, not fixtures you invented.

Snapshot format compatibility is the same class of problem wearing a different hat, and it is worse, because the artifacts live longer than your daemon does. A baked template snapshot may have been produced weeks ago by an older toolchain and is sitting in object storage waiting to be restored by whatever binary happens to be running when someone creates a sandbox. If your upgrade changes the memory backend, the device model, or the VMM version in a way that makes existing snapshots unrestorable, you have not shipped a daemon upgrade — you have shipped a fleet-wide cold-start event, where every create that used to take a p50 of 179 milliseconds now takes about three seconds while it boots from scratch and re-bakes. That is survivable. Discovering it during a traffic peak is less so.

Rollout discipline: one host, then wait

A daemon upgrade is not a deploy in the web sense, where a bad version affects requests currently in flight and then stops mattering. A bad daemon version affects the VMs on that host for as long as they live. The blast radius is not "traffic during the rollout"; it is "every guest on every host you have already upgraded." So the rollout is deliberately slow, and the pace is set by how long it takes for a subtle bug to become visible.

• Canary one host that has real VMs on it. A host with nothing running proves only that the binary starts. Pick a host with a mixed population — a long-lived app, a database, some short-lived sandboxes — because those exercise different adoption paths.
• Bake for longer than your slowest feedback loop. If a broken idle reaper only shows up after the idle timeout, and that timeout is an hour, a fifteen-minute bake window proves nothing about the reaper.
• Watch the reconcile counters, not the process state. "The service is active (running)" is the least informative signal available to you. Adopted, quarantined, wedged, reclaimed, and unaccounted are the numbers that tell you what happened.
• One host at a time, with the control plane's drain flag set before the restart and cleared after reconcile completes. A restarting host stops heartbeating for a few seconds, and a control plane that infers death from silence will start relocating work you did not ask it to relocate.
• Drain hosts you cannot upgrade in place. If the new version genuinely cannot adopt the old version's VMs, stop pretending it is an upgrade: stop scheduling to the host, let its workloads move or expire, then replace the machine.

The comparison that matters here is not blue-green versus canary in the usual sense, because you cannot run two daemons owning the same guests. It is closer to this:

• Blast radius — Stateless API rollout: requests in flight during the swap, seconds. Host daemon rollout: every guest on every upgraded host, for the guest's whole remaining lifetime.
• Rollback cost — Stateless API rollout: repoint traffic, done in seconds. Host daemon rollout: the old binary must still be able to adopt state the new one may have already migrated, which is a property you have to design for in advance or not have at all.
• What 'healthy' means — Stateless API rollout: error rate and latency on the new version. Host daemon rollout: the count of guests successfully adopted, which is a number that does not exist unless you built it.
• Safe pace — Stateless API rollout: as fast as your metrics resolve, often minutes. Host daemon rollout: as slow as your slowest background loop, often hours.

Verification that actually catches it

Now the part that cost me the most time to learn, and it is embarrassingly mundane. I once spent an afternoon debugging why a fix was not working in production, with the deployed version endpoint cheerfully reporting the version containing the fix. The endpoint was not lying about what it believed. The deploy tooling had re-pushed a stale artifact from a cache, and the binary on disk was the old one, compiled from a tree where the version string had already been bumped.

Never ask a service what version it is. Ask the filesystem what binary it is running, and ask the kernel when that binary was mapped.

A version endpoint reports a string that was compiled in. It cannot tell you that the artifact was built from the commit you think, or that the file currently on disk is the one you pushed, or that the running process is even executing the file that is on disk right now (replace a binary under a running process and `/proc/<pid>/exe` will show it as deleted while the old code keeps running happily). Verify the physical facts.

#!/usr/bin/env bash
# Verify a host agent deploy from OUTSIDE the process's own opinion.
set -uo pipefail

HOST="${1:?usage: verify-agent.sh <host> <expected-sha256>}"
WANT="${2:?}"
BIN=/usr/local/bin/pandastack-agent

ssh "$HOST" bash -s -- "$BIN" "$WANT" <<'REMOTE'
set -uo pipefail
BIN=$1; WANT=$2

# 1. The artifact on disk. Checksum, not version string: the version
#    string is whatever was compiled in, which is exactly the thing a
#    re-pushed stale artifact gets wrong.
GOT=$(sha256sum "$BIN" | cut -d' ' -f1)
[ "$GOT" = "$WANT" ] || { echo "FAIL binary sha $GOT != $WANT"; exit 1; }

# 2. When it landed. An mtime older than the deploy you just ran means
#    the deploy re-pushed something from a cache and told you it worked.
echo "binary mtime: $(stat -c %y "$BIN")"

# 3. Is the RUNNING process actually executing that file? If the binary
#    was swapped under a process that never restarted, /proc/<pid>/exe
#    resolves to a path marked (deleted) and the old code is still live.
PID=$(systemctl show -p MainPID --value pandastack-agent.service)
EXE=$(readlink "/proc/$PID/exe")
case "$EXE" in
  *"(deleted)"*) echo "FAIL running process holds a DELETED binary: $EXE"; exit 1 ;;
esac
[ "$EXE" = "$BIN" ] || { echo "FAIL running exe $EXE != $BIN"; exit 1; }

# 4. Did it adopt what was here? A clean start with zero adopted VMs on a
#    host that had thirty is the actual failure this whole post is about,
#    and it looks identical to a successful deploy in every other check.
curl -sS --unix-socket /run/pandastack/agent.sock \
  http://localhost/internal/reconcile | tee /dev/stderr | \
  grep -qE '"quarantined":0,.*"unadopted":0' || {
    echo "FAIL reconcile reported unadopted or quarantined VMs"; exit 1; }

echo "OK"
REMOTE

The alert to add alongside it is "VMs running but unadopted": count the VMM processes on the host, count the VMs the daemon believes it supervises, and page when those numbers disagree for more than one reconcile interval. That divergence is the signature of every failure mode in this post — a guest whose state file was unreadable, a slot the allocator will hand out twice, a VM nobody will ever reap or bill for. It is also, usefully, one of the very few alerts that fires on the deploy that half-worked, which is the deploy that hurts.

Honest limits: adoption makes restarts survivable, not free

I want to be careful not to oversell this. Everything above buys you one specific property: the daemon's lifetime stops being coupled to the guests' lifetimes. That is a large property and it is not the same as "upgrades are free."

• There is a supervision gap. Between the old process exiting and the new one finishing reconcile, nothing is watching. A guest that panics in that window stays dead until reconcile notices, and any event stream you produce has a hole in it. Keep the gap short and make everything downstream tolerate it.
• Guest-facing protocol changes do not adopt. If you change the wire format between the daemon and the in-guest agent, adopted VMs are still running the old guest side and will keep running it until they die. There is no upgrade path for a running guest's baked-in agent; there is only waiting. Old VMs have to drain out naturally, and until they do, your new daemon must speak both versions.
• Rollback is asymmetric. Forward migration of a state record is usually easy; going back means the old binary meets a record it has never seen. If it errors instead of ignoring unknown fields, your rollback is a quarantine event on every host.
• Long-lived guests extend every window. A short-lived sandbox population churns out old assumptions within an hour. A hosted app or a managed database can sit there for months, which means a compatibility shim you added "temporarily" is load-bearing for a very long time. Write down when it can be removed, or it never will be.
• Some upgrades genuinely cannot be done in place. Kernel changes, hypervisor major versions, a snapshot format break. Recognise those early and route them to a drain-and-replace, rather than discovering mid-rollout that adoption cannot work and improvising.

The summary

If your daemon owns processes that must outlive it, the defaults are against you. Set `KillMode=process` and a `TimeoutStopSec` short enough that a wedged shutdown fails fast, then go further and launch each guest into its own scope so no operation on your service unit can reach it. Write per-VM state to disk atomically with a schema version, and start every run with a reconcile pass that adopts what it finds, quarantines what it cannot read, and reclaims only what it can prove is dead. Never change the state schema and the boot path in the same release. Roll out one host at a time behind a drain flag, watch the adoption counters rather than the process state, and verify by checksum and mtime on the host rather than by asking the service what it thinks it is.

Do that and restarting the daemon becomes what it should be: a boring operation that no customer can detect. Skip it, and you will find out on a Friday.