How to Drain a Host Without Dropping Everything on It

Every fleet eventually needs a specific machine to go away. A kernel CVE with a patch you cannot defer, a cloud provider emailing you a maintenance window you did not schedule, a disk crossing 90% at 2am, a NIC dropping one packet in ten thousand in a way that only shows up as mysterious p99, or the boring case: demand fell and you would like to stop paying for the box. The machine leaving is not the interesting part. What is on it when it leaves is.

Kubernetes gave the whole industry the word "drain" and, with it, the impression that this is a solved problem you invoke rather than a design you own. For stateless replicas behind a load balancer that is roughly true — kill one, another starts, the deployment controller does the grieving for you. For a host holding a customer's live session, a Postgres primary, or an app with requests in flight, drain is the hard part of running a platform, and the naive implementation is how a routine maintenance window becomes an incident report.

I'm Ajay, I build PandaStack — a Firecracker microVM platform — so my fleet's hosts each own live customer VMs with unsaved state inside them. I have drained hosts on purpose and had hosts drained for me by a cloud provider with no warning, and the difference between those two experiences is entirely down to machinery I had to write. This is that machinery: cordon, classify, drain per class, deadline, verify. Plus the honest limit at the end, which is that there is no version of this where nothing notices.

Drain is three verbs, and conflating them is the first bug

Almost every bad drain I have seen started by treating this as one operation. It is three, they have different failure modes, and they can and should be separately triggerable:

1. Cordon — the host stops receiving new work. Cheap, instant, fully reversible, and safe to do on a hunch. If you are unsure whether a host is misbehaving, cordon it and go look; you have lost nothing but a little capacity.
2. Drain — the existing work leaves, by whatever mechanism each workload's class allows. Slow, partially reversible, and the part with all the interesting failure modes.
3. Terminate — the machine actually goes. Irreversible, and it must be gated on a verification step rather than on a timer, because a timer does not know whether the box is empty.

Keep them separate in your tooling and your runbook. The number of times "I just wanted to stop it taking new work" has escalated to "and then it terminated" because both lived behind one script is not zero. Cordon should be a one-line, low-privilege operation that any on-call engineer can do without a second thought. Terminate should require the verification output as an argument.

Step 1: Cordon, and beware the scheduler's stale view

Cordoning is a flag on the host record that the scheduler honours as an exclusion. Ours lives next to the capacity fields the agent heartbeats in with, so it travels along the same path as everything else the scheduler reads. Do not implement cordon by removing the host row, and do not implement it by making the host stop heartbeating — silence is how you say "dead," and a drain is emphatically not death. If your control plane cannot distinguish a draining host from a dead one, it will do the death thing to it, which is the exact behaviour you were trying to avoid.

Now the trap that catches people, and caught me. Schedulers cache their capacity view, because querying the agents table on every single create is a write-amplification problem you notice at scale. Ours caches for 30 seconds. So you flip the cordon flag, watch the update commit, tell the channel "host is cordoned," and for the next half-minute the scheduler cheerfully keeps placing new workloads on the machine you are about to drain. You then discover this partway through the drain, when the count of things on the host goes up instead of down.

The local refusal is the one I would not skip. The host knows it is draining with zero propagation delay, and a create request that arrives anyway should be rejected with a retryable error so the scheduler picks someone else. That turns a cache-coherency problem into a retry, which is a category of problem you have already solved.

#!/usr/bin/env bash
# cordon.sh -- take a host out of scheduling. Reversible, cheap, boring.
set -euo pipefail
HOST="${1:?usage: cordon.sh <agent-id> [reason]}"
REASON="${2:-manual}"

# 1. Authoritative flag in the control plane.
psql "$PGURL" -v ON_ERROR_STOP=1 <<SQL
UPDATE agents
   SET draining    = true,
       drain_reason = '$REASON',
       drained_at   = now()
 WHERE id = '$HOST';
SQL

# 2. Tell the host directly. It refuses placements immediately, with no
#    dependence on how stale anybody's cached agent list is.
curl -fsS -X POST "http://$HOST:7070/v1/admin/cordon" \
  -H "X-Node-Token: $PANDASTACK_NODE_TOKEN" \
  -d '{"draining":true,"reason":"'"$REASON"'"}'

# 3. Prove it. If the scheduler still lists this host as eligible after
#    its cache window, the cordon did not take and you must not proceed.
sleep 35
psql "$PGURL" -tAc \
  "SELECT id, draining FROM agents WHERE id = '$HOST';"
echo "cordoned; now go look at what is actually on it"

One more property worth designing in: the drain flag must survive an agent restart. If your host daemon comes back up after a deploy advertising full capacity because the flag only lived in memory, you have un-cordoned a machine you were about to power off, and the scheduler will helpfully fill it back up for you.

Step 2: Classify what is actually on the box

You cannot drain a host with one strategy, because "what is on it" is not one kind of thing. Before touching anything, take an inventory and sort it into classes, because each class has a different correct answer and a different cost of being wrong. My classes:

• Ephemeral and nearly done — a sandbox with a short TTL, a CI job, a code-interpreter session with four minutes left on the clock. The right answer is usually to let it expire. Draining it costs a customer a running job; waiting costs you four minutes.
• Stateless and restartable — an app whose whole state is in a git commit and a managed database. Redeploy it elsewhere, flip traffic when the new one is healthy, tear down the old one. This is the class Kubernetes' drain was designed for and the class where it genuinely works.
• Stateful and pausable — a live VM with memory nobody has written down: an agent mid-task, a long-lived dev sandbox, a notebook kernel with an hour of state in it. These can be snapshotted and restored on another host. Not free, but survivable.
• Stateful and irreplaceable — a database primary with a durable volume. Its data is on a disk attached to this specific machine, so "move it" means "rebuild it from the archive somewhere else," which is failover, not migration.
• Genuinely stuck — the workload whose guest is wedged, whose process ignores signals, whose customer is asleep in another timezone. Every fleet has a few. Plan for them explicitly rather than being surprised at the deadline.

The inventory step is not optional and it is not the same as reading your control plane. Do both, and compare them, because they disagree more often than anyone is comfortable admitting. I will come back to that in the verification section, but the short version is: a workload the control plane forgot about is still burning CPU on the host and will still be there after you have declared the drain complete.

Step 3: A different strategy per class

Here is the comparison I keep in the runbook, because at 2am nobody remembers which lever applies to which thing.

• Ephemeral work — Strategy: let TTLs expire, cordon and wait. Cost to the customer: none. Cost to you: you wait for the longest TTL on the box, which is why a cap on TTL is a drainability feature, not just a billing one.
• Stateless apps — Strategy: deploy a replacement elsewhere, health-check it, flip, then tear down the old one. Cost to the customer: zero if the flip is atomic, a few dropped requests if it is not. Cost to you: a full build and boot per app, so budget minutes, not seconds.
• Pausable stateful VMs — Strategy: snapshot memory plus disk, ship it, restore on another host. Cost to the customer: a pause. Cross-host restore lands in the 1.2 to 3.5 second range for us, versus 400 to 750ms if it stays on the same box — which it cannot, because the box is what you are draining. Cost to you: object-storage bandwidth and a guest that notices the time jump.
• Database primaries — Strategy: failover from the archive onto a healthy host, promote, repoint. Cost to the customer: a connection drop and a real gap; our managed Postgres creates take 30 to 90 seconds to reach ready, so treat this as an announced maintenance event, not a silent one. Cost to you: it must be practised, or the first time you run it will be during the incident.
• Stuck workloads — Strategy: notify, wait out the grace budget, then evict with a record of having done so. Cost to the customer: their thing dies. Cost to you: the honesty of admitting in advance that this class exists, instead of pretending the deadline will never be reached.

The snapshot-and-restore path deserves a caveat I would rather you hear from me than discover. When a guest is snapshotted and resumed some seconds later on another machine, it resumes believing no time has passed. Its monotonic clock is frozen through the pause and its wall clock is whatever it was at snapshot time. That is fine for most code and quietly catastrophic for anything doing TLS certificate validity checks, token expiry, or lease arithmetic — I have watched a resumed guest reject a perfectly valid certificate because as far as it was concerned the certificate had not been issued yet. Resync the clock on resume, from inside the guest, as part of the restore path rather than as an afterthought. Otherwise your successful migration produces a VM that is alive and confidently wrong.

Step 4: The deadline, and what happens when you reach it

There are two ways to get this wrong, and they are symmetric. A drain with no deadline never finishes: one wedged workload holds the host forever, the maintenance window closes, the CVE stays unpatched, and the drain quietly becomes a permanently cordoned machine you are still paying for. A drain that sends SIGKILL to everything at the deadline is an outage you scheduled in advance and gave a friendly name.

What actually works is escalation with a per-class grace budget. Not one deadline for the host — a deadline per workload, sized by what that workload costs to lose, all bounded by an outer host-level deadline that is allowed to be reached. Then the drain always terminates, and the things that die are the things you decided in advance were acceptable to kill.

// Drain runs until the host is empty or the outer deadline hits. Two rules
// carry all the weight: every workload gets a grace budget sized by its
// class, and the loop re-reads the inventory each pass rather than trusting
// the list it started with. A drain that acts on a stale snapshot of what
// is running is how you "successfully" drain a host that still has VMs on it.
func (h *Host) Drain(ctx context.Context, deadline time.Time) error {
	ctx, cancel := context.WithDeadline(ctx, deadline)
	defer cancel()

	if err := h.Cordon(ctx); err != nil {
		return fmt.Errorf("refusing to drain an un-cordoned host: %w", err)
	}

	tick := time.NewTicker(5 * time.Second)
	defer tick.Stop()

	for {
		// Re-read from the HOST, not the control plane. The control plane
		// is a claim about what should be running; the host knows.
		live, err := h.InventoryLocal(ctx)
		if err != nil {
			return err
		}
		if len(live) == 0 {
			return h.VerifyEmpty(ctx) // never trust the count alone
		}

		for _, w := range live {
			switch classify(w) {
			case ClassEphemeral:
				// Cheapest possible drain: do nothing, it expires itself.
				// Only force it if its TTL outlives our deadline.
				if w.ExpiresAt.After(deadline) {
					h.evictWithNotice(ctx, w, 60*time.Second)
				}

			case ClassStateless:
				// Build elsewhere, health-check, flip, then tear down.
				// The old one keeps serving until the new one answers.
				h.redeployElsewhere(ctx, w)

			case ClassPausable:
				// Snapshot memory + disk, restore on another host. The
				// guest sees a multi-second time jump; resync its clock
				// on resume or watch it reject valid TLS certificates.
				h.snapshotAndRestoreElsewhere(ctx, w)

			case ClassDurable:
				// A volume does not move. Failover from the archive onto
				// a healthy host, then repoint. Announce this one.
				h.failoverFromArchive(ctx, w)

			case ClassStuck:
				// Escalate: ask nicely, then insist, then admit defeat.
				// SIGKILL is a legitimate final step. It is not step one.
				h.escalate(ctx, w, graceFor(w))
			}
		}

		select {
		case <-tick.C:
		case <-ctx.Done():
			// Deadline reached. Log exactly what is still here, by name,
			// before anything irreversible happens to the machine.
			remaining, _ := h.InventoryLocal(context.WithoutCancel(ctx))
			return fmt.Errorf("drain deadline with %d workloads remaining: %v",
				len(remaining), ids(remaining))
		}
	}
}

// escalate is the only place a workload dies unwillingly, and it is loud
// about it. Signals in order, each with room to actually work.
func (h *Host) escalate(ctx context.Context, w Workload, grace time.Duration) {
	h.notify(w.Owner, "host maintenance: workload will be evicted", grace)
	_ = h.signal(ctx, w, syscall.SIGTERM) // "please stop"
	if h.waitGone(ctx, w, grace) {
		return
	}
	_ = h.signal(ctx, w, syscall.SIGTERM) // "I mean it"
	if h.waitGone(ctx, w, 30*time.Second) {
		return
	}
	log.Printf("EVICT id=%s owner=%s reason=drain-deadline", w.ID, w.Owner)
	_ = h.signal(ctx, w, syscall.SIGKILL)
}

Two things in there are load-bearing and easy to leave out. The loop re-reads the inventory every pass instead of iterating the list it fetched at the start — otherwise a workload that appeared during the drain, because your cordon had not propagated yet, is invisible to the thing whose entire job is emptying the host. And the deadline branch logs what is left by identifier before returning, because "drain timed out" with no names attached is a message that costs you fifteen minutes of SSH archaeology while a maintenance window burns.

Step 5: Verify the host is actually empty

The control plane's view of what is running is a claim. The host's view is a fact, and they disagree more often than you would like. A workload can die without its row being updated. A row can be deleted while its process keeps running, which is my least favourite of the two because nothing is watching it any more and it will not show up in any dashboard you have. A rolling deploy that half-completed leaves both kinds behind. So verification is a reconciliation, not a count.

#!/usr/bin/env bash
# verify-empty.sh -- reconcile the control plane against reality, then
# clean up what the drain left behind. Run this BEFORE you terminate.
set -uo pipefail
HOST="${1:?usage: verify-empty.sh <agent-id>}"

echo "== control plane thinks =="
psql "$PGURL" -tAc \
  "SELECT id, status FROM sandboxes WHERE agent_id = '$HOST'
     AND status NOT IN ('deleted','stopped');"

echo "== the host knows =="
# Actual hypervisor processes. This is the number that matters.
pgrep -af firecracker | awk '{print $1, $NF}'

echo "== orphans =="
# Network namespaces with no matching process. Each one holds a /30 out
# of the 16,384 pre-allocated slots, plus a veth pair and NAT rules.
ip netns list | awk '{print $1}' | grep '^ns-' | while read -r ns; do
  id="${ns#ns-}"
  pgrep -f "$id" >/dev/null || echo "orphan netns: $ns"
done

# Loop devices still attached to deleted backing files. These are the
# ones that quietly eat a disk you thought you had freed: the file is
# unlinked, the space is not returned until the loop is detached.
losetup -a | grep -i deleted || true

# Copy-on-write rootfs directories with no live VM. Reflinked clones
# look small in du until the parent template is removed underneath them.
for d in /var/lib/pandastack/vms/*; do
  id="$(basename "$d")"
  pgrep -f "$id" >/dev/null || echo "orphan vm dir: $d"
done

Orphan cleanup is the unglamorous half of drain and the half that determines whether you can reuse the host. Network namespaces are the classic: each sandbox on our hosts gets a dedicated namespace, a veth pair, a tap device and NAT rules, carved out of a pool of 16,384 pre-allocated /30 subnets. Leak them during a drain and the host comes back from maintenance with a slowly shrinking address pool and no error message about it, until one day allocation starts failing for reasons that look nothing like the drain that caused them.

Loop devices are the other one, and the way they hurt is specific. You delete a VM's backing file, the disk usage does not go down, and you spend a while distrusting df. The file is unlinked but a loop device still holds a reference, so the space comes back only when the loop is detached. On a host you drained because the disk was filling, that is a genuinely funny failure to hit.

The honest limits

There is no live migration in this design. I do not stream a running guest's dirty pages to another host while it keeps executing, converge, and cut over in a way nobody perceives. That capability exists, it is genuinely impressive, and it costs a memory-tracking apparatus plus a network fabric assumption I have not been willing to take on. What I have instead is snapshot and restore, which means the guest stops, its memory is written down, and it starts again somewhere else — a pause, measured in seconds when it crosses a host boundary.

So: you cannot drain a host without something on it noticing. That is the honest statement, and anyone telling you otherwise is either running live migration or has not looked closely. The goal is not to make "noticing" impossible. The goal is to make it mean a brief pause and a resumed session rather than a lost one, and to make the cases where a workload genuinely dies be cases you chose in advance, announced, and can point at in a log line with a name and a reason attached.

The other limit worth naming: everything above assumes you get to choose when the host goes. Sometimes you do not. A provider maintenance event that recreates an instance underneath you does not read your drain flag, and the recovery path is a different discipline entirely — durable artifacts in object storage, so a host that vanishes without ceremony can be rebuilt from something that outlived it. Drain is the planned path. Make sure it is not your only one.

The summary

Split drain into cordon, drain and terminate, and let a human do the first one on a hunch. Make the cordon real at the host, not just in a table a cache is lying about. Inventory what is on the machine and sort it by class, because "let it expire," "redeploy elsewhere," "snapshot and restore," and "failover from archive" are four different operations with four different costs. Give each workload a grace budget and the host an outer deadline that is allowed to be reached, escalating through signals rather than opening with the last one. Then verify by reconciling the control plane against what is actually running, and clean up the namespaces, loop devices and directories nobody else will. Do that and a maintenance window is a paragraph in a status page. Skip the verification step and it is an incident review.