Lease vs Heartbeat: How a Fleet Decides a Node Is Dead

A scheduler's job looks like arithmetic: rank the hosts, pick the one with the most free memory, place the workload. But the arithmetic happens downstream of a harder question — which hosts are even eligible to be scored? Eligibility is a claim about a machine you cannot see, inferred from a message it sent some time ago, over a network allowed to be slow for reasons neither of you controls. Every genuinely interesting outage I have had lives in that gap.

I'm Ajay; I built PandaStack, a Firecracker microVM platform, so I run a fleet of Linux hosts that each own live customer VMs and a control plane that decides, continuously, which of them may be handed new work — and, separately, which may still touch the work it already has. This post is the two mechanisms people confuse for one another, the limit you cannot engineer around, and the four ways I have watched the decision go wrong. One of them is caching, and it is so cheap to get wrong that it deserves a name.

Two mechanisms, two different assertions

A heartbeat is a node saying "I was alive at time T." It's a push on a fixed interval, and a convenient place to carry payload — on PandaStack each agent's heartbeat brings capacity metadata: free CPU, free memory, how many of the 16,384 pre-allocated network slots remain, and flags like whether that host can stream guest memory on demand from object storage. The scheduler scores on that metadata. The heartbeat's failure mode has one shape: staleness. The last one you have is older than you would like, and you have to decide what that means.

A lease is a different animal: a time-bounded claim of ownership. "I own these workloads until T plus the TTL, and to keep owning them I must come back and renew." Also a push, but with an expiry contract attached, and the contract obligates the holder as much as it protects it. Its failure mode is not staleness; it is expiry while the node is perfectly fine. The network hiccuped, the host noticed nothing, and its right to act quietly lapsed underneath it.

Be pedantic about this, because they are not the same question. A heartbeat answers "is it there?" A lease answers "may it still act?" You can answer the first yes and the second no, and that combination — alive but fenced — is exactly the state a safe system must be able to express. Most teams ship one mechanism doing both jobs badly: a heartbeat treated as authorization, or a lease treated as a health signal. Then a network blip either drops live VMs or, worse, doesn't.

You cannot tell a dead node from a slow network

Here is the part no engineering removes. When a message stops arriving, no local observation distinguishes "the process died" from "the process is fine and the packets are late." That is not a gap in your monitoring; it is a property of asynchronous networks, which is why the literature is full of failure detectors described as explicitly unreliable. You are not building a system that avoids being wrong — you are building one that chooses which way it is wrong.

There are two directions. Fail toward availability: keep the node eligible and assume the silence is transient — get it wrong and you place work on a corpse, so the create hangs and you learn about it from a support ticket. Fail toward safety: declare it dead, stop scheduling to it, fence it out of shared state — get it wrong and you just evicted live customer workloads from a machine that was doing its job.

Choose by asking what the wrong answer costs you. If a bad placement means one job retries elsewhere three seconds later, fail toward availability without guilt. If your "declare dead" path tears down running VMs with unsaved state in them, be conservative about declaring death and aggressive about making it reversible. That asymmetry is the real input to your timeout policy, not a number from a 2014 blog comment.

Timeouts are a policy decision, so pick a ratio

Every liveness timeout I have seen defended as "the right value" was a ratio wearing a costume. Don't pick a magic constant; pick two quantities and let the third fall out.

1. Renew interval — how often the node attempts to prove itself. Chosen against your write budget: N nodes times one write per interval is load you are signing up for permanently.
2. Expiry as a multiple of the interval — literally how many consecutive misses you tolerate before the claim lapses. This is the knob that encodes your availability-versus-safety choice, and it is the one people set to 2 and then get paged about.
3. The detection window that falls out — expiry plus however long the control plane takes to notice it. This is the number your SLO cares about, and it is always larger than the expiry alone.

On PandaStack I run a 10-second refresh against a 60-second expiry, which tolerates five consecutive missed renewals before a node loses its claim. That is a deliberate bias: wrongly fencing one of my hosts evicts live customer microVMs, so I would rather absorb a minute of a genuinely dead host holding a claim it cannot use. For stateless HTTP handlers a much tighter ratio would be defensible. These are configuration choices, not laws.

// Leaser renews this node's ownership lease. The contract is not "try to
// keep the lease alive" -- it is "if I cannot renew, I stop acting on the
// workloads the lease covers, BEFORE it expires, not after."
type Leaser struct {
	db       *sql.DB
	nodeID   string
	interval time.Duration // 10s -- how often we try to renew
	ttl      time.Duration // 60s -- how long a successful renew buys us
	gen      atomic.Uint64 // fencing token: monotonically increasing
	valid    atomic.Bool   // may this node act right now?
	expires  atomic.Int64  // unix nanos of current expiry
}

func (l *Leaser) Run(ctx context.Context) error {
	t := time.NewTimer(0)
	defer t.Stop()

	for {
		select {
		case <-ctx.Done():
			// Planned shutdown is not death. Hand the lease back so the
			// control plane learns in milliseconds instead of 60 seconds.
			l.release(context.WithoutCancel(ctx))
			return ctx.Err()
		case <-t.C:
		}

		// Never let a renew outlive the window it is renewing. A renew
		// still in flight after `interval` is already a failed renew.
		rctx, cancel := context.WithTimeout(ctx, l.interval)
		gen, expiresAt, err := l.renew(rctx)
		cancel()

		if err != nil {
			// The lease is a clock, not a mood. Past expiry we are fenced
			// whether or not we agree, so drop privileges locally too.
			if time.Now().UnixNano() > l.expires.Load() {
				l.valid.Store(false)
				log.Printf("lease EXPIRED node=%s: quiescing, refusing new work", l.nodeID)
			}
			t.Reset(backoff(l.interval))
			continue
		}

		l.gen.Store(gen)
		l.expires.Store(expiresAt.UnixNano())
		l.valid.Store(true)
		t.Reset(jitter(l.interval)) // do NOT let the fleet renew in lockstep
	}
}

// jitter spreads renewals so 200 nodes don't hit the lease table on the
// same tick and then all miss together when it hiccups.
func jitter(d time.Duration) time.Duration {
	return d/2 + time.Duration(rand.Int63n(int64(d)))
}

// MayAct is the guard every mutating operation calls first. It returns the
// fencing token to pass down to the write, so a slow action started under
// lease N cannot land after lease N+1 was granted elsewhere.
func (l *Leaser) MayAct() (gen uint64, ok bool) {
	if !l.valid.Load() || time.Now().UnixNano() > l.expires.Load() {
		return 0, false
	}
	return l.gen.Load(), true
}

Three details matter more than the structure. The renew carries a context timeout no longer than the interval — one still in flight when the next is due is already a failed renew, and stacking them thunders the lease store exactly when it is struggling. The retry is jittered; nothing turns a two-second database blip into a fleet-wide simultaneous expiry faster than 200 nodes renewing on the same tick. And the failure branch fences the node locally. Waiting to be told you lost your lease misses the point: the entity that would tell you is the one you cannot reach.

Split brain, and why ownership has to be written down

Now the failure that actually cost me something. Split brain is usually explained as two nodes both believing they are leader, which makes it sound like a consensus problem you fix with Raft. In a fleet that places workloads it shows up much dumber: the control plane runs a mutating query against shared state without asking who owns the rows.

The shape is always the same. Someone writes a reaper, or a cleanup pass in a deploy, that says "find workloads matching this condition and delete them." It is flawless on a single host, because there every row is yours. Add a second host, run it from both, and one node deletes the other's live customer workloads — not from a race, not from a partition, but because the query said "these rows exist" and nothing said "these rows are not yours."

-- DANGEROUS. "These rows exist" is not "these rows are mine." Run this
-- from a control plane that talks to every host, or from a peer node
-- during a rolling deploy, and you have just deleted another host's live
-- customer workloads. Nothing in the query objects. It did what you said.
DELETE FROM sandboxes
WHERE  status     = 'running'
  AND  updated_at < now() - interval '10 minutes';

-- SAFE. Scoped by ownership AND by the fencing token the caller held when
-- it decided to act. Both predicates are cheap; together they turn a
-- catastrophic class of bug into a no-op.
UPDATE sandboxes
   SET status    = 'stopping',
       lease_gen = $3
 WHERE id             = $1
   AND owner_agent_id = $2   -- I may only touch what I own
   AND lease_gen     <= $3;  -- ...and only if I am not a stale writer

-- 0 rows affected means "not yours, or you are stale." That is a normal
-- outcome, not an error. Count it, log it, and do nothing else.

The fix is an ownership column on the record and every mutating query scoped by it. It is boring: one column, one predicate, one index. And it converts an entire category of catastrophic bug into a query that affects zero rows — exactly the outcome you want, because "I tried to touch a workload that isn't mine and nothing happened" is a system working correctly, not an error to page on. Keep a counter on it anyway: a spike in zero-row mutations means something upstream is confused about ownership.

Fencing tokens belong here too. Ownership answers "is this mine?"; a fencing token answers "is my claim still current?" A monotonically increasing lease generation, issued on each renew and threaded into every write, closes the gap where a node starts an action holding a valid lease, stalls inside a slow object-storage call, and completes it against state another node has since legitimately taken over. With a generation on the row, that stale write loses the comparison and lands nowhere.

The caching trap: never cache a negative

This is my favourite failure, because it is not a distributed-systems failure at all. It is a category error, committed by careful engineers doing an obviously reasonable optimisation, and it manufactures outages on a healthy fleet. The setup: your scheduler queries the agents table on every create, so you cache the result for a few seconds. Entirely sensible — capacity numbers are approximate anyway, and a stale view of free memory costs nothing worse than a slightly suboptimal placement. Positive caching degrades gracefully. That is the whole reason the optimisation is attractive.

The trap is the nodes absent from that snapshot. A host that joined inside the cache window is not in your list. A host whose lease was renewed a moment after you snapshotted looks, to the cache, exactly like one with an expired lease. Downstream, "not in the list" becomes "not eligible," which becomes "dead" — and you return 503s while that host sits there healthy with free memory, wondering why nobody calls. I have shipped this. The fleet was fine; the cache was lying, and the lie ran in the direction that fails closed.

Never serve a negative answer from a cache when that negative answer triggers an irreversible or user-visible action. Re-read authoritatively before you fail closed.

The asymmetry is the insight. A stale positive — "this host has 12 GB free" when it now has 9 — produces a slightly worse decision. A stale negative — "there are no eligible hosts" — produces a rejection, a page, or a teardown. Same cache, same staleness, wildly different blast radius. So serve the fast path from cache, and when the cached answer is about to make you reject work or fence a node, spend the round trip. You pay only where being wrong is expensive.

From the client side none of this is distinguishable from a genuinely full fleet — both look like "no eligible host." An argument for callers treating capacity errors as retryable, and a better one for you not manufacturing them:

import random
import time

from pandastack import Sandbox


def create_with_retry(template="base", attempts=4):
    """From the client, a false-positive exclusion and a genuinely full
    fleet are indistinguishable: no host was eligible, here is a 503.

    So treat 'no capacity' as retryable-but-not-forever. Backoff plus
    jitter, a hard attempt cap, and a real exception at the end -- an
    infinite retry loop against a fleet-wide liveness bug is how a small
    incident becomes a large one.
    """
    delay = 0.25
    for i in range(attempts):
        try:
            return Sandbox.create(template=template, ttl_seconds=900)
        except Exception:
            if i == attempts - 1:
                raise
            time.sleep(delay + random.random() * delay)
            delay *= 2


with create_with_retry() as sbx:
    r = sbx.exec("uname -a", timeout_seconds=30)
    print(r.exit_code, r.stdout.strip())

Three designs, compared

• Detection speed — Heartbeat-only: fast and tunable; you set the staleness threshold. Lease-only: bounded by the TTL, never faster than expiry plus polling delay. Heartbeat + lease + ownership: fastest, because the questions are separate — a stale heartbeat pulls a host out of scheduling while its lease keeps running.
• Split-brain safety — Heartbeat-only: none. Nothing stops a stale node from acting, or a peer from acting on its workloads. Lease-only: good against two nodes claiming one workload, useless if your mutating queries never check the claim. Heartbeat + lease + ownership: the lease bounds the right to act, the ownership predicate enforces it at the write.
• False-positive cost — Heartbeat-only: low; it usually just removes a host from scheduling, reversible the moment it reports again. Lease-only: high; expiry means eviction or takeover of live work. Heartbeat + lease + ownership: lowest — the cheap reversible action and the expensive irreversible one fire on different signals.
• Complexity — Heartbeat-only: trivial; one table, one timer. Lease-only: moderate; renewal, expiry, clock discipline, self-fencing. Heartbeat + lease + ownership: highest, but the marginal cost over lease-only is one column, one predicate, one token. The hard part is discipline, not code.

If you build one thing, build the heartbeat — you cannot schedule at all without capacity data. If you already run workloads whose destruction would upset a customer, you needed the ownership column yesterday, and it is the cheapest of the three by a wide margin.

What to instrument (the tail, not the mean)

Liveness bugs hide in averages. Mean heartbeat age across a fleet is a flat, boring line right through the incident, because the two misbehaving hosts are drowned by the fifty that are fine. Instrument the distribution and the exclusions instead.

• Distribution of heartbeat age — p50, p95, p99, max. The max is the one that pages. A p99 creeping toward your staleness threshold fleet-wide is your control-plane database warning you quietly before it warns you loudly.
• Lease renewal latency, measured on the node — how long the renew takes, not just whether it succeeded. Renewals succeeding at four seconds against a ten-second interval are a fleet-wide outage with a fuse on it.
• Count of nodes excluded from scheduling, broken down by reason — stale heartbeat, expired lease, draining, insufficient capacity, missing artifact. An undifferentiated "eligible hosts: 3" gauge tells you something is wrong and nothing about what.
• "Excluded then immediately reappeared" — incremented when a node re-enters the eligible set within one detection window of being removed. This is your false-positive detector and the most valuable metric on the list, because a healthy fleet flapping in and out of eligibility is invisible in every other measurement you have.

-- The query I actually run when someone says "the fleet is rejecting
-- creates." Sort by heartbeat age and the answer is usually visible in
-- the first two rows.
SELECT a.id,
       a.region,
       now() - a.heartbeat_at                       AS heartbeat_age,
       l.expires_at - now()                         AS lease_remaining,
       (now() - a.heartbeat_at > interval '60 seconds') AS stale,
       (l.expires_at IS NULL OR l.expires_at < now())   AS fenced,
       a.cpu_total    - a.cpu_used                   AS free_cpu,
       a.memory_mb_total - a.memory_mb_used          AS free_mem_mb,
       a.draining
FROM   agents a
LEFT   JOIN agent_leases l ON l.agent_id = a.id
ORDER  BY heartbeat_age DESC;

-- Read it as three columns, not one: `stale` says nobody has heard from
-- it, `fenced` says it may no longer act, `draining` says we did this on
-- purpose. A host that is fenced but not stale is the interesting case --
-- it is alive and talking, and we took its keys away anyway.

Draining is not dying, and your deploy must know the difference

Last one, and it bites during routine operations rather than incidents. A planned drain must be an explicit state, not the absence of a heartbeat.

The failure: you ship a new version of the host agent. The deploy stops the old process, and for however long the new binary takes to start, open its database handle, and renew, that host stops heartbeating. If the only thing your control plane infers from silence is death, you have just told it that a host holding live customer VMs died — during a deploy you performed on purpose. That is anywhere between a brief scheduling gap and a fleet-wide eviction rolling across your infrastructure one host at a time, in an orderly fashion, exactly as designed.

The fix is to make intent explicit and separable from liveness. A draining host sets a flag the scheduler honours — no new placements — while its lease keeps renewing and its existing workloads stay untouched. Restarting the agent must not signal anything about the VMs it supervises: microVMs are separate processes with their own lifecycle, and a supervisor restart should reattach, not reap. The drain flag must survive the restart, so the new process doesn't come up advertising capacity thirty seconds before you power the machine off.

When this is overkill

I would rather you skip all of this than cargo-cult it, so here is where it genuinely does not pay for itself.

• You have one node. No split brain with one writer, no ownership ambiguity, no placement decision. A process supervisor and an alert on "is it up" is the correct amount of machinery; adding leases buys you a new way to have an outage.
• A stale placement is harmless. If work is idempotent, queued, and retried elsewhere on failure, failing toward availability costs a few seconds and queue semantics have already solved your problem.
• Your workloads are stateless and cheap to recreate. Fencing protects state that two writers would corrupt; with none, kill-and-reschedule beats any lease you would write.
• You are on a platform that already does this. Kubernetes, Nomad, and every managed database worth the name ship opinionated liveness machinery; a second layer on top produces two systems that disagree, which is worse than either alone.
• The ownership column, however, is never overkill. If more than one process can issue a mutating query against a shared table, add it now. It's one column.

Where PandaStack fits: this is the layer underneath the thing customers buy. Agents heartbeat with capacity metadata, leases refresh on a short interval against a longer expiry, stale hosts are excluded from scheduling, and every mutating query against a sandbox row is scoped by the owning agent — that last one exists specifically because it once wasn't. The visible product is a sandbox coming up by snapshot restore at a p50 of 179ms. The invisible one is that a rolling agent deploy doesn't take your VMs with it.

The takeaway: heartbeat and lease answer different questions and you need both; every timeout is a bet on which failure mode you prefer; ownership is a column, not an architecture. And if you take one thing, take the caching rule — positive caching degrades gracefully, negative caching manufactures outages.