Leader Election When Your Compute Can Be Cloned

Somewhere in every platform there is a thing that must happen exactly once. A nightly billing run. A schema migration. A reconcile loop that decides which VMs get reaped. A consumer draining a queue where processing a message twice means charging a card twice. The requirement is embarrassingly simple to state and the failure is embarrassingly expensive to explain, which is a combination that shows up in a lot of incident reviews.

For most of computing history the answer was "run one copy of it." You had a box, the box ran the cron, and if the box was down the cron did not fire, which everybody understood as an outage rather than a correctness bug. That answer stopped working the moment you had two hosts. It got worse when deploys became rolling. And it becomes genuinely interesting on infrastructure where compute is ephemeral, restorable from a snapshot, and forkable — because on that kind of infrastructure, "one process" is not a thing you can assert. It is a thing you have to continuously prove.

Why "just run one instance" is not a design

The single-replica plan has one virtue: it is true most of the time. That is also its entire problem, because the plan does not degrade — it inverts. Here is the list of ordinary, non-exotic events that produce two instances of your singleton, none of which require a network partition or a Byzantine fault or anything else you would put in a conference talk.

• A rolling deploy. The new pod is up and healthy before the old one has finished draining. For a window measured in seconds to minutes, both are running your cron loop. Every orchestrator does this on purpose — overlap is how you get zero-downtime deploys — so this is not a misconfiguration, it is the feature working.
• A restart that is not a stop. A process that hangs on shutdown, gets SIGKILLed after the stop timeout, and is replaced while its old self was still mid-transaction. The supervisor believes there is one; the database has seen two.
• A liveness probe that lies. The health endpoint returns 200 because the HTTP server is fine while the worker goroutine deadlocked twenty minutes ago. Nothing restarts, nothing fires, and you find out from a customer.
• An operator with a keyboard. Somebody runs the migration by hand "just to check" while the automated runner is on its scheduled tick.
• A pause and a resume. A VM stopped for forty seconds and then continued from exactly the instruction it was on, with every in-memory belief about the world intact and forty seconds out of date.
• A snapshot restored more than once. This one has no analogue in the single-server world at all, and it is the one worth the rest of this post.

Notice what these have in common: in every case, each individual process is behaving correctly according to everything it can observe. Nobody is buggy. The system is buggy, and the bug is that liveness of a process is not a fact any process can establish about itself.

The clone problem: a snapshot is a leader factory

On snapshot-based infrastructure, the create path is a restore. PandaStack has no warm pool of idle VMs — every sandbox create restores a baked Firecracker snapshot, which is why a create lands around 179ms at p50 rather than the roughly 3 seconds a genuine cold boot takes. That is a lovely property for boot latency and a hostile one for singletons, because a snapshot is a reusable, byte-identical starting state. Restore it once and you have a process. Restore it twice and you have two processes with the same memory, the same file descriptors' worth of beliefs, the same cached lease, the same "I am the leader" boolean sitting in the same heap address.

The classic restart case is much kinder than this, and it is worth being precise about why. A restarted process comes up empty. It has to go and ask something who the leader is, and asking is where the correctness lives. A restored process never asks. It resumes mid-sentence with the answer it already had, and the answer was true when it was written down.

Forking is the same hazard on a different medium. PandaStack's fork is a disk fork: the parent is paused, its rootfs is copied for a consistent view, the parent resumes, and each child boots fresh from that copy — same filesystem, no inherited processes or RAM. Children land in 400 to 750ms on the same host, 1.2 to 3.5 seconds across hosts. So the children do not inherit a leadership variable in memory. They inherit something arguably worse, because it is durable: whatever the parent had written to disk. A PID file. A lock file. A `/var/run/leader.json`. A cached lease token in a SQLite file. A machine-id that half your tooling uses as the holder identity. Fork ten workers off a parent that was the leader and you have ten workers whose disks all say they are the leader, and they will all wake up and act on it.

Leases: a lock, but with an expiry and a receipt

The primitive that actually works here is not a lock. It is a lease. The distinction is one word and it is the whole subject: a lock is held until released, and a lease is held until it expires. A lock without an expiry is a deadlock waiting for a pager, because the release path runs on the machine you are trying to defend against losing. Every distributed lock that ever wedged production wedged because the holder died between acquire and release, and the only recovery was a human with a psql prompt at an unsociable hour.

A usable lease has three parts and you need all three.

• An expiry, evaluated by the store rather than by the holder. `expires_at < now()` where `now()` is the database's clock, not the worker's. The instant you let each worker judge its own expiry against its own clock, you have N clocks and a correctness argument that depends on all of them agreeing.
• A renewal, run at a small fraction of the TTL. If the lease is 30 seconds, renew every 10. That ratio is not superstition: it is how many consecutive renewal failures you can absorb before you lose the lease, and one is not enough on any network you did not personally build.
• A fencing token: a number that goes up on every change of holder and is handed back to the acquirer. Without this, everything above is a very good heuristic. With it, it is a correctness argument.

A lock says "nobody else may proceed." A lease says "nobody else may proceed for the next 30 seconds, and here is a receipt proving you were the holder when you started." Only the second one survives being suspended.

Fencing tokens, explained properly

Fencing is the part people skip, and it is the part that makes the rest sound rather than merely likely. The idea: every time the lease changes hands, a monotonically increasing integer is incremented. The acquirer receives it. Every write to the protected resource carries it. And — this is the load-bearing clause — the resource itself rejects any write whose token is lower than the highest it has already accepted.

Why it matters is the scenario nobody plans for and everybody experiences. Consider the sequence:

1. Worker A acquires the lease with token 7 and starts a 90-second billing run.
2. Worker A stops. Not crashes — stops. A stop-the-world GC pause, a host under memory pressure, a hypervisor descheduling the vCPU, or on this kind of platform, a hibernate. From inside A, nothing happened.
3. The lease expires. Worker B acquires it with token 8 and does the billing run correctly.
4. Worker A resumes, mid-instruction, still holding a local variable that says it is the leader, and finishes writing its results.
5. Without a fence, step 5 lands. Two billing runs, the second one silently overwriting or duplicating the first. With a fence, step 5 is a write carrying token 7 against a resource that has accepted 8, and it fails — loudly, deterministically, with zero rows updated.

The reason this is worth arguing about on ephemeral infrastructure specifically is that step 2 stops being exotic. On a normal server, a 40-second stall is a bad day you write a postmortem about. On a platform where pause, hibernate and wake are ordinary API calls that users and autoscalers make deliberately, a process being frozen for a minute and then continuing is not an anomaly. It is a product feature. Any design whose safety rests on "a process would never be stopped for that long" is a design that will be violated on purpose, by your own control plane, on a Tuesday.

The corollary that trips people up: fencing only works if the resource can check it. A Postgres table can. An S3 object can, with a conditional write. A filesystem generally cannot, and a `curl` to a third-party API definitely cannot. If your singleton's side effect is "send an email," no lease design in this post makes that exactly-once — you need idempotency keys on the receiving side instead, which is the same insight wearing a different hat.

Clocks, and what a restored guest believes the time is

Lease expiry is a statement about time, so it is worth being explicit about which clocks you are trusting and how badly they can lie.

Start with the wall clock inside a restored VM, because it is worse than people expect. Firecracker resumes vCPUs with `CLOCK_REALTIME` frozen at the instant the snapshot was taken, and nothing in the guest fixes it on its own — these microVMs have no RTC on x86 and the templates do not run NTP. A guest restored from a snapshot baked three weeks ago genuinely believes it is three weeks ago. That is not a thought experiment: it shipped as a live incident here, where an upstream rotated onto a TLS certificate issued after the seed was baked and every app deploy's `git clone` died with `server certificate verification failed`, because the guest was rejecting a perfectly valid certificate as not yet valid. The fix is a `date -u -s @<epoch>` pushed over the guest exec bridge on every restore-family path — snapshot-restore create, resume, and wake — and it is deliberately best-effort, because a stale clock is degraded service while a failed create is an outage.

Now the trap that catches the careful people. Having been burned by wall clocks, everybody reaches for a monotonic clock to measure their renewal interval, which is normally the right instinct. It is the wrong instinct here. A suspended guest's monotonic clock does not count the suspension — the vCPUs were not running, so nothing ticked. Your renew loop wakes up, computes `time.Since(lastRenew)`, gets 200 milliseconds, and concludes it is comfortably within its 30-second lease. The database disagrees by 39 seconds. The monotonic clock is not broken; it is faithfully reporting elapsed execution time, which is simply not the quantity lease expiry is denominated in.

The implementations, ranked

1. A leases table in the Postgres you already have

For the overwhelming majority of singleton jobs, this is the correct answer, and the reason is not technical elegance. It is that you already run this database, you already back it up, you already page on it, and it is already the thing your job writes its results to — which means the lease and the fenced write can be in the same transaction, against the same clock, with no cross-system consistency argument to get wrong.

-- One row per named singleton. `holder` is advisory and mostly for humans
-- reading the table at 3am; `token` is the part that carries correctness.
CREATE TABLE job_leases (
  name        text        PRIMARY KEY,
  holder      text        NOT NULL,
  token       bigint      NOT NULL,
  expires_at  timestamptz NOT NULL
);

-- ACQUIRE-OR-RENEW, atomically, in one statement. There is no SELECT followed
-- by an UPDATE for a second process to slip between, and `now()` is evaluated
-- by the database -- one clock, not one clock per worker.
INSERT INTO job_leases AS l (name, holder, token, expires_at)
VALUES ($1, $2, 1, now() + $3::interval)
ON CONFLICT (name) DO UPDATE
   SET holder     = EXCLUDED.holder,
       -- A renewal by the same holder keeps the token. A takeover bumps it.
       -- That CASE expression is the entire fencing scheme.
       token      = l.token + (CASE WHEN l.holder = EXCLUDED.holder THEN 0 ELSE 1 END),
       expires_at = EXCLUDED.expires_at
 WHERE l.holder = EXCLUDED.holder
    OR l.expires_at < now()
RETURNING token, expires_at;

-- Zero rows returned means somebody else holds an unexpired lease. That is a
-- normal, boring outcome: sleep and try again. It is not an error and it must
-- not be retried in a tight loop, because the tight loop is how you turn one
-- follower into a denial-of-service against your own database.

And the enforcement side, which lives with the resource:

-- The fence is enforced by the RESOURCE, not by the leader. The leader can be
-- wrong about whether it is the leader -- that is the whole premise. The
-- resource cannot be wrong about which token it last accepted.
UPDATE invoice_run_state
   SET last_run_at = now(),
       run_by      = $1,
       fence       = $2          -- the token returned by the lease statement
 WHERE job = 'nightly-invoice-run'
   AND fence < $2;               -- monotonic: a stale token can never win

-- 0 rows updated means you were superseded while you were not looking. Stop.
-- Do not log a warning and carry on. Carrying on is precisely the bug this
-- column exists to prevent, and it is the one that double-charges customers.

Postgres advisory locks (`pg_try_advisory_lock`) are the tempting shortcut and they are subtly worse for this. They are tied to the session, so they release when the connection drops — which sounds like an expiry but is not one, because a wedged process holding an open TCP connection keeps the lock forever while a healthy process behind a connection pooler can lose it without noticing. They also have no token, so there is nothing to fence with. They are excellent for guarding a migration inside a single short-lived connection and a poor fit for a long-running leader.

2. etcd or Consul, when you genuinely need the semantics

These are purpose-built and they give you two things a table does not: a watch, so a follower learns about the vacancy in milliseconds instead of on its next poll, and a session or lease abstraction with a keepalive already implemented. etcd's revision number is a ready-made fencing token; Consul sessions give you the same shape. If you are already running Kubernetes you have an etcd, and the client libraries for this pattern are mature.

The cost is that you are now operating a consensus system, and consensus systems are unforgiving about disk latency and clock skew in ways that a table in your application database is not. Reach for these when failover speed genuinely matters — sub-second, not sub-minute — or when you need many singletons coordinating and watches beat polling. Do not reach for them because the leases table felt insufficiently serious.

3. Redis, with an honest note about Redlock

`SET key value NX PX 30000` on a single Redis instance, with a random value you check before deleting, is a perfectly reasonable lease for a job whose worst-case double-execution is annoying rather than expensive. It is fast, everybody has one, and the expiry is native. Pair it with a fencing token — Redis `INCR` gives you a monotonic counter — and check that token at the resource, and you have covered the paused-leader case.

What deserves care is Redlock, the multi-instance algorithm. It has been publicly disputed for years — Martin Kleppmann argued it relies on timing assumptions that do not hold under GC pauses and clock jumps, and Salvatore Sanfilippo responded defending the design and its assumptions. That argument is not settled by this blog post and I am not going to pretend it is. The practically useful summary: the disagreement is about whether the algorithm is safe *without* fencing at the resource, and both camps agree that a fencing token checked by the resource makes the question much less interesting. If you have a fence, single-instance Redis is fine for a large class of jobs. If you do not have a fence, no lock implementation on this list saves you, and the multi-instance one just makes the failure harder to reason about. Read both sides before you build on it.

4. "Just one replica and hope"

Worth naming honestly, because it is a legitimate choice for a genuinely idempotent job — one where running twice produces the same result as running once, and where the downtime of running zero times is acceptable. If your reconcile loop reads desired state and converges actual state, running two of them is wasteful and harmless. The failure is not choosing this. The failure is choosing it for a job that is not idempotent and then discovering the distinction from the finance team.

The comparison, compressed:

• Postgres leases table — Fencing: native, a bigint column and one CASE expression. Failover: as fast as your poll interval, so seconds. Ops cost: zero new systems. Failure mode: your database is a single point of failure, which it already was. Use when: almost always.
• Postgres advisory lock — Fencing: none, there is no token to hand out. Failover: on connection drop, which is not the same as on death. Ops cost: zero. Failure mode: connection poolers and wedged-but-connected processes. Use when: guarding a short migration inside one connection.
• etcd or Consul — Fencing: native (etcd revisions, Consul sessions). Failover: sub-second via watches. Ops cost: a consensus cluster to operate, back up and page on. Failure mode: unforgiving about disk latency and quorum loss. Use when: failover speed genuinely matters or you already run one.
• Redis SET NX PX — Fencing: bolt-on via INCR, and you must add it. Failover: fast. Ops cost: low if you already have Redis. Failure mode: single-instance means a failover can lose the key entirely; Redlock's safety is publicly contested. Use when: the job is cheap to double-run and speed matters.
• One replica and hope — Fencing: none. Failover: manual, at the speed of a human reading a page. Ops cost: nominally zero, occasionally enormous. Failure mode: rolling deploys and silent hangs. Use when: the job is truly idempotent and you have written down why.

The renew loop, with the sharp edges left in

The acquire statement is the easy half. The loop around it is where the design decisions live, and there are exactly three that matter: how often you renew relative to the TTL, what you do when a renewal returns an error rather than an answer, and whether the work checks the token or merely the boolean.

// leaseLoop holds a named singleton lease and publishes the current fencing
// token to whoever is doing the work. The contract is deliberately brutal: if
// a renewal has not landed comfortably before the lease expires, we revoke
// leadership locally BEFORE the expiry we promised the database.
package singleton

import (
	"context"
	"database/sql"
	"log/slog"
	"sync/atomic"
	"time"
)

const (
	leaseTTL = 30 * time.Second // how long the DB grants it for
	renewIn  = 10 * time.Second // renew at TTL/3, so two renewals may fail
	// The gap between renewIn and leaseTTL is your entire tolerance for a
	// slow query, a GC pause, a paused VM, or a bad five seconds of network.
	// A 1:3 ratio is the usual starting point. Tightening it below 1:2 means
	// one unlucky round-trip demotes a perfectly healthy leader.
)

// Token is 0 whenever we are not the leader. Every unit of work reads it once
// at the start and passes it to the fenced write at the end.
type Lease struct {
	db     *sql.DB
	name   string
	holder string // MUST be unique per process instance -- see the fork note
	token  atomic.Int64
}

func (l *Lease) Token() int64 { return l.token.Load() }

func (l *Lease) Run(ctx context.Context) {
	t := time.NewTicker(renewIn)
	defer t.Stop()
	for {
		// Deliberately re-derived from the database on every tick. We never
		// compute "I am still the leader" from local elapsed time, because a
		// guest that was paused for 40 seconds measures that as roughly zero.
		tok, err := l.acquireOrRenew(ctx)
		switch {
		case err != nil:
			// Unknown, not "no". But unknown must be treated as no, because
			// the alternative is two leaders. Fail closed on the local side;
			// the fencing token protects the resource if we are wrong.
			slog.Warn("lease renew failed, standing down", "name", l.name, "err", err)
			l.token.Store(0)
		case tok == 0:
			l.token.Store(0) // somebody else has it; that is fine
		default:
			if prev := l.token.Swap(tok); prev != 0 && prev != tok {
				// We lost it and got it back with a new token. Anything still
				// in flight under `prev` is now fenced and will fail its write.
				slog.Warn("lease token advanced, in-flight work is fenced",
					"name", l.name, "old", prev, "new", tok)
			}
		}
		select {
		case <-ctx.Done():
			l.release(context.WithoutCancel(ctx))
			return
		case <-t.C:
		}
	}
}

// release is an optimisation, not a correctness mechanism. A clean shutdown
// hands the lease over in milliseconds instead of leaseTTL. A dirty one -- a
// SIGKILL, a reclaimed host, a microVM that simply stopped existing -- skips
// it entirely, which is exactly why the expiry has to exist at all.
func (l *Lease) release(ctx context.Context) {
	_, _ = l.db.ExecContext(ctx,
		`DELETE FROM job_leases WHERE name = $1 AND holder = $2`, l.name, l.holder)
	l.token.Store(0)
}

The subtle line is the error branch. A failed renewal is not "you lost the lease" — it is "you do not know," and those are different facts. But you have to *act* on unknown as though it were a loss, because the alternative behaviour is to keep working while possibly superseded, and that is the two-leaders outcome the whole apparatus exists to prevent. Standing down when you did not need to costs you a few seconds of availability. Not standing down when you needed to costs you the invariant.

This asymmetry is worth stating as a rule, because it is easy to get backwards and it generalises well beyond leases: when the failure modes are asymmetric, fail toward the cheap one. Losing leadership unnecessarily is cheap. Holding it wrongly is not.

What PandaStack does, and the lesson that cost the most

The fleet-level version of this problem is the same problem with the names changed: which host owns a given sandbox, and which hosts should receive new placements. Each agent registers itself in a shared `agents` table and heartbeats every 10 seconds; the scheduler treats an agent whose heartbeat is older than 30 seconds as stale and excludes it from placement entirely. Ownership of a specific sandbox lives in a separate `leases` table, joined against the agent row. That is deliberately a much longer-lived record than the heartbeat, because a sandbox's state is genuinely pinned to one host — a managed database sits on that host's local volume — so "who owns this" and "is that owner alive" are two different questions that must not be collapsed into one query.

Collapsing them is exactly the bug we shipped. The lease lookup originally filtered on liveness, so a lease pointing at a dead agent came back indistinguishable from no lease at all, and the caller cheerfully fell through to "pick a fresh host" — routing an id-scoped request to a machine that had never heard of that sandbox. The fix was to make "lease exists but its holder is dead" a distinct, explicit error from "no lease," and to forbid callers from responding to it by re-picking. There is no sensible failover for a request that is fundamentally about one host's local disk. The honest answer is "its host is gone," and an honest error beats a confusing success every time.

The second lesson is the expensive one, and it generalises past this codebase: never trust a cache for a *negative* answer about liveness. The edge caches the agent list to avoid hitting Postgres on every create. Absence from that cached snapshot was being read as "this agent is dead" — but the list is filtered on a fresh heartbeat, so a single slow heartbeat at the wrong instant evicted a completely healthy host for the whole cache TTL. Measured in production before the fix: roughly one in eight lookups on one edge returned a stale-agent verdict, including a deploy that failed ten seconds after its own sandbox had booted fine, against a host that was active with a four-second-old heartbeat. A cached positive is a fact that was true recently. A cached negative is very often just ignorance with a timestamp on it, and the correct response to ignorance is to go and ask.

Making a fork drop its inherited leadership

Back to the case that started this. If your workers are sandboxes and you fork them to get warm, dependency-installed children, every child inherits the parent's disk — including anything on it that asserts leadership. The remedy is two lines of hygiene at child startup, and both matter.

from pandastack import Sandbox

# A parent worker that has done real setup: deps installed, caches warm, and --
# this is the landmine -- a leadership marker written to its own disk while it
# was happily the leader.
parent = Sandbox.create(
    template="base",
    ttl_seconds=3600,
    metadata={"role": "worker-parent"},
)
parent.exec("mkdir -p /var/run")
parent.exec("printf '%s' '{\"leader\":true,\"token\":41}' > /var/run/leader.json")
parent.exec("hostname > /etc/singleton-holder-id")

# fork() copies the parent's DISK and boots each child fresh -- no inherited
# processes, no inherited RAM. That is the good news. The bad news is that the
# disk is where people persist leadership, so three children now come up each
# holding a file that says "leader, token 41".
children = parent.fork_tree(count=3, metadata={"role": "worker-child"})

for child in children:
    # Rule one: a clone is never the holder. Revoke locally FIRST, contend for
    # the lease SECOND. A child that starts work on the strength of an
    # inherited marker has already double-processed something.
    child.exec("rm -f /var/run/leader.json")

    # Rule two: regenerate the identity. If the holder string is inherited too,
    # the child does not contend for the lease -- it RENEWS the parent's,
    # because `WHERE l.holder = EXCLUDED.holder` cannot tell them apart. Two
    # processes, one holder id, one token, and a fence that never fires.
    child.exec(f"printf '%s' {child.id} > /etc/singleton-holder-id")

    print(child.id, child.exec("cat /etc/singleton-holder-id").stdout.strip())

for child in children:
    child.kill()
parent.kill()

The second rule is the one that gets missed. Revoking the marker file but keeping an inherited holder identity — a hostname, a machine-id, a UUID written to disk at bake time — means the child does not contend for the lease at all. It renews the parent's, because the `WHERE l.holder = EXCLUDED.holder` clause cannot distinguish them. Both processes then hold what they each believe is an exclusive lease, both see the same token, and the fence never fires because nothing ever changed hands. The identity has to be regenerated per process instance, not per image.

The same reasoning applies to hibernate and wake, and it is the case most likely to bite you in production, because a woken sandbox comes back with its RAM intact and its clock re-synced to a wall-clock time it has not experienced. Whatever your process believed about leadership before hibernation, it still believes. Treat wake exactly as you treat process start: assume nothing, revoke locally, re-acquire from the store, and get a fresh token before touching anything that matters.

The compressed version

1. Decide whether the job is genuinely idempotent. If it is, stop reading and run two of them — the coordination you are about to build has a failure rate too, and it is not obviously lower than the thing it prevents.
2. If it is not, use a lease, never a lock. Expiry is not a nice-to-have; it is the only thing standing between you and a manual unlock at 3am.
3. Let the store own the clock. `now()` on the database, not `time.Now()` on the worker, and definitely not a monotonic clock on a guest that can be paused.
4. Renew at a third of the TTL, and treat a renewal you did not see succeed as a loss. Unknown is not yes.
5. Issue a fencing token and check it at the resource. If the resource cannot check a token, you do not have exactly-once — you have idempotency keys or you have hope.
6. Revoke leadership on the way in, not just acquire it. Process start, VM wake, snapshot restore, fork: every one of those is a moment where inherited leadership must be dropped before anything else runs.
7. Regenerate the holder identity per instance. An inherited identity turns contention into renewal, which is the one failure the fence cannot catch.
8. Never cache a negative about liveness. A cached "alive" is stale but harmless; a cached "dead" invents outages.

None of this is new — leases and fencing tokens predate every platform mentioned here by decades. What ephemeral, snapshot-based infrastructure changes is the frequency. The scenarios that made these mechanisms sound paranoid on a rack of long-lived servers — a process frozen for a minute, two processes booting from identical state, a machine whose clock is confidently three weeks wrong — are now routine operations that your own control plane performs on purpose, several times a minute, at 179 milliseconds a go. The mechanisms did not get more necessary. The world just got faster at producing the conditions they were designed for.