How to write a health check that catches real failures
The standard health check is four lines and returns 200 OK unconditionally. It answers one question: is the HTTP server accepting connections? That question was rarely the interesting one, and the endpoint will keep saying everything is fine while your database is unreachable, your queue is stalled, and every real request returns 500.
The other direction is worse. A health check that verifies every dependency turns a slow third-party API into a full outage, because your platform kills healthy instances for failing a check about something they do not control. Both mistakes are common and they have the same root cause: one endpoint doing two different jobs.
There are two checks, and they answer different questions
Kubernetes named these clearly enough that the vocabulary is worth borrowing even if you never touch Kubernetes.
- Liveness — Is this process wedged? A failure means restart me. It must not depend on anything external, because restarting your process cannot fix someone else's database.
- Readiness — Should this instance receive traffic right now? A failure means take me out of rotation. It may depend on things this instance needs to serve a request, and it is allowed to fail temporarily without anything being killed.
The single most valuable change most teams can make is splitting these apart. One endpoint that both restarts you and de-registers you cannot be tuned, because every threshold that makes it a good liveness check makes it a bad readiness check.
Liveness: almost nothing
A good liveness check verifies that the process can still do work, and nothing else. If your event loop is blocked, if a deadlock has your worker pool stuck, if memory pressure has made the process useless, this should notice. If a third-party API is down, this should not care.
// Liveness. No dependencies. The only failure mode it detects is
// "this process is no longer able to serve anything".
let lastTick = Date.now();
setInterval(() => { lastTick = Date.now(); }, 1_000).unref();
app.get("/livez", (_req, res) => {
// If the event loop has been blocked for 30s, we are wedged.
if (Date.now() - lastTick > 30_000) return res.status(503).send("stalled");
res.status(200).send("ok");
});For many applications the honest liveness check really is a bare 200. That is fine. The point is that it is deliberately bare rather than accidentally bare.
Readiness: what this instance needs to serve one request
Readiness checks the things without which this instance cannot do its job. The test for including a dependency is simple: if it is down, would every request to this instance fail? If yes, include it. If some requests would still work, leave it out and let those requests fail on their own.
A database your app cannot function without qualifies. A recommendations service that degrades to a default list does not. A cache does not, unless you cannot serve without it, in which case it is not a cache.
app.get("/readyz", async (_req, res) => {
if (shuttingDown) return res.status(503).json({ status: "draining" });
try {
// Cheap, and with a timeout shorter than the probe's own timeout.
await withTimeout(db.query("select 1"), 2_000);
} catch (err) {
return res.status(503).json({ status: "db_unavailable" });
}
res.status(200).json({ status: "ok" });
});
// select 1 -- not a count, not a join, not a query that touches user
// tables. You are checking that a pooled connection works, not
// benchmarking the database on a five-second interval forever.Startup is a third case, and it is why deploys fail
An app that takes 45 seconds to become ready — JIT warm-up, a large in-memory index, migrations, connection pools — will fail a readiness probe that starts checking after five seconds with three retries. The platform concludes the deploy is broken and rolls back, and the logs show a perfectly healthy application starting normally.
Two fixes, and you want both. Use a startup probe if your platform has one, which gives a generous window before the normal readiness schedule takes over. And where you cannot, set the initial delay to cover your realistic worst-case boot rather than your average.
Worth stating plainly: if you cannot work out why a deploy fails while the app appears fine, this is the first thing to check. It is by a wide margin the most common cause.
Health checks are half of graceful shutdown
The other place health checks earn their keep is deploys. When your instance receives SIGTERM, it should immediately start failing readiness while continuing to serve in-flight requests. That ordering is what makes a deploy invisible to users.
The reason is that load balancers learn about your state on a delay. If you stop accepting connections the instant SIGTERM arrives, requests already routed to you are cut, and users see a handful of 502s that nobody can reproduce afterwards.
let shuttingDown = false;
process.on("SIGTERM", async () => {
shuttingDown = true; // /readyz now returns 503
// Give the load balancer time to notice before refusing connections.
// This should exceed (probe interval x failure threshold).
await sleep(10_000);
server.close(async () => { // finish in-flight requests
await db.end();
process.exit(0);
});
});
// Without the sleep, in-flight requests are cut and you get a small
// spike of 502s on every deploy that nobody can explain later.The mistakes worth naming
- One endpoint for both jobs. Restart-me and remove-me-from-rotation cannot share a threshold. Split them.
- Checking dependencies in liveness. Turns a dependency blip into a fleet-wide restart and a thundering herd.
- Expensive checks. A probe running every five seconds across twenty instances is a lot of queries. Keep it to select 1, and cache the result briefly if you must check something costly.
- Checking things you do not need. Every dependency in a readiness check is a new way for your app to be marked down for something it can survive.
- No timeouts on the checks themselves. A hanging dependency should produce a fast, specific failure, not a probe timeout with no detail.
- Binding the health endpoint to localhost. It works locally and the platform's probe can never reach it, which produces a deploy failure with entirely healthy logs.
- Returning 200 with a body saying "degraded". Nothing reads the body. The status code is the entire interface.
A third endpoint, for humans
The status code is all a probe understands, but people debugging at three in the morning want more. A separate detailed endpoint — behind authentication, since it describes your internals — is a good place for it.
GET /internal/health (authenticated; not used by any probe)
{
"version": "1.24.0",
"commit": "a3f91c2",
"uptime_seconds": 84213,
"checks": {
"database": { "ok": true, "latency_ms": 3 },
"cache": { "ok": true, "latency_ms": 1 },
"queue": { "ok": false, "error": "connection refused", "since": "2026-08-25T09:14:02Z" }
}
}Note that the queue being down does not make this endpoint return 503, because it is not what readiness is for. It is information for a human, not a signal for a scheduler — and keeping those two things separate is the whole idea.
Wrapping up
Split liveness from readiness. Keep liveness free of anything external. Put in readiness only what this instance genuinely needs to serve a request, with a timeout on each check. Give slow-starting apps a startup window. Fail readiness on SIGTERM before you stop accepting connections.
That is maybe forty lines of code, and it converts your health endpoint from a formality into the thing that makes deploys invisible and outages short.
Frequently asked questions
What is the difference between a liveness and a readiness probe?
They trigger different actions, which is why they need different logic. A liveness probe answers whether the process is wedged, and a failure means restart me — so it must never depend on anything external, because restarting your process cannot fix someone else's database, and a shared dependency failing would restart your whole fleet at once. A readiness probe answers whether this instance should receive traffic right now, and a failure means take me out of rotation until I say otherwise, which is a reversible, cheap action. That is why readiness may check the database while liveness may not. If you have only one endpoint doing both jobs, every threshold that makes it a good liveness check makes it a bad readiness check, and you end up tuning it into uselessness in one direction or the other.
Should my health check query the database?
In readiness, yes if your app cannot serve any request without it; in liveness, never. The test for including a dependency is whether every request to this instance would fail while it is down — a primary database usually qualifies, a recommendation service that degrades gracefully does not, and a cache does not unless you cannot serve without it, in which case it is not really a cache. When you do check, keep it to something like select 1 with an explicit timeout of a second or two: you are verifying that a pooled connection works, not measuring database performance, and a probe running every few seconds across every instance adds up to a lot of queries. Wrap it so that a hanging database produces a fast, specific 503 rather than a probe timeout that tells you nothing.
Why does my deploy fail even though the app starts fine?
Nearly always because the readiness probe starts checking before the app is ready and gives up before it becomes ready. An application that needs forty-five seconds to warm caches, build an index, run migrations, or establish connection pools will fail a probe configured with a five-second initial delay and three retries, and the platform correctly concludes the deploy is broken while your logs show a perfectly normal startup. The fix is a startup probe if your platform offers one, which grants a generous window before the ordinary readiness cadence begins, or an initial delay set to your realistic worst-case boot time rather than your average. The second most common cause is the health endpoint binding to localhost instead of 0.0.0.0, which is unreachable from the probe and produces the same symptom of a failed deploy with healthy logs.
How often should health checks run?
Frequently enough to notice a problem quickly, rarely enough that the checks are not themselves a load. Every five to ten seconds is a reasonable readiness default, with a failure threshold of two or three so that a single blip does not pull an instance out of rotation. Liveness can be considerably slower — every fifteen to thirty seconds with a higher threshold — because restarting is a heavy action you want to be quite sure about. The arithmetic worth doing is the one nobody does: a check every five seconds across twenty instances is four hundred and eighty checks a minute, and if each one runs a database query, that is a permanent load you added for monitoring. It also sets the floor for graceful shutdown, since your drain delay needs to exceed the interval multiplied by the failure threshold.
Should a health check return details about what is failing?
Give probes a status code and humans a separate endpoint. Automated probes read only the status code — nothing parses the body, and returning 200 with a body saying degraded means the platform considers you entirely healthy no matter what the text says. So keep the probe endpoints minimal and let the code carry the meaning. Then add a separate detailed endpoint behind authentication that reports version, commit, uptime, and per-dependency status with latencies, which is what someone actually wants at three in the morning. Keep it authenticated, since it describes your internal architecture and dependency names, and let it report failures that do not affect readiness — a broken queue worth investigating but not worth removing the instance from rotation is exactly the case that endpoint exists for.
Keep reading
49ms p50 cold start. Fork, snapshot, and scale to zero.