Nearly every service has a /health endpoint, and most of them are wrong in one of two opposite ways. The first kind returns 200 OK unconditionally, which tells you nothing — it confirms the web server can answer a request, not that the service works. The second kind checks everything it depends on, which sounds thorough and quietly turns a minor dependency blip into a total outage. Getting a health check right means understanding what it is actually for, and the answer is different for the two kinds of check Kubernetes and load balancers rely on.
Liveness and readiness answer different questions
The single most important distinction is between two checks that look identical and mean opposite things. A liveness check answers “is this process broken and in need of a restart?” A readiness check answers “can this instance handle traffic right now?” They trigger completely different actions: a failed liveness check restarts the container, while a failed readiness check just removes the instance from the load balancer until it recovers. Confusing them is the source of most health-check disasters.
Because the consequences differ so sharply, the two checks should verify different things. Liveness should be almost trivial. Readiness can be a little more involved. And the most dangerous mistake — the one that causes cascading outages — is putting readiness-style dependency checks into a liveness probe.
Liveness: check almost nothing
A liveness check should verify only that the process itself is alive and not deadlocked. It should not check the database, or downstream services, or anything external. The question it answers is narrow: is this process so broken that the only fix is to kill and restart it?
The reason to keep it minimal is that a restart cannot fix an external problem. If your database is down and your liveness check tests the database, the check fails, Kubernetes restarts your container, the new container also cannot reach the database, so it fails and restarts again — a crash loop caused entirely by the health check, on a service whose own code is completely fine. The database being down is not something restarting your app will ever fix, so the liveness check must not care about it. A good liveness check often just returns 200 as long as the process can respond, which is enough to catch a genuinely hung or deadlocked process.
Readiness: check what you need to serve a request
A readiness check has more license to look outward, because failing it is safe — the instance is temporarily pulled from rotation, not killed. Here it is reasonable to verify that the things this instance needs to serve traffic are actually available: a database connection can be acquired, a required cache is reachable, essential startup work has completed.
But even here, restraint pays off, and the trap is checking dependencies you do not strictly need. If your readiness check tests a non-critical downstream service — a recommendation engine, an analytics sink — then when that minor service has a hiccup, every one of your replicas reports not-ready at the same moment, all of them leave the load balancer simultaneously, and your perfectly healthy service goes completely dark because something non-essential twitched. Check only what this instance genuinely requires to do its core job, and let degraded-but-functional stay in rotation.
The shared-dependency cascade
The failure mode that turns a small problem into a big one deserves spelling out, because it is subtle and it is common. When every replica of a service checks the same shared dependency in its health check, that dependency becomes a single point of failure for the entire fleet — not because the service needs it to be perfect, but because the health check made them all fail in unison.
Picture fifty replicas, all readiness-checking the same database. The database has a two-second blip. All fifty checks fail within that window. All fifty leave the load balancer at once. Now there are zero healthy instances, the load balancer has nowhere to send traffic, and a two-second database wobble has become a full outage that outlasts the wobble itself, because the replicas take time to be marked ready again. The database recovered in seconds; the outage lasted minutes. It is the same self-inflicted amplification that turns a slowdown into a full outage when retries pile up in lockstep. This is why a readiness check should lean toward “can I serve at all?” rather than “is everything perfect?” — and why a service that can serve cached or degraded responses should report ready even when a dependency is struggling.
Startup is its own phase
One more check exists precisely to fix a common bug: the startup probe. Some applications take a while to become ready — warming caches, loading models, running migrations. If your liveness check runs during that slow startup, it fails because the app is not up yet, so Kubernetes kills the container before it ever finishes starting, forever. The app is not broken; it is just slow to boot, and the liveness probe never gives it the chance.
A startup probe solves this by holding off the liveness and readiness checks until the application signals it has finished starting. The liveness probe only begins applying once startup succeeds, so a legitimately slow boot is no longer mistaken for a hang. If you have ever seen a container that restarts endlessly and never quite comes up, an over-eager liveness probe with no startup grace is the first thing to suspect — the same pattern described in the guide to diagnosing pods that will not start.
The short version
Make liveness trivial — it should catch a hung process and nothing else, and it must never depend on anything a restart cannot fix. Make readiness check only what this instance truly needs to serve its core traffic, never non-essential dependencies whose blip would take the whole fleet down at once. Use a startup probe for anything slow to boot. And whenever you are tempted to add “just one more check,” ask what happens when that thing has a brief hiccup and every replica fails the check simultaneously — because that question is exactly how a thorough-looking health check becomes the cause of the outage it was meant to prevent.