A Decision Tree for Pods That Will Not Start

A pod is not running and something is on fire. The instinct is to start restarting things, and the instinct is wrong — the pod status already narrows the cause to one subsystem, and reading it properly will save you twenty minutes of guessing.

This is the decision tree I would want a new team member to have. It is organised by what kubectl get pods shows you, because that is the one thing you always have.

Always start here

Two commands, in this order, before any theorising:

kubectl describe pod <name>        # events at the bottom are the payload
kubectl logs <name> --previous     # --previous reads the CRASHED container

The Events section at the bottom of describe is where the scheduler, kubelet, and container runtime all report what went wrong, in plain language. Most incidents are solved right there, and people skip it because the output is long.

The --previous flag is the other one people forget. When a container is crash-looping, kubectl logs without it shows the current attempt, which is usually a few milliseconds old and empty. The logs you want belong to the instance that already died.

Status: Pending

Pending means the scheduler has not placed the pod on a node. The container runtime is not involved yet, so nothing about your image or your application code matters. This is a scheduling problem, full stop.

Look at the events for a FailedScheduling message — it names the reason for every node it rejected, which is unusually helpful once you know to read it.

kubectl describe pod <name> | grep -A10 Events
# 0/6 nodes are available: 3 Insufficient cpu, 2 node(s) had untolerated taint,
# 1 node(s) didn't match Pod's node affinity.

Insufficient cpu / memory. Your requests do not fit anywhere. Note it is requests, not limits, that drive scheduling — a pod requesting 4 CPUs will not schedule on a node with 3.5 free even if it would never actually use them. Check what is actually free:

kubectl describe node <node> | grep -A6 "Allocated resources"

The fix is usually to lower the request to what the workload genuinely uses rather than what someone guessed. Over-requesting is endemic and it fragments your cluster: every pod reserving triple what it needs means a third of your capacity is stranded.

Untolerated taint. The node is marked as reserved for something else and your pod has no toleration for it. Common on GPU nodes, spot instances, and control-plane nodes. Either add a toleration or accept that the node is not for you.

Node affinity mismatch. Your nodeSelector or affinity rules exclude every node. Frequently a typo in a label, or a rule written for a cluster that has since been rebuilt with different labels.

Volume node affinity conflict. A subtle one. Your PersistentVolume lives in one availability zone and the scheduler wants to put the pod in another. The pod can never schedule anywhere except that zone, and if that zone is full you are stuck.

Status: ImagePullBackOff or ErrImagePull

The scheduler did its job; the kubelet cannot fetch your image. Read the exact error, because three quite different problems share this status.

kubectl describe pod <name> | grep -A5 "Failed"

“not found” or “manifest unknown” — the tag does not exist. Usually a typo, or a CI pipeline that failed to push while the deploy went ahead anyway. Verify from your laptop with docker manifest inspect <image>.

“unauthorized” or “pull access denied” — a registry credentials problem. Check the pod actually references a pull secret, and that the secret is in the same namespace as the pod. Image pull secrets are namespaced, and a secret that works in staging is invisible to a pod in production. This catches people constantly.

kubectl get secret <pull-secret> -n <namespace>   # must exist HERE
kubectl get pod <name> -o jsonpath='{.spec.imagePullSecrets}'

Timeouts or DNS failures — the node cannot reach the registry at all. Egress firewall, a broken NAT gateway, or private-registry DNS that does not resolve from inside the cluster. Test from a node rather than from your machine, because your machine is not the one having the problem.

One more that looks like a bug and is not: if you push a new image under an existing tag and the node already has that tag cached, imagePullPolicy: IfNotPresent will keep serving the old one. Your deploy “succeeds” and runs stale code. Use immutable tags — a digest or a build number — and this entire class of confusion disappears.

Status: CreateContainerConfigError

A precise and rather helpful status: the kubelet cannot assemble the container’s configuration because something it references does not exist. Almost always a missing ConfigMap or Secret, or a key missing from one that does exist.

kubectl describe pod <name> | grep -i "configmap\|secret"
# Error: configmap "app-config" not found

Check for the obvious namespace mistake first, then check the key rather than just the object — a ConfigMap that exists but lacks the key your pod asks for produces the same status. Ordering matters here too: if your deployment and its ConfigMap are applied together, the pod can start before the ConfigMap lands and fail on the first attempt, then recover on retry. That intermittent version is worth recognising so you do not chase it.

Status: CrashLoopBackOff

This one is misunderstood often enough to be worth stating plainly: CrashLoopBackOff is not an error. It is Kubernetes telling you it is deliberately waiting before another restart, because your container keeps exiting. The container started fine. Your application is the problem, or something it depends on is.

The backoff is exponential and caps at five minutes, which is why a pod that has been failing for an hour seems to take forever to retry.

Get the exit code first — it partitions the problem immediately:

kubectl get pod <name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'
kubectl logs <name> --previous

Exit code 0 — the container ran to completion and stopped. Kubernetes restarts it because the restart policy says to. Your process is not a long-running server: it did its work and exited, or your entrypoint runs something that terminates. For genuinely finite work, use a Job, not a Deployment.

Exit code 1 — a generic application failure. The logs are your only path; read --previous. Usually a config error, a failed database connection at boot, or a missing environment variable.

Exit code 137 — SIGKILL. Nearly always OOMKilled, covered below.

Exit code 143 — SIGTERM. Something asked it to stop politely. Often a failing liveness probe, not the app itself.

That last case deserves emphasis because the misdiagnosis is so common. If your liveness probe is too aggressive — a short initialDelaySeconds on an application that takes 40 seconds to warm up — Kubernetes will kill a perfectly healthy container mid-startup, forever. It never gets far enough to pass the probe, so it is killed, restarted, and killed again. The application looks broken and is fine. Use a startupProbe for slow-starting applications and let the liveness probe apply only after startup succeeds.

OOMKilled

Exit code 137 with reason: OOMKilled means the container exceeded its memory limit and the kernel killed it. Not a graceful shutdown — an instant SIGKILL with no chance to flush anything.

kubectl get pod <name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
kubectl top pod <name>   # needs metrics-server

Two causes, and they need opposite responses. Either the limit is genuinely too low for the workload, in which case raise it. Or you have a leak, in which case raising the limit just buys time before the same kill happens later — you have converted a fast crash into a slow one.

Distinguish them by watching memory over time. A leak climbs steadily and never plateaus; a legitimate need rises and levels off. For JVM workloads specifically this is worth real care, because a JVM inside a container will happily size its heap against the limit and get killed by the kernel while believing it has headroom. Java Flight Recorder with a rolling recording is the cheapest way to catch what allocated the memory, since the recording survives the kill.

Init containers: stuck before the real work starts

A pod showing Init:0/2 is not running your application yet. Init containers run to completion, in order, before any app container starts, and if one hangs the pod sits there indefinitely with no obvious complaint.

kubectl logs <pod> -c <init-container-name>    # -c is required for init containers
kubectl get pod <pod> -o jsonpath='{.status.initContainerStatuses[*].state}'

The classic offender is a “wait for the database” init container that loops until a dependency responds. When the dependency is genuinely down, the init container waits forever and the pod never progresses. That is arguably correct behaviour, but it is worth recognising on sight rather than assuming the application is broken.

Note that init containers restart the entire pod on failure, so a flaky init step produces a pod that repeatedly returns to Init:0/2 — a distinct pattern from CrashLoopBackOff, and one people frequently conflate.

Running but throttled

A pod can be perfectly healthy, pass every probe, and still be unusably slow because it is being CPU throttled. This one has no status to alert you: kubectl get pods shows 1/1 Running and everything looks fine.

CPU limits are enforced by the kernel’s CFS quota, which allocates your container a slice of CPU time per 100ms period. Exceed it and your process is stopped until the next period begins. The effect is latency spikes rather than errors, so it shows up in your p99 rather than your logs.

# Throttling is visible in cgroup stats, not in kubectl
kubectl exec <pod> -- cat /sys/fs/cgroup/cpu.stat | grep throttled
# nr_throttled and throttled_usec climbing = you are being throttled

This bites multi-threaded runtimes hardest. A JVM or Go process that sees many CPUs and sizes its thread pools accordingly will burn its entire quota in a fraction of the period, then stall. The mismatch between “CPUs the process can see” and “CPU time the cgroup permits” is the root cause, and it is why setting a CPU limit well below what the runtime detects tends to produce worse latency than setting no limit at all.

Memory limits behave differently and the distinction matters: exceeding a memory limit gets you killed instantly, while exceeding a CPU limit just makes you slow. That asymmetry is a reasonable argument for always setting memory limits and being cautious with CPU limits.

Running, but not Ready

The pod says 1/1 Running but 0/1 ready, and it receives no traffic. The readiness probe is failing. Unlike liveness, a failing readiness probe does not restart anything — it just removes the pod from Service endpoints, which is why the symptom is “deployed successfully, receives nothing”.

kubectl describe pod <name> | grep -A3 Readiness
kubectl exec <name> -- curl -s localhost:8080/health   # test from inside

Test the probe endpoint from inside the container. A probe hitting the wrong port or path fails while the application is perfectly healthy, and from outside the two look identical.

The trap worth internalising: never make a readiness probe depend on a downstream service. If your /health checks the database and the database blips, every replica reports not-ready simultaneously, every pod leaves the load balancer at once, and a brief database wobble becomes a total outage. Readiness should answer “can this instance serve traffic”, nothing more.

Terminating forever

A pod stuck in Terminating for many minutes is usually one of three things. Your application ignores SIGTERM and waits out terminationGracePeriodSeconds before being killed — fix the signal handling. A finalizer is blocking deletion, visible in kubectl get pod <name> -o yaml under metadata.finalizers. Or the node is genuinely gone and the control plane cannot confirm the pod is dead.

Force deletion exists but think before reaching for it. --force --grace-period=0 removes the API object without confirming the container stopped, so for a StatefulSet with attached storage you can end up with two instances believing they own the same volume.

When the pod is fine and the network is not

Sometimes everything reports healthy and traffic still does not arrive. At that point you are debugging the network, not the pod. Check that the Service selector actually matches the pod labels — a mismatch produces a Service with zero endpoints and no error anywhere:

kubectl get endpoints <service>    # empty means the selector matches nothing

If endpoints are populated and traffic still fails, suspect NetworkPolicy. A default-deny policy with an incomplete allow rule blocks traffic silently by design, which is exactly what makes it hard to see — our network policies guide covers the ordering rules. For flow-level visibility into what is actually being dropped, Hubble’s flow logs will show you the verdict per connection, which beats inference. And if the traffic never reaches the cluster at all, the problem is at the edge — worth checking your routing layer, whether that is Ingress or the Gateway API.

The short version

Pending is the scheduler. ImagePull is the registry. CreateContainerConfigError is a missing ConfigMap or Secret. CrashLoopBackOff is your application, and the exit code tells you which flavour. OOMKilled is memory, and you must decide whether it is a leak before raising the limit. Not-Ready is a probe. Terminating is a signal handler or a finalizer.

Run describe and read the events before forming a theory. The answer is in there far more often than the amount of time people spend guessing would suggest.