A downstream service gets slow. Every caller times out and retries. The retries triple the load on a service that was already struggling, so it gets slower, so more calls time out, so more retries pile on. Within a minute the thing is completely down, and it is your retry logic that finished it off.
This is a retry storm, and it is worth understanding because the naive implementation — the one in most codebases — actively causes it.
Retries convert a slowdown into an outage
The important property is that retries multiply load exactly when a system can least afford it. A service running at 70% capacity absorbs a small hiccup. The same service at 70% capacity, with every client configured to retry three times, suddenly faces up to triple the request rate the moment latency crosses the timeout threshold.
It gets worse with depth. If A calls B calls C, and each layer retries three times, a single user request can become nine calls to C. Retries compound multiplicatively through a call chain, so a modest policy at each hop becomes an enormous amplification at the bottom — and the bottom is usually the database, the least elastic thing you own.
That alone argues for retrying at one layer only, usually the outermost one that can still do something useful, rather than defensively at every hop because each team added it independently.
Jitter matters more than backoff
Exponential backoff is the standard advice and it is only half the fix. Backoff spaces out one client’s retries. It does nothing about the fact that all your clients are retrying on the same schedule.
Picture a service that goes down for ten seconds. A thousand clients fail at roughly the same moment, and every one waits exactly one second, then two, then four. They retry in perfect synchronised waves. The service comes back up, gets hit by a thousand simultaneous requests, falls over again, and the cycle repeats — sometimes for a very long time.
Jitter breaks the synchronisation by randomising each client’s wait:
// Backoff alone: everyone retries at the same instant
long delay = baseDelay * (1L << attempt);
// Full jitter: spread arrivals across the whole window
long capped = Math.min(maxDelay, baseDelay * (1L << attempt));
long delay = ThreadLocalRandom.current().nextLong(capped + 1);Full jitter — picking uniformly between zero and the capped backoff, rather than adding a small random offset to it — spreads the retries across the entire window. If you implement one thing from this article, implement jitter. Backoff without it merely schedules your thundering herd more politely.
Only retry what retrying can fix
A depressing share of retry logic retries things that will never succeed. A 400 will be a 400 next time. A validation failure is not transient. Retrying it burns capacity to arrive at the same answer, and during an incident that capacity is precisely what you do not have.
Retry on timeouts, connection failures, 429, and 5xx. Do not retry 4xx other than 429 — and treat 429 specially, because the server has told you it is overloaded and often told you exactly how long to wait in Retry-After. Honour it rather than substituting your own guess.
Retrying also requires that the operation is safe to repeat. A retried payment that actually succeeded the first time but returned a timeout will charge twice unless the server deduplicates. That is a server-side property you have to build deliberately — see our idempotency keys guide for the mechanism. Without it, aggressive retries do not just risk an outage, they risk duplicate side effects.
Budgets beat per-request limits
“Three attempts per request” sounds bounded and is not, because it scales with traffic. At ten thousand requests per second, that policy authorises thirty thousand.
A retry budget caps retries as a fraction of total traffic — say, retries may not exceed 10% of successful requests, measured over a rolling window. Under normal conditions retries are rare and the budget is never touched. During a broad failure the budget exhausts almost immediately and retries stop entirely, which is the correct behaviour: when everything is failing, retrying is not going to help and is actively harmful.
This is what a circuit breaker gives you structurally. Once failures cross a threshold it stops sending requests at all for a cooling-off period, then lets a trickle through to test recovery. The value is not just protecting the caller — it is giving the failing service room to recover instead of being held down by a load it cannot shed.
Where the breaker lives matters. In a service mesh it belongs in the mesh configuration rather than duplicated in every application, which also means it is consistent across services written in different languages. The boundary discussion in our API gateway versus service mesh guide is the right place to decide which layer owns it.
What to do about it
Audit your timeouts before your retries: a client timeout shorter than the server’s typical response time means you are retrying requests that were going to succeed, which is the fastest way to manufacture a storm from nothing. Then add jitter everywhere, retry at one layer rather than every layer, retry only what is retryable, and put a budget or breaker in front of it.
None of this is exotic, and most of it is a few lines. The reason it is worth doing deliberately is that the failure mode only appears under load, which means it appears for the first time during an incident — and by then your retry logic is on the wrong side of the fight. If you want to see it before then, this is exactly the behaviour worth exercising in a fault-isolation drill.