𝗥𝗲𝘁𝗿𝗶𝗲𝘀 𝗣𝗿𝗼𝗯𝗹𝗲𝗺
Retries are dangerous.Most engineers think:
“Retry = reliability”
Reality:
Retry = potential system overload
If not handled properly, retries can cause:
• Traffic spikes
• Duplicate processing
• Cascading failures
Why? Because failures are rarely isolated.
If one service is struggling, blind retries just amplify the pressure.
Better approach:
• Add exponential backoff (wait a little longer after each failure)
• Add jitter (add small random delays so retries don’t happen all at once)
• Limit retry attempts (try only a few times, then stop)
• Use dead letter queues (move failed requests to a separate place to handle later)
But “smart retries” go deeper:
• Make operations idempotent (so duplicates don’t break things)
• Retry only transient failures (not logical or validation errors)
→ Transient failures are temporary issues:
• Network timeouts
• Service temporarily unavailable (5xx)
• Rate limits (429)
These *can succeed if retried later.*
Non-transient failures should NOT be retried:
• Bad requests (4xx like validation errors)
• Incorrect data
• Business logic failures
These will fail every time — retries just waste resources.
Simple rule:
If retrying later can change the outcome → retry
If the input is wrong → fix, don’t retry
• Use circuit breakers (stop retries when a dependency is down)
• Prioritize retries (not all failures are equally important)
• Add observability (track retry rates, not just failures)
Golden rule:
Don’t retry immediately.
Don’t retry forever.
Don’t retry blindly.
Reliability isn’t about retrying more.
It’s about retrying 𝘀𝗺𝗮𝗿𝘁𝗹𝘆.