My view is that resilience mechanisms work best when they're targeted rather than applied uniformly. Circuit breakers are valuable, but they're only one tool, and overusing them can make systems harder to reason about.
A few principles I think hold up well:
-
Assume partial failure is normal. In a distributed system, requests will time out, packets will be dropped, dependencies will become slow rather than fail outright, and retries will happen. Design APIs and workflows with those expectations.
-
Protect resources before protecting correctness. Resource exhaustion (thread pools, connection pools, queues, memory) often causes larger outages than the original dependency failure. Bulkheads, concurrency limits, backpressure, and load shedding are often more effective than circuit breakers alone.
-
Make failures cheap. Timeouts should be explicit and relatively short. Waiting 30 seconds for a dependency that almost never recovers in 30 seconds usually just ties up resources.
On circuit breakers
I'm more selective than "put one on every client."
Circuit breakers are most useful when:
- A dependency experiences prolonged outages.
- Calls are expensive and pile up during failures.
- Opening the circuit meaningfully reduces load or improves recovery.
- You have a sensible fallback.
They're less useful when:
- Failures are brief and retries with jitter solve the problem.
- The dependency has high variance in latency but low failure rates.
- Incorrect tuning would cause oscillation or unnecessary outages.
- The operation is so critical that failing fast provides little benefit.
One common issue is that circuit breakers become another distributed state machine to tune. If thresholds aren't chosen carefully, they can amplify incidents by opening and closing repeatedly.
I'd prioritize these, roughly in order
- Reasonable request timeouts
- Idempotent operations
- Retry budgets with exponential backoff and jitter
- Concurrency limits
- Backpressure
- Load shedding
- Bulkheads (isolating pools of resources)
- Circuit breakers where they demonstrably help
- Graceful degradation and fallbacks
Retries deserve special attention
Retries are both one of the best resilience tools and one of the biggest outage multipliers.
Good retries typically have:
- Exponential backoff
- Randomized jitter
- Maximum retry count
- Overall deadline
- Retry only on transient failures
- Idempotent operations (or idempotency keys)
Without those, retries can turn a struggling service into a completely overloaded one.
Observability is part of resilience
Resilience mechanisms need feedback. I'd want visibility into:
- Success rate
- Latency distribution (not just averages)
- Timeout rate
- Retry rate
- Circuit breaker state changes
- Queue lengths
- Concurrency utilization
- Saturation of thread and connection pools
A circuit breaker opening should be an observable event, not something you discover after customers report problems.
Graceful degradation beats perfect availability
For user-facing systems, it's often better to return a partial response than an error.
Examples:
- Show cached recommendations if personalization is unavailable.
- Omit optional widgets if a downstream service is slow.
- Serve stale-but-valid data rather than blocking.
- Reduce feature richness under heavy load.
This usually provides a better user experience than treating every dependency as mandatory.
A modern trend I like
There's been increasing emphasis on adaptive concurrency limits and load-aware systems instead of relying primarily on circuit breakers. Rather than waiting until failures accumulate enough to trip a breaker, these approaches reduce concurrency as latency increases, helping prevent overload before it cascades.
So if I were designing a new service today, I'd probably start with:
- Strict deadlines and timeouts
- Idempotency
- Retry budgets with jitter
- Concurrency limiting and backpressure
- Good observability
- Graceful degradation
Then I'd add circuit breakers only for dependencies where production data shows they materially improve stability. In many systems, they end up being the exception rather than the default.
