System design · Day 9
Reliability: design for failure, not hope
Use timeouts, retries, redundancy, observability, and graceful degradation to keep a system useful when parts fail.
Failure is not an exceptional event in a distributed system. Machines restart, networks pause, dependencies slow down, and valid requests arrive in combinations nobody rehearsed.
Contain failure with time budgets
Every network call needs a timeout. Without one, a slow dependency consumes threads and connections until the caller also becomes unavailable. Set the timeout from the user's total latency budget, leaving room for the rest of the request path.
Retries help only when the error is temporary and the operation is safe to repeat. Use exponential backoff with jitter, cap the attempts, and add idempotency keys for writes. Otherwise a retry storm can turn a partial failure into a complete outage.
Redundancy needs a recovery plan
Run critical services across multiple failure domains and keep replicas of important data, building on Day 8. But redundancy is not recovery: backups must be restorable, failover must be practiced, and the team must know which system can operate in a reduced mode.
Graceful degradation protects the core journey. If recommendations fail, show popular items. If analytics is delayed, queue the events as in Day 7. Decide which features may become stale, which may disappear, and which must remain strongly correct.
Observe what users experience
Metrics show trends, logs explain individual events, and traces follow a request across services. Start with latency, traffic, errors, and saturation. Then define a service-level objective such as 99.9% of reads completing successfully under 300 milliseconds.
Alert on symptoms that affect users rather than every unusual internal value. Reliability is a loop: observe, respond, learn, and change the design. Tomorrow's capstone applies this loop to a complete service.
“Resilience is the ability to keep the important promise when an unimportant part breaks.”