Section 11: Losing a Continent Before Lunch - Multi-Region Reliability & Disaster Recovery
Executive Summary: This section takes the design through its failure domains: node, availability zone, region, dependency, and the one that hurts most in practice, bad data or a bad deploy. We run three active-active regions, each with a full index copy and each sized N+1, behind health-checked GeoDNS/anycast. PlaceDB keeps a single writer region with async cross-region replication, so reads never depend on the writer. We walk the region health lifecycle, the writer-region failure path, a scenarios table, why the weekly rebuild doubles as the disaster recovery plan, and the availability math behind 99.99%.
It's 12:10 on Tuesday. Sam is about to search for ramen, and the Americas region has just lost its network spine. The interviewer wants to know one thing: does Sam notice?
Failure Domains, Smallest to Largest
Name what can break first; each domain has a different answer:
| Domain | Example | Containment |
|---|---|---|
| Node | A search node dies | Every shard has 3 copies per region; the router skips the dead one |
| AZ | One zone loses power | Shard copies spread across zones; stateless services in every zone |
| Region | Americas is unreachable | GeoDNS/anycast moves traffic to the two healthy regions |
| Dependency | Geocoder or Reviews is down | Fallbacks: last known device location; stale aggregates in the index |
| Bad data / deploy | A feed corrupts a market, or a ranking change ships wrong | Circuit breakers, flags, versioned rebuilds |
Interviewer: "Most candidates stop at region failure."
You: "Region failure is loud, so it's the easy one. Bad data is quiet, and it replicates faithfully to all three regions. Active-active doesn't protect you from it at all."
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson