Section 10: Late, Never Lost, and Able to Prove It - Reliability, Failure & Disaster Recovery
Executive Summary: Section 9 attacked the design one race at a time. This section walks whole failure domains, from one scheduler node to the region: what stops, how it recovers, and what the Payments team's daily-settlement-report experiences. One rule holds everywhere: the store's epoch fence keeps every timer to one owner, so each failure becomes lateness that the misfire policy and idempotent runs absorb, never a lost timer or a timer with two owners. We finish with disaster recovery, load shedding, and the observability that turns "late, never lost" into a measured fact.
The interviewer wipes a corner of the board. "Forget single races. It's 06:58 UTC on a Monday. Tell me what breaks, and what Payments sees at 07:00."
You don't promise that nothing fails. You promise something narrower: an acknowledged run fires, possibly late, never lost. Reliability work means turning each failure into a bounded delay and knowing the bound.
The Failure Matrix
| Failure | What stops | What keeps working | Recovery | Worst case for the 07:00 settlement run |
|---|---|---|---|---|
| Scheduler node crash | Firing on its 64 partitions | The other 960 partitions, API, workers | Lease expires in ≤ 10 s; new owner bumps the epoch, runs a catch-up scan | Fires ≈10–13 s late |
| etcd quorum loss | Ownership changes: no failover, no rebalancing | Live owners keep firing their last-known partitions; API; workers | Quorum returns, leases reissued; a node that died mid-outage gets a new owner, catch-up + misfire policy | On time if its owner is alive; else late by the outage (RUN_ONCE covers up to 1 h) |
| Store range leader election | Writes to affected ranges | Other ranges | Firing transactions retry with backoff | A few seconds late on a few partitions |
| Queue outage | Handoff to workers | Firing: runs commit as PENDING | Outbox sweeper re-enqueues PENDING > 5 s | Late by the outage; possible duplicate sends absorbed by run_id |
| Tenant worker fleet crash | Execution | Dispatch (the platform SLO is still met) | No heartbeat → LOST at 30 s → retry after 30 s, then 2 min | Succeeds on attempt 2 or 3; run_id keeps it to one report |
| API outage | New schedules, cancels, status reads | Every timer already stored | Stateless instances restart | None: tonight's run was promised yesterday |
| AZ loss | A third of the nodes; one replica per range | Store consensus quorum (2 of 3), etcd quorum | 320–384 partitions (5–6 nodes × 64) reassigned within a lease TTL | ≈10–13 s late |
| Region loss | Everything in-region | Nothing in-region | DR promotion, RTO ≤ 15 min | ≤ 15 min late, possibly delivered twice |
Three rows carry the philosophy.
etcd: keep firing, because the fence lives in the store. The interviewer pushes: "etcd is down and leases can't renew. Do you stop firing?" No. A node that can't renew keeps firing its last-known partitions. The lease clock never made firing safe; the store's epoch fence does. If another node has taken a partition over, the stale node's next firing transaction fails the fence and it drops that partition. Without etcd quorum nobody can take over, so live owners fire on time. Only a node that dies mid-outage leaves its 64 partitions waiting until etcd returns, then catch-up and the misfire policy run. The honest cost: this path runs perhaps once a year, and a rarely exercised mode breaks on the day you need it, so it goes on the game-day list.
Queue outage: why the outbox exists. Because the firing transaction writes the run as PENDING and enqueues only after commit, losing the queue doesn't lose a single run. A 10-minute outage at the average rate leaves 1,200 × 600 = 720,000 PENDING rows in the store. The sweeper backs off exponentially while the queue is down, so recovery isn't met by a thundering re-send; then it drains them at dispatch capacity: 720,000 / 25,000 ≈ 29 s. The rejected design, enqueuing inside or before the transaction, would have lost or invented runs here.
API outage: why the two availability numbers differ. The scheduling API targets 99.9% (≈43 min/month); the trigger path targets 99.99% (≈4.3 min/month). An API outage blocks new promises, visibly, to a caller who can retry. A trigger outage silently breaks promises we already made. That asymmetry is why the two paths share no runtime dependency except the store.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson