Section 10: Better Silent Than Wrong - Failure Modes, Reliability & Security
Executive Summary: This section takes the finished generator and asks two uncomfortable questions: how does it break, and what does it give away? We walk a failure-mode table, from node crashes to client buffer exhaustion, with how each is detected, what the system does and how far the damage spreads. One rule runs through all of it: fail closed. An outage is recoverable, but a duplicate primary key is corruption. Then we spend the 99.99 % budget deliberately, with the client buffer, retries to other nodes, and the handful of metrics that catch trouble early. Finally we decode Leo's ID as an attacker would: it reveals the exact millisecond, region, worker and a slice of business volume. IDs are not secrets, so they are never tokens, and an opaque public form covers the rest.
The interviewer taps the architecture diagram. "You've defended clocks and leases in the abstract. Now run it for a year in production. What breaks, and what does a caller see when it does?"
You: "Let me answer with a table first, then the one rule that makes every row consistent. The rule matters more than the rows."
The Failure-Mode Table
| Failure | How we detect it | What the system does | Blast radius |
|---|---|---|---|
| Generator node crash | Health checks fail; callers' RPCs error | Client library retries another node in the region. The dead node's lease expires after 10 s; replacements lease a different free worker ID right away (18 of 128 slots in use) | Requests in flight on one node; the caller's 100-ID buffer hides it |
| Clock rollback ≤ 10 ms | now < last_ms in the mint loop |
Spin until the clock passes the last-issued millisecond | A few ms of extra latency on one node |
| Clock rollback > 10 ms | Same check, larger gap | Stop minting, return UNAVAILABLE, mark unhealthy, alert |
One node out of rotation until time is fixed |
| Lease loss (GC pause, slow etcd) | No successful renewal for 5 s | Node self-fences and returns UNAVAILABLE; re-acquires and honors reserved_until_ms before serving |
One node |
| etcd outage in a region | Renewals fail on every node | Reservations stop committing, so every node stops minting within ~3 s (self-fence at 5 s is the backstop); the region stops minting; callers fail over to a neighbor region's generators (Section 9) | Whole region, mitigated by cross-region failover at higher latency |
| Network partition | Node can't reach etcd, or callers can't reach node | Node cut off from etcd self-fences at 5 s. Node cut off from callers keeps its lease but gets no traffic. Neither case can produce two owners | One node, or one zone's nodes |
| Sequence overflow (4,096 in 1 ms) | Sequence counter wraps | Block until the next millisecond | Under 1 ms of added latency; at 25 IDs/ms per node it's rare |
| Client buffer exhaustion | Buffer empties faster than refill | Caller blocks on a synchronous NextIds; if every node is unreachable, the caller's write fails |
One caller process |
Interviewer: "Three of those rows end in 'the node stops minting.' Isn't that a lot of self-inflicted downtime?"
You: "Yes, on purpose. Those rows are the generator protecting the one thing callers can't repair themselves."
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson