Section 11: Degrade Accuracy, Never Availability - Reliability, Failure Modes & Observability

Executive Summary: This section takes every way the limiter can fail, from a crashed LB to a bad filter build, and gives each its detection, behavior and recovery. One rule governs them all: a failure may make the limiter less accurate but must never take traffic down. Then come the SLOs that prove fairness, the telemetry behind them, and the "why was I throttled?" lookup for Contoso's ticket.

The interviewer circles the allocator: "Everything you've built depends on this. Kill it. Then kill everything else, one box at a time."

The one-line idea: a rate limiter exists to protect services, so its own outage must never become theirs. When a piece fails, the limiter gets less accurate, and it does so on a timer we chose in advance.

The Share State Machine

Every bucket on every LB is in one of four states.

• BOOTSTRAP: a key this LB hasn't seen runs at quota ÷ 500 with 1 token: 20 req/s for Contoso. With a grant inside 50 ms, bootstrap adds at most ~0.9% overshoot (Section 8) and may briefly false-block. The grant carries the token count (≤ 1), not a fresh burst.
• LEASED: the normal state. The bucket's rate is the granted share (Contoso's 250 req/s), renewed by each 1 s report. Denials on a key whose grant said saturated: false trigger an out-of-cycle report.
• STICKY: the 3 s TTL has lapsed without renewal, so the LB keeps enforcing its last share. Sticky shares alone sum to at most the quota. Outage overshoot comes only from LBs that join a key mid-outage (each adds quota ÷ 500) or were cut off and had their share reclaimed (each adds it). Contoso's worst case: 460 joiners × 20 = 9,200 req/s, +92%, for ≤ 5 min (STATIC caps: until recovery). That, and false blocks as traffic moves, is why STICKY is time-boxed.
• FAIL_OPEN: after 5 minutes the shares are too stale to trust, so failMode: OPEN rules allow everything. failMode: STATIC rules never get here: Directory's 2M req/s protective cap keeps its last shares for the whole outage.

Interviewer: "Fail open sounds like you've turned off protection."

You: "Only for the fairness rules. Rules that protect a backend from overload are STATIC, so they hold indefinitely. Failing closed (D5's rejected option) would make a limiter outage an outage for every service behind it. The flip trigger is abuse rules such as login attempts or signups. An attacker would love us to fail open there, so those rules are STATIC too."

Quick check before moving on: