The Millisecond Gatekeeper: Designing a Distributed Rate Limiter

At 02:00 in West Europe, a customer's nightly sync job wakes up and starts calling a directory API 14,000 times a second. Its contract says 10,000. Somewhere in a fleet of 500 load balancers, without any of them asking a central counter, the extra 4,000 requests a second get a polite 429 Too Many Requests and a hint to come back in a second. Every other customer's traffic is untouched. That decision has to happen in microseconds, millions of times a second, and it must never be the reason the cloud goes down.

Why Interviewers Love This Question

Rate limiting is the most-asked "small" system design question, and most candidates give the small answer. Three traps sit just under it:

• The Redis-Counter Trap: The candidate puts an INCR in front of every request. At 5 million requests a second, with three rules each, that's 15 million round trips per region. One service-wide cap becomes a single key needing about 2 million operations a second, and the network hop alone eats the whole latency budget. Signal: never did the arithmetic.
• The Algorithm-Recital Trap: The candidate spends twenty minutes comparing five algorithms and never says how 500 machines share one tenant's limit. Signal: knows the textbook, not the system.
• The Fail-Closed Trap: The candidate treats "can't reach the limiter" as "deny." Now the rate limiter's outage is everyone's outage. Signal: hasn't been paged for one.

The passing performance turns each trap into a design decision:

• No remote call on the hot path: decisions come from local token buckets.
• Demand-weighted rate-share leases from a regional allocator divide a quota across machines, and a global allocator does the same across regions.
• Bounded inaccuracy: at most 10% overshoot, under 0.1% false blocks. Degrade accuracy, never availability.
• Rules shipped like code: versioned, shadowed, rolled out in stages and rolled back in a minute.