Rate Limiting — Part 3: Advanced & Distributed Scale

maang.io System Design Series

What is this?

Our nightclub has become a chain. Same brand, same "100 people per room" fire code, but now there are dozens of doors, the bouncers talk to each other over a slightly laggy radio, and head office wants the rule enforced accurately across the whole building without making every guest wait at the door while a bouncer phones in to check the count. Welcome to rate limiting at scale, where every clean idea from Parts 1 and 2 meets a messy trade-off.

This is the chapter where the easy answers run out. We'll confront the N×limit problem head-on, weigh exact versus approximate enforcement, build the hybrid limiter that real systems run (and stare at the precision-vs-latency dial it forces on you), distinguish a rate limit from a concurrency limit, peek at adaptive limiting, and settle the question every senior engineer eventually faces: when your limiter's own store goes dark, do you fail open or fail closed?

The one-line idea: distributed rate limiting is the art of getting accurate enough counting across many nodes without paying a network round trip on every request — and every design here is a deliberate trade between precision, latency, and resilience.