Section 2: Bounded Inaccuracy, Unbounded Availability - Requirements (Functional & Non-Functional)

Executive Summary: This section turns Section 1's answers into a contract. The functional requirements cover what owners, load balancers and throttled clients can each do. The non-functional requirements put numbers on the promises: at most 1 ms added at p99, at most 10% overshoot over any 10 s window, false blocks under 0.1%, and rule propagation in 5 s regionally, about 30 min globally and 60 s in an emergency. We then name what's out of scope and settle the core tension: we maximize latency and availability, and bound the inaccuracy.

The interviewer asks the question that separates a wish list from a design: "What exactly will you demo, and which of your promises could I check with a load test?" For a rate limiter the second half matters more than usual, because its failures are quiet. A limiter that's too loose looks healthy until a backend falls over. One that's too tight looks healthy while a paying customer files a ticket saying they were throttled below their contract. Every requirement below has to be one you could be paged on.

The Functional Contract

There are three audiences, so you group the requirements by audience.

Service owners (the control plane):

• Rule CRUD with versioning. Rules are scoped to a service, an API and any mix of dimensions (tenant, caller service, user or client id). Every change creates an immutable version with an author, so "what was the rule at 02:00?" always has an answer.
• Composite rules. One request can match several rules, and all of them apply. Contoso's GET /v1/users call is checked against the Directory API's protective cap of 2M req/s per region and Contoso's own 10,000 req/s rule. The request is admitted only if every matching bucket admits it.
• Global limits. A small set of rules, such as an enterprise tenant's contract, apply across all regions, split into region shares.
• SKU templates and overrides. Product defines Free / Standard / Premium / Enterprise templates, and a per-tenant override takes precedence over the template.
• Shadow mode. A new or changed rule logs "would have throttled" for at least 24 h before the owner promotes it to enforcement.
• Emergency override. An on-call engineer can change or disable a limit everywhere within 60 s. Every override carries a TTL, so a 3 a.m. fix can't quietly become permanent policy.
• Dashboards. Admitted vs throttled per rule, top throttled tenants, shadow would-throttle counts, and a "why was I throttled?" lookup.

The data path: Decisions are made locally, in the Envoy edge filter and in the mesh sidecar, by one rule engine with the same semantics in both places. The Billing → Directory caller rule (50,000 req/s) is evaluated in the same pass as Contoso's, and throttling Contoso never touches Billing.

Throttled clients: They get HTTP 429 Too Many Requests with Retry-After and RateLimit headers, or gRPC RESOURCE_EXHAUSTED. A throttled client should learn when to come back, not only that it was refused.

Interviewer: "Why is shadow mode a functional requirement? Sounds like a deployment detail."
You: "Because this platform's likeliest outage isn't a crash. It's a correct system faithfully enforcing a wrong rule. A misplaced zero throttles most of a tenant's traffic while every component reports green. Shadow mode shows the blast radius on real traffic before it's real."

Key Insight: Here the control plane carries as much functional weight as the data plane. Owners live in versioning, shadow mode and overrides. Clients only ever see the 429.

Quick check before moving on: