Section 3: Fifty Million Questions a Second - Scale Estimation & Workload Analysis

Executive Summary: This section turns the fleet into numbers: 50 million flag evaluations a second, 35,000 client fetches a second at peak, a write rate that rounds to zero, and 100,000 long-lived streams that must all hear about a kill within five seconds. Each estimate is derived aloud and then makes a decision. Evaluation stays in the process, clients get values from the edge, and the engineering budget goes to fan-out and propagation latency.

The interviewer puts the marker down. "Before any boxes: how big is this thing?" Here the arithmetic decides where evaluation may happen at all.

You: "I'll assume about 2,000 services running 100,000 instances, pods or VMs, across 10 regions, so roughly 10,000 per region. About 30,000 active flags in production. For daily peaks I'll use 3× the average. Stop me if that's off by an order of magnitude."

Interviewer: "About right."

The Read Path: Where Remote Evaluation Dies

Start with isEnabled. A busy service checks flags on almost every request, so call it 500 evaluations per second per instance on average.

Fleet-wide:   100,000 instances × 500 evals/s  =  50M evaluations/s
Peak (3×):    50M × 3                          = 150M evaluations/s

Local (in-memory rules):  ~1 µs per evaluation
  500 evals/s × 1 µs  = 0.5 ms of CPU per second per instance
                      = 0.05% of one core

Remote (an RPC per check):  ~0.5–1 ms of network per evaluation
  fleet load          = 50M RPC/s average, 150M RPC/s at peak
  per check           = 500–1,000× the local cost

Interviewer: "50 million RPCs a second is big, but large companies run RPC systems at that rate."

You: "Buildable, but it becomes a tier-zero service every request at the company depends on. Every check pays 0.5 to 1 ms instead of 1 µs, and when the flag service has a bad minute, all 2,000 services have the same bad minute. Local evaluation costs 0.05% of a core and has no remote dependency. I'd go remote only for flags whose rules need data a service mustn't hold, such as a PII list, and only for those flags."

Key Insight: The estimate sets the architecture's most important rule: evaluation never leaves the process. At 50 million checks a second, a remote call on the evaluation path turns a configuration system into a single point of failure for the whole company.

Quick check before moving on: