Section 3: Memory-Seconds, Not Requests - Scale Estimation & Workload Analysis

Executive Summary: This section turns one input, 30 billion invocations a day in one region, into rates, concurrency, sandboxes, memory, hosts, cold-start rate, bandwidth, storage, placement load and billing volume. Three results shape the architecture. Memory binds, not requests or CPU. Two-thirds of that memory holds idle warm sandboxes, so keep-alive is the main cost lever. And per-host sandbox creation is tiny, so cold starts are a latency problem, not a throughput problem.

The interviewer taps the requirements from Section 2: "A large public-cloud region. Size it." You state the inputs aloud so they can be corrected. 30 billion invocations a day (about 900 billion a month). Peak = 3× average. Durations are heavy-tailed: median ~50 ms, mean 400 ms, with a 15-minute ceiling. The mix is 50% sync, 30% async, 20% event-source mappings.

From Invokes to Sandboxes

RATE         30,000,000,000 ÷ 86,400 s        ≈ 347K/s   → ~350K invokes/s average
PEAK         347K × 3                          ≈ 1.04M invokes/s  (call it ~1M/s)

CONCURRENCY  Little's law: L = λ × W
             peak:    1.04M/s × 0.4 s          ≈ 416K → ~420K busy executions
             average: 350K/s × 0.4 s           ≈ 140K busy

RESIDENT     warm-idle pool ≈ 2× busy          = ~840K idle
             420K busy + 840K idle             = ~1.26M sandboxes resident at peak

Little's law runs on the mean, not the median: a minority of long functions holds most of the sandboxes.

"Why is the idle pool twice the busy pool?" "Section 2 promised at most 1% cold invocations. Most functions run less than once a minute, so a sandbox serves one request and then waits minutes for the next. The idle pool is the price of that promise."

Quick check before moving on: