Section 3: The Worst Second of the Day - Scale Estimation & Workload Analysis

Executive Summary: This section turns 3,000 teams and their schedules into a workload. The scheduler fires 100 million runs a day, which averages about 1,200 per second. That average is misleading. At 00:00:00 UTC, 300,000 runs come due in the same second, 250 times the average. We size dispatch capacity at 25,000 runs per second so that herd drains in 12 seconds. Storage is modest (about 1.56 TB hot, dominated by run history) and bandwidth is trivial. The number that shapes the architecture is the store's write load at the drain peak: about 100,000 row writes per second, a delete-heavy load that outgrows a single database primary and sets up 1,024 logical partitions across 16 scheduler nodes.

The requirements from Section 2 are on the board. The interviewer caps the marker. "Before you draw a single box, how big is this?"

You: "Inputs first, so you can correct them. Then the run rate, the worst second, API and worker traffic, storage, and the write load on the store. I expect that last one to matter most."

Tenants (internal teams)               3,000
Active cron schedules                  2,000,000, averaging 25 runs/day each
One-off delayed jobs submitted         50,000,000/day (delay seconds to 30 days, average ≈ 1 day)
One-off jobs cancelled before firing   ≈ 10,000,000/day
Mean run duration                      5 s (median 800 ms, p99 10 min)

Runs: The Average Is the Easy Part

Recurring runs:   2,000,000 schedules × 25 runs/day   = 50M runs/day
One-off runs:     50M submitted/day                   = 50M runs/day
Total fired:      50M + 50M                           = 100M runs/day
Average rate:     100,000,000 ÷ 86,400 s              = 1,157/s  ≈ 1,200 runs/s
Peak minute:      10 × average (top of the hour)      ≈ 12,000 runs/s

You: "One simplification, stated once. About 10M of those one-offs get cancelled before they fire, like the Checkout service's expire-cart-8812 when the customer checks out first. So only about 40M one-offs actually fire. I'm keeping 100M in all the capacity math anyway. Over-counting by 10% is conservative, and it keeps every later number simple."

1,200 runs per second is comfortable for almost any modern system. If the day were flat, this would be a short interview.