Section 2: 86 Billion Events a Day - Scaling Strategy & Non-Functional Requirements

Executive Summary: Section 1 fixed what we detect and for whom. This section puts numbers on it. One million events a second on average and three million at design peak become 86.4 billion events and 86.4 TB of raw telemetry a day, 3,000 vCPU of stream processing, 60 million rule evaluations a second, 300 GB of keyed detection state and five million raw detections a day. We set the SLOs we can honestly own (p99 ≤ 30 s from our edge to the first sink, measured as freshness on 99.9% of minutes) and the one we can't (source delivery latency). Then we cut the fleet into twelve fixed-size stamps of 250k events a second with 2× headroom, and write down the order in which the system sheds load, which never touches identity, control-plane or audit data.

The interviewer leans back. "Fine, you know what you're detecting. How big is it?" Answer with a chain, not a single number: rate, volume, storage, compute, rules, state, alerts. Each link sizes a box you'll draw in Section 3 — Core System Design Deep Dive, so walk it out loud once.

The Napkin Math, Once

Start from the prompt's "millions of events a second" and make it concrete. Twenty thousand tenants with 150M identities between them average 1M events/s, and we design for 3M events/s at peak: the diurnal swing plus incident bursts, which are exactly when we can't fall behind.

Rate        1M events/s avg, 3M/s design peak           (largest tenant: 100k/s at peak)
Per day     1M x 86,400 s             = 86.4 billion events/day
Raw         1 KB/event                -> 1 GB/s avg, 3 GB/s peak, 86.4 TB/day
Normalized  400 B/event (binary)      -> 34.56 TB/day
Compute     1 vCPU ~ 2,000 events/s   -> 3M / 2,000 = 1,500 vCPU at 100%
            run at 50% at peak        -> 3,000 vCPU fleet
Rules       150 enabled x 20,000 tenants       = 3M (tenant, rule) instances
            ~20 candidate rules per event      -> 60M evaluations/s at peak
            2.5 distinct groupBy entities/event -> 7.5M keyed records/s
Alerts      5M raw detections/day (~58/s) -> 200k incidents/day (25:1)

The 2,000 events/s per vCPU (parse, enrich and detect) is an assumption we commit to load-testing, not a fact, and you should say so.

Then ask what the bytes are, because it decides the degradation story. By volume, network flow logs are 55%, sign-in logs 15%, Azure Activity Log 10%, directory audit 5% and other resource diagnostics 15%. Identity, control-plane and audit together are only 30% of the bytes, yet they produce the large majority of high-severity detections. Contoso's attack in Section 1 lives entirely inside that 30%.

Behavioral baselines dominate state: 150M identities × 1 KB is 150 GB, next to 100 GB of window counters (500M entity–rule keys × 200 B), 15 GB of last-seen sign-ins and about 35 GB of sequence matches, dedupe sets and suppression keys. A stamp's 25 GB across 250 task slots is about 100 MB each. Alerts stay small: even the largest tenant's day is about 1 GB per [tenantId, day] partition, well under Cosmos DB's 20 GB logical-partition limit.

Key Insight: The expensive number is not 86.4 TB. It's 60M rule evaluations a second. Bytes are rented from a log; evaluations burn CPU, and they grow with tenants × rules, not with bytes. That's why Section 5's Detection Engine indexes rules by (tenantId, sourceType) instead of testing every event against all 3M instances.

Quick check before moving on: