Section 3: Ten Million In, Four Thousand Out - System Characteristics
Executive Summary: Before drawing any boxes, you name what kind of workload this is and work out its size. Security telemetry is append-only, loses value as it ages, arrives in three priority classes, is ordered by time, is queried one tenant at a time, comes mostly from a few heavy tenants, and is written about 2,500 times more often than it is read. Then you derive the numbers ledger that every later section cites: 10M events/s becomes 10 GB/s raw, 2 GB/s on the wire, 86.4 TB stored per day, 60 stamps, 256 partitions per stamp, a 518 TB queue window and 72.5 PB of steady-state storage.
The interviewer caps the marker. "What kind of workload is this, and how big?" The character tells you which storage engines are even candidates. The size tells you how many you need and where they break.
The Character of the Workload
| Characteristic | What it means here | Architectural implication |
|---|---|---|
| Immutable, append-only | An event is a fact; nobody updates a sign-in | Log-structured, columnar storage; compaction, not in-place writes |
| Value decays with age | Last hour is incident response; last year is retro-hunts; year five is an auditor | Tiered storage with a different cost and latency per tier |
| Mostly events, some metrics | Logs/events dominate; metrics are regular time series | Events kept at full fidelity; metrics can be rolled up and downsampled |
| Priority classes | P0 20% (sign-ins, audit, alerts), P1 30%, P2 50% (flows, verbose diagnostics) | Separate lanes; P0 is never sampled or shed, P2 is shed first |
| Time-ordered, ever-growing | Every event has a timestamp; almost every query has a time range | Partition by time; retention means dropping whole partitions, which is cheap |
| Tenant-scoped reads | A SOC queries its own tenant; internal Microsoft security may see cross-tenant aggregates under policy | TenantId is the first predicate and a physical layout key; the cross-tenant path is separate and audited |
| Heavy-tailed tenants | 50,000 tenants averaging ≈ 200 events/s; the top 500 produce 50% of volume | Placement must handle both the 1M events/s tenant and the long tail |
| Write-heavy | ~10M events/s in, ~4,000 queries/s out | Optimise the write path for throughput; the read path for pruning |
Key Insight: "Append-only and time-ordered" is the most useful thing you can say about this workload. Deletion becomes dropping a day. Tiering becomes copying a day to the archive and letting the original expire. Replay becomes re-reading an offset range. A workload with in-place updates gets none of these for free.
The decay curve maps onto storage tiers. The lake is written in parallel with the hot tier from day zero, not fed by it:
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson