Section 4: One Stamp, Copied Sixty Times - High Level Architecture
Executive Summary: Producers push batches to a stateless front door that acknowledges only once the batch is durable in Event Hubs. Independent consumer groups then feed an ADX hot tier, a Parquet lake (the system of record) and a detection hot path, and one KQL router fronts every storage tier. All of it is packaged as a stamp, a cell designed for 1M events/s, run at half that at planned peak and copied 60 times, with a small global control plane deciding which tenant lives where. We follow one Contoso event from vm-web-07 to an analyst's screen, then defend keeping two copies of the data.
The interviewer slides the marker across. "You've got the numbers. Now draw it."
You start with the unit of deployment, not the boxes. Nobody wants to own a single pipeline at 30M events/s peak, so you draw one cell and say the platform is that cell, repeated.
The Boxes, Left to Right
| Box | Its one job | Detail in |
|---|---|---|
| Producers | 3M agents and service emitters write to a local WAL, batch (10 s or 1 MB) and push | Section 5 |
| Global edge | Points each producer at its tenant's home stamp (endpoint learned from the tenant registry at enrollment), behind Azure Front Door with WAF | Sections 6 and 10 |
| Ingestion Front Door | TLS, tenant binding from the credential, quotas, light validation, produce, then 202 | Section 6 |
| Event Hubs | Three hubs by class, 256 partitions, 72 h retention: buffer, fan-out, replay | Section 7 |
| ADX Writer and ADX | Seconds-scale search over the last 30 days (7 on SSD) | Sections 8 and 9 |
| Lake Writer and lake | Every class for 365 days in Parquet; the system of record | Section 8 |
| Blob Archive | Immutable P0 copies for years 2 to 7 | Section 8 |
| Detection hot path | A third consumer group, read lag p99 ≤ 5 s; the engine is the Azure Real-Time Threat Detection System topic | Section 11 |
| Query Router | One KQL endpoint: ADX for ≤ 30 days, async lake search job for ≤ 1 year, restore job beyond | Section 9 |
Notice what the front door never talks to: storage. Producers depend on one durable thing, the queue, so storage can be slow or even down for up to three days without agents noticing.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson