Section 8: Three Tiers, One Truth - Deep Dive: Storage Layer
Executive Summary: Storage holds 72.5 PB at steady state, takes 10M events/s, answers recent searches in seconds and keeps P0 data for seven years. Three tiers: ADX hot (30 days, 7 on SSD), a Delta Lake or Iceberg lake as the system of record (365 days), and an immutable Blob Archive for P0 (years 2–7). Two writers consume the queue independently, each committing idempotently before it checkpoints. Retention is cheap; deletion is expensive, and we say so.
The interviewer taps the two boxes under the queue. "Two stores fed from one stream. Convince me that isn't just paying twice."
Why Three Tiers
Most queries hit the last day, investigations reach back weeks, auditors ask about last year. No single store is both cheap per TB and fast per query (as Sections 3 and 4 showed, a year in ADX costs about ten times the lake per TB-month), so each tier serves one access pattern.
| Tier | Holds | Size (steady state) | Query latency | Role |
|---|---|---|---|---|
| ADX hot | All classes, 30 d (7 d on SSD) | ≈ 3.2 PB, 0.75 PB SSD | p95 ≤ 5 s (24 h), ≤ 30 s (30 d) | Index and cache. Can be rebuilt. |
| ADLS lake (Delta/Iceberg) | All classes, 365 d | 31.5 PB | Async search job, p95 ≤ 15 min | System of record |
| Blob Archive | P0 only, years 2–7 | 37.8 PB | Restore job, hours | Compliance, immutable (WORM) |
The writers are independent consumer groups, because exporting the lake from ADX would let every ADX outage hit the system of record. With two writers, ADX is a derived copy: a paired stamp can rebuild its hot tier from the lake, newest first. The P0/P1 lake accounts are RA-GZRS, so the pair reads the secondary without an account failover, at about 3 hours per day of history. P2's lake is ZRS, and the P0 archive sits in a GRS account because the Archive tier isn't supported on ZRS or GZRS. Once ADX's 30 days have passed, you can't rebuild a lake from it.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson