Section 3: Storage Binds, Reads Dominate - Scale Estimation & Workload Analysis

Executive Summary: 500,000 messages/s at 1 KB each is 500 MB/s in and 43.2 TB/day. At a 3× peak that becomes 1.5 GB/s. Five consumer groups turn it into 7.5 GB/s of reads, and RF 3 adds 3 GB/s of replication. Seven days of retention is 907 TB on disk, and that number alone sets the cluster at 30 brokers (10 per zone, 63% full). Each broker writes only 150 MB/s and sends 350 MB/s, 11% of its NIC. The page cache holds about 11 minutes of tail: tail readers never touch a disk, replays always do. Every number here is our interview's assumption, not any company's real figures.

The interviewer taps Section 2's requirements. "Put numbers on it. Then tell me which resource you run out of first." The trap is to size this by message rate. A log that keeps a week of history is a storage system first.

Traffic: Half a Million a Second

You: "I'll take an average ingest of 500,000 messages/s. Times 86,400 seconds, that's 43.2 billion messages a day. Ari's order-58213 is one."

You: "Size per message is 1 KB as stored. That's after the producer batches and compresses, which is Decision 9: linger about 5 ms, batches of tens of KB, lz4 or zstd. I rejected per-message sends, which multiply requests and lose cross-record compression. The flip trigger is a sub-millisecond latency need: linger 0, smaller batches. So average ingress is 500 MB/s. Traffic is bursty, so I plan for 3×: 1.5 million messages/s and 1.5 GB/s at peak."