Section 3: Machines Read More Than Humans - Scale Estimation & Workload Analysis

Executive Summary: 50,000 hosts emit 2 million log lines a second, 86.4 TB a day. Compression and bloom filters keep 30 days in 285 TB, where a full inverted index would cost 9 to 18 times more. Metrics come to 100 million series, 10 million samples a second, 50 TB of tiers and 900 GB of RAM. Alert rules issue about 80% of metric queries. The 60-second paging promise becomes a 51-second budget, and each number gets the decision it forced.

"You said 50,000 hosts and a page within a minute. How much data is that, and who reads it?"

State the inputs first so they can be corrected:

You: "Each regional cell gets provisioned for 40% of global load, the largest region plus headroom. Everything I derive now is global."

Logs In: Hosts to Bytes

Line rate:     50,000 hosts × 40 lines/s        = 2,000,000 lines/s (2M/s)
Peak design:   3 × 2M                           = 6M lines/s
Raw bytes:     2M × 500 B                       = 1 GB/s   (3 GB/s peak)
Raw per day:   1 GB/s × 86,400 s                = 86.4 TB/day
Per host:      40 × 500 B                       = 20 KB/s  (≈1.7 GB/day)

The agent keeps a 1 GB disk buffer and ships zstd batches; Kafka holds 24 hours behind the gateways.

Agent buffer:  1 GB ÷ 20 KB/s                   = 50,000 s ≈ 13.9 h (~14 h)
               at 3× peak                       ≈ 4.6 h
Wire (zstd 5×): 1 GB/s ÷ 5                      = 200 MB/s (600 MB/s peak)
Logs topic, 24 h:    200 MB/s × 86,400 s        = 17.3 TB/day
                     × RF 3                     ≈ 52 TB
Metrics topic, 24 h: 10M samples/s × ~10 B      ≈ 100 MB/s ≈ 8.6 TB/day
                     (compressed, labels amortized) × RF 3 ≈ 26 TB
Kafka total:         52 + 26                    ≈ 78 TB of disk

Why pay for 78 TB of Kafka when the same data lands in object storage? The short answer is a 24-hour replay window; Section 6, High-Level Architecture, takes that pushback in full.