Section 3: Thirty Million at Once - Scale Estimation & Workload Analysis

Executive Summary: The corpus is modest: 2 billion annotations in 2.6 TB today, a design target of 10 billion in three years. The shape of the writes is the real story. New content trickles in at about 62 annotations a second, but one catalog-wide algorithm run is 30 million annotations, and five at once peak at 50,000 a second, an 800× burst. Reads are a steady 6,500 QPS. One title holds 3.2 million annotations, enough to make a naive Cassandra partition 3.3 GB. Four numbers pick the architecture.

The interviewer nods at the board. "Requirements are clear. How big is this?" You write the agreed inputs in a corner, because every later number comes from them:

Catalog:      100,000 video assets, avg 45 min;  shots avg 4 s
Today:        2B annotations, ~1.3 KB stored payload each
Producers:    ~200 schemas, ~60 algorithms/teams, 2,000 human annotators
Per run:      ~300 annotations per asset per algorithm
New content:  300 assets/day

The Corpus, Today and in Three Years

Video:        100,000 × 45 min = 4.5M min = 270M s   (75,000 hours)
Shots:        270M s ÷ 4 s                 = 67.5M shots
Today:        2B × 1.3 KB                  = 2.6 TB logical
New content:  300 assets × 60 algos × 300  = 5.4M/day ≈ 2B/year
New algos:    ≈ 0.3B/year                  → ≈ 2.3B/year
In 3 years:   2B + 3 × 2.3B ≈ 8.9B         → design for 10B live

Netflix has described its annotation service, Marken, at about 1.9 billion annotations and 2.6 TB, about 1.37 KB each. Our 1.3 KB is that figure, rounded.

Interviewer: "Where does 300 per asset come from? Night Heist alone has 51,200 subtitle lines."

You: "It's an average per algorithm. A scene classifier emits a few dozen tags per asset; char-detect v3 emitted 1,040 on Night Heist, a 124-minute film. Subtitles are the dense outlier, 32 languages × 1,600 lines, and I'll treat that density separately because it breaks something else."