Section 3: Nineteen Lookups per Fetch - Scale Estimation & Workload Analysis

Executive Summary: A billion fetches a day sounds enormous, but it works out to 11,600 fetches a second, about 2.3 Gbps of inbound traffic, and 30 crawl nodes sized with Little's law. Bandwidth is not the constraint; the politeness rules of the websites are. Storage is dominated by page content (125 TB for the latest copies), not metadata. The real surprise is the link path: every parsed page yields 40 outlinks, so the "have we seen this URL?" question is asked 220,000 times a second, 19 times the fetch rate. That number decides the seen-URL set's design.

Section 2's requirements are on the board. The interviewer caps the marker. "How big is this, really?"

You: "I'll start from the fetch budget and the corpus, because everything else falls out of those two. Correct me on the inputs and the rest moves with them."

Corpus kept fresh          5.0 billion pages
Known URLs                 20 billion (discovered, including never-fetched)
Daily fetch budget         1.0 billion fetches/day
Average page               100 KB raw HTML, 25 KB compressed (gzip/brotli, 4:1)
Mean fetch time            0.8 s (DNS cached; connect + TLS + first byte + body)

Splitting the Budget: Recrawls vs Discovery

You: "A billion fetches a day isn't all new pages. Most of it keeps the 5 billion we already have fresh. I'll split it 750M recrawls and 250M first-time fetches, and the recrawls split by how fast pages change."

Tier A (hot: homepages, news fronts, feeds)   10M pages × 24/day       = 240M/day
Tier B (warm)                                 210M pages × 1/day       = 210M/day
Tier C (cold, the long tail)                  4.78B pages ÷ 16 days    ≈ 300M/day
                                                                         --------
Recrawls                                      240M + 210M + 300M       = 750M/day
Corpus check                                  10M + 210M + 4.78B       = 5.0B pages ✓
First-time fetches                            1.0B − 750M              = 250M/day

Our running example lives in Tier A: news.example.com's homepage is refetched every hour, which is how its new article gets found within minutes. Section 9's recrawl scheduler decides which page belongs in which tier; here we only need the totals.