Section 6: Every Host Has Exactly One Owner - High-Level Architecture
Executive Summary: This section draws the crawler as ten components built around one rule: each website's politeness state lives on exactly one machine. Thirty crawl nodes, each a frontier shard with its own fetchers and DNS and robots.txt caches, are partitioned by registered domain. Bodies go into append-only WARC segments in object storage, and Kafka carries only ~1 KB pointers to a processing fleet that parses pages, dedupes them and feeds new links back through a 32-shard seen-set. We walk harbor-bridge-closed through the system end to end and show how it slows down, rather than falling over, when processing lags.
The interviewer slides the marker across: "You've sized it and modeled the data. Draw it."
The tempting picture is a queue, a pool of fetchers, a parser and an arrow back to the queue. That's Section 1's BFS loop with boxes drawn around it, and it has a hole you should name before you draw anything. With 11,600 fetches a second spread across a shared pool, who makes sure news.example.com gets no more than its 5 requests a second? Every fetcher would have to ask some shared authority before every request. So start from the opposite rule: one machine owns each host, and only that machine ever fetches from it.
The Components
| Component | Job | Size (Section 3) | Deep dive |
|---|---|---|---|
| Seed & hint ingestion | Takes seeds, sitemaps and IndexNow pings; verifies ownership; rate-limits hints per host | Small; hints only | Section 9 |
| Recrawl scheduler | Partitioned by domain like the frontier: each crawl node runs the scheduler for its own domains, scanning their URL-table ranges for next_fetch_at coming due |
750M recrawls/day | Section 9 |
| Crawl node ×30 | Frontier shard, fetchers, DNS cache and robots.txt cache for its domains | 500 concurrent fetches, ~625/s each | Section 7 |
| Object storage | Append-only 1 GB WARC segments; each append is durable and readable once acknowledged | ~10 MB/s per node, a segment closes every ~100 s | Section 5 |
| Kafka | DocumentFetched events and link batches |
~12 MB/s + ~22 MB/s | Section 11 |
| Processing fleet | Parse, extract and normalize links, exact and near-duplicate checks | ~55 cores average, ~10 × 16-core nodes | Section 8 |
| Seen-set service | "Seen this URL?" for every extracted link | 32 shards × ~5 GB, ~6,900 lookups/s each | Section 8 |
| URL table | One row per known URL | 20B rows × ~200 B = 4 TB | Section 5 |
| Host table | robots.txt, allowance, error counters, resolved IPs | 100M hosts × ~2 KB = 200 GB | Section 7 |
| Indexer | Consumes accepted documents | Out of scope | — |
Two loops share these boxes. The discovery loop runs fetch, parse, seen-set, new URL row, frontier: 380M new URLs a day. The refresh loop runs scheduler, frontier, fetch, URL row update: 750M recrawls a day. Both end at a crawl node on purpose. Whatever wants a URL fetched (a new link, a recrawl, a hint, an admin's urgent request) asks the node that owns the host, and that node alone decides when.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson