Web Crawler System Design

The prompt sounds like the first exercise in a networking course: "Design a web crawler for a search engine. It should discover pages across the public web, download them, and keep the copies fresh enough for the index, about a billion pages a day, without ever being the reason a website falls over."

Everyone knows the textbook version: a queue of URLs, a set of the ones you've seen, and a loop that fetches a page, pulls out its links and pushes the new ones onto the queue. That loop is correct, and it is about a tenth of the answer. Run it at a billion fetches a day and it knocks small websites offline, because the most-linked host is always at the front of the queue. It wanders forever inside an infinite calendar. It downloads the same article under forty different URLs. And by the time it has finished one lap of the web, the first pages it fetched are weeks out of date, so the index serves yesterday's news.

The Trap Built Into a Familiar Question

Two failures show up again and again:

• The BFS Loop: draws the queue, the seen-set and the while loop, adds "and we run a thousand of these", and stops. Nobody asked how a thousand workers avoid hitting the same small site at once, what robots.txt says when the server returns a 503, or how the seen-set answers 220,000 lookups a second. Signal: knows the algorithm, hasn't thought about the system or the people on the other end of it.
• The Distributed Everything: shards the queue by URL hash, puts a Bloom filter in front of a graph database, streams every page body through Kafka and renders every page in headless Chrome. Each piece sounds senior. Together they make politeness need global coordination, silently drop millions of new URLs a day, and cost ten to a hundred times the CPU per page. Signal: reaches for machinery before doing the arithmetic.

The passing performance starts from two numbers. A billion fetches a day is about 11,600 fetches a second, and because each fetch spends most of its time waiting on someone else's server, that means roughly 9,300 connections open at once, spread across thousands of different hosts so that no one host sees more than its share. Every parsed page yields about forty links, so the real hot path is the 220,000 "have we seen this URL?" lookups a second, nineteen times the fetch rate. From there the design follows: a frontier partitioned by domain so each crawl node owns its hosts' politeness with no coordination, an exact fingerprint set held in memory, content stored in large append-only segments, and a scheduler that spends 750 million recrawls a day according to how often each page actually changes.