Section 7: The Line Outside Every Website - Deep Dive: The URL Frontier & Politeness

Executive Summary: The candidate dives into the frontier first, because it is the one component that touches both the business goal (fetch what matters first) and the hard requirement (never overload a site). A plain priority queue gets the first right and the second badly wrong. The fix is a Mercator-style frontier: front queues ordered by priority, back queues with one host each, and a heap that says which host may be contacted next. Around it sit strict RFC 9309 robots.txt handling, an adaptive per-host rate that rises and falls with the site's health, and a per-IP cap for shared hosting.

Section 6's architecture is on the board. The interviewer leans back: "Where would you like to go deeper?"

You: "I'd take three dives, in this order. First the frontier, because it is where priority and politeness collide, and politeness is the requirement we can never trade away. Second, deduplication in Section 8, because link lookups at 220,000 a second are the hidden hot path. Third, the recrawl scheduler in Section 9, because that is where the 750M daily recrawls get spent well or wasted. Frontier first."

Why a Priority Queue Is Not Enough

Start with the tempting design: one priority queue per crawl node, highest score first.

Interviewer: "Why not just that? You want important pages first."

You: "Because importance clusters by host. The top of that queue would be thousands of news.example.com and Wikipedia URLs, and 500 fetchers would pop them at once. That is hundreds of concurrent requests to one site, the outage the prompt told us not to cause. Priority answers which URL next. Politeness answers which host may be contacted now. They need two different structures."

The numbers say the same thing. At 11,600 fetches/s and a default of one connection per host, about 9,300 connections are open at any moment, so roughly 9,300 different hosts are in flight at once, nearly all at one connection each. To keep its 500 concurrent fetches busy, each crawl node needs hundreds of hosts in flight and many more waiting their turn. The frontier's real job is to spread work across millions of hosts while still favoring the important ones.