Section 1: A Guest on a Hundred Million Websites - Problem Understanding & Clarifications

Executive Summary: "Crawl the web" sounds like a loop you could write in an afternoon. The real problem is a guest visiting a hundred million websites at once, on a fixed daily budget, without wearing out its welcome. This section breaks the prompt into what, who, why and how, then runs five clarifying questions that each remove something from the design. It ends with a written list of assumptions and the two ways candidates usually fail this question before they draw anything.

The interviewer reads the prompt aloud:

"Design a web crawler for a search engine. It should discover pages across the public web, download them, and keep the copies fresh enough for the index — about a billion pages a day — without ever being the reason a website falls over."

The last clause is the unusual part. Most systems protect themselves from their users; this one has to protect everyone else from itself. Every website you touch is someone else's server, often a small one, and nobody asked you to visit.

Decomposing the Prompt

Before asking anything, map the territory out loud in four parts.

• What it does. Four verbs: discover URLs, fetch pages, store what came back, refresh it later. Discovery and refresh compete for the same fetch budget.
• Who it serves. There are two audiences. Downstream is the index team, who consume what you fetch and care about coverage and freshness. On the other end are webmasters, who never signed up for your traffic and can block you with one line in robots.txt.
• Why it exists. A search index is only as good as its crawl. A page you never fetched can't be ranked, and a page you fetched three months ago gives users stale answers. The crawler sets the upper limit on search quality.
• How it runs. Nobody waits on a crawler request. It is a batch-shaped system that never stops, built from a frontier (the queue of URLs to fetch next) and a large pool of asynchronous fetchers. The figure that matters is fetches per second sustained, not response time.