Section 2: A Throughput System With Manners - Requirements (Functional & Non-Functional)

Executive Summary: This section turns Section 1's assumptions into a contract: nine functional requirements, an explicit out-of-scope list, and a non-functional half that puts a number on throughput, freshness, politeness, robustness and durability. Two ideas hold it together. Availability for a crawler is measured as delivered throughput, not uptime. And when requirements conflict, politeness wins every time.

The interviewer caps the marker. "Okay. What exactly are we promising?" "Crawl the web and keep it fresh" is a mission statement. "Fetch a billion pages a day, refetch every hot page within an hour, and never open a second connection to a host that hasn't earned it" is a contract you can test and page someone on.

The Functional Contract

Nine requirements. Each one maps to a part of the design you'll build later.

1. Seed and discover. Start from a seed list, then find new URLs in the links of every page we parse. Discovery never stops.
2. Fetch, respecting robots.txt. Download pages over HTTP and HTTPS, HTML only. Before any fetch, check the host's robots.txt under RFC 9309. For news.example.com, that means we never request /search, whatever triggered the fetch.
3. Parse and extract links. Pull outlinks from every new or changed page, normalize them, and feed them back to discovery.
4. Deduplicate URLs and content. Don't fetch the same URL twice by accident, and recognize the same page under a different URL. The homepage link to …/harbor-bridge-closed and the shared copy with ?utm_source=social must collapse to one record.
5. Store the latest copy plus a 30-day archive. The index needs the current version; the archive is for reprocessing and debugging, not history.
6. Emit documents to the indexer. Every stored page produces a "document fetched" event for the downstream index team.
7. Recrawl to stay fresh. Refetch known pages on a schedule matched to how often they change and how much they matter, using conditional GETs so an unchanged page costs almost nothing.
8. Admin controls. Operators can set a host's policy (rate cap, block, with a reason), request an urgent recrawl of specific URLs, and add seeds.
9. Accept push hints. Read sitemaps and accept IndexNow-style pings, so a site can tell us a page exists or changed.

The interviewer pokes at the last one: "If the site pushes us the URL anyway, why not let it push the content too? Saves a fetch." Because then the site decides what the index sees. A ping is a hint, never the truth: it can move a URL up the queue, but we still fetch the page ourselves, as our own user agent, under the same robots.txt and rate limits as everything else. In the running example, the IndexNow ping for harbor-bridge-closed arrives at 09:00:30. Hints are batched and verified before they reach the frontier, so the page is still fetched by our own crawler at 09:02:04.

Quick check before moving on: