Section 12: New Lanes, Same Politeness - Follow-up Feature Evolution

Executive Summary: The interviewer asks for more: JavaScript rendering, regional fetching, news in seconds, AI-training opt-outs, history, and learned priorities. Each lands as a new lane attached to the existing pipeline or as a flip trigger we named in advance finally firing. Two things never move: the crawl node that owns a domain owns its politeness, and one robots engine allows zero violations.

The interviewer sits back. "It runs. A billion fetches a day. Now the search team says a slice of the web is JavaScript-only, the news team wants articles in seconds, legal wants AI-training opt-outs honored, and someone wants a model to set priorities. Evolve it."

Two ways to fail: rebuilding the frontier for every feature, or bolting on a second crawler with its own idea of politeness. The interviewer wants each feature to arrive through a door the design left open.

Key Insight: Politeness lives in exactly one place: the crawl node that owns a registered domain. Any new feature that needs to touch a website goes through that node's back queue. A feature that bypasses it is a second crawler, and a second crawler is how you become the reason a site falls over.

JavaScript Rendering: The Day-One Trigger Fires

You: "Day one was HTML only, and we named the trigger then: the share of important pages whose main content needs JavaScript, which we have tracked since Section 10's JavaScript-only challenge. The parser flags a page whose raw HTML has almost no text but loads large script bundles. Once that share is material for Tier A and Tier B, we add a render lane."

The lane sits after processing, not before fetching; the fetch is unchanged. The parser emits a needs_render event, a pool of headless Chromium workers pulls the body from its WARC segment, executes it, and hands the rendered DOM back to the normal path: SimHash, content dedupe, the enriched document record (the processing hop of DocumentFetched) with a rendered flag.

Interviewer: "Why not just render everything?"

You: "That was the rejected alternative on day one. A render costs 10 to 100 times our 10 ms parse: 100 ms to a second. Rendering all 475M parsed pages a day turns a 55-core fleet into 550 to 5,500 cores. Rendering a prioritized slice, say 5% as an illustration, is about 275 pages a second, or up to about 275 cores at the expensive end. The slice is chosen by importance."

The subtle part is what rendering fetches. A page pulls scripts, stylesheets and API calls from its host, and each is a request to a website. Those subresource fetches go back through the crawl node that owns the domain, so they obey robots.txt and spend the host's allowance. They are also cached hard: bundles have content-hashed filenames shared across a site, so rendering a thousand pages from news.example.com fetches its main bundle once.

Quick check before moving on: