Section 13: A Billion Polite Requests - Final Summary & Candidate Scorecard

Executive Summary: This closing section turns the module into a way to grade yourself. We replay the interview arc and name the signal each phase was recording, then lay out a 100-point scorecard across five dimensions, with what a top score sounds like for a crawler and where points leak away. Score bands map to readiness levels and tell you which sections to re-run, and a final catalogue shows where this question hides in real loops under other names.

Almost every candidate opens with a queue of URLs, a set of seen URLs, and a loop that fetches, parses and pushes new links back. That picture is correct, and it earns very few points. The interviewer was listening for what came after it. Does the candidate treat politeness as the binding constraint rather than an afterthought? Do they notice that link lookups, not fetches, are the hot path? Do they treat freshness as a fixed budget to spend, not a loop to run faster? Those habits separate a Staff answer from a BFS loop with servers drawn around it.

The Arc, Replayed as Signal

Read the right-hand column as the interviewer's notebook.

Key Insight: Not one signal is "knows Kafka" or "can name Bigtable". Each one is about the web being someone else's system: you cannot speed up a slow host, trust a sitemap blindly, or catch up after an outage by firing harder.

Quick check before moving on: