Section 10: The Web Fights Back - Interviewer Deep-Dive Challenges

Executive Summary: The design is on the board: 30 domain-partitioned crawl nodes, a Mercator-style frontier, a 32-shard seen-set, SimHash and a tiered recrawl scheduler. Now the interviewer asks how it breaks, and none of the answers adds a box. The public web is infinite in places, hostile in others, and sometimes lies about what it is. Each of eight challenges gets the probe, the real question, a model answer with a mechanism and a number, and the senior line.

Challenge 1: Spider Traps

"calendar.example.org/events?month=2026-11 links to ?month=2026-12, which links to 2027-01, forever. What does your crawler do?"

What they're really asking: do you know your seen-set can't protect you from URLs that are genuinely new?

Model answer: Section 8's seen-set only stops repeats, and every month is a new URL. The defense is layered limits applied before a URL enters the frontier:

• Shape limits: URL length ≤ 2,048 characters, path depth ≤ 16, and no repeated path segments (/a/b/a/b/a/b, the mark of a relative-link bug).
• A per-host page budget for unproven hosts, say 10,000 pages. A host earns more through signals that are hard to fake: links from other hosts, pages the indexer keeps.
• Template yield: group a host's URLs by pattern (/events?month=*). If a pattern keeps producing SimHash near-duplicates, stop expanding it. Empty future months look almost identical, so this trap shows up fast.
• Priority decay: links found deep in chains of dynamic pages sink below useful work.

Interviewer: "Can't you just detect traps exactly?"

You: "No. From outside, an infinite site and a very large one look alike. I can only bound the cost."

The senior line: "I don't promise to detect every trap. I promise that no trap can cost more than one host's budget."