Section 11: The Web Is Not Your Retry Buffer - Reliability, Failure Handling & Recovery

Executive Summary: A crawler is a throughput system, so reliability means two things: the daily fetch budget survives component failures, and no failure on our side ever turns into extra load on someone else's site. This section walks the failure table component by component, shows why one rule (write durable state before you acknowledge) keeps every failure down to a few redundant fetches, and ends with the recovery rule: catch-up is capped by politeness, never by backlog.

The interviewer turns from the whiteboard. "Thirty crawl nodes, a Kafka cluster, 32 seen-set shards, a wide-column store. Something is always broken. Walk me through what breaks and what it costs."

You: "Let me start with the scorecard I'm defending, because it isn't 99.99% uptime. Section 2 set the target at 95% of the daily fetch budget on 99% of days, and zero robots.txt violations, ever. So for each failure I'll ask two questions: how much throughput do we lose, and could it make us impolite? The second one matters more."

The Rule Underneath Everything

Every recovery story in this section rests on one ordering rule: make the state durable first, then acknowledge. We chose at-least-once delivery with idempotent writes (rejected: exactly-once transactions). The price is that some work gets done twice. The rule makes sure work is never lost, so "done twice" is the worst case.

Three places this shows up:

• Fetch bodies. A crawl node appends each body to its open 1 GB WARC segment in a store where every append is replicated before it's acknowledged and is readable at once (an HDFS/Colossus-style file, or an append blob). Only after that acknowledgement does the node publish DocumentFetched, then update the URL row and complete the lease. A segment closes about every 100 s (Section 3), but closing is bookkeeping, not durability. If the node dies before an append is acknowledged, the URL row still says "due", so the page is simply fetched again.
• New URLs. A discovered link becomes a URL row first and is marked in the seen-set second. Crash between the two and the next sighting of that link retries an idempotent upsert. Reverse the order and a crash leaves a URL marked "seen" with no row behind it. Nothing would ever crawl it.
• Outcomes. Every write is keyed on (URL fingerprint, fetch time), so a replayed event overwrites itself.

Key Insight: In a crawler, a duplicate fetch costs one request and a lost URL costs coverage you'll never notice is missing. Order every write so that a crash produces duplicates, never holes.

Quick check before moving on: