Section 8: Seen It Before? - Deep Dive: Deduplication — URLs and Content

Executive Summary: Deduplication happens twice. Before a fetch, every link is normalized, fingerprinted, and checked against an exact in-memory set, 220,000 times a second, in batches. After a fetch, every body gets a SHA-256 and a 64-bit SimHash, so a page we already have under another URL joins one cluster with one canonical URL. The candidate defends the exact set over a Bloom filter with one number, 3.8 million lost URLs a day.

The frontier from Section 7 is on the board. You pick the next dive yourself:

You: "Deduplication next. Section 3 showed the hidden hot path: 19 billion link lookups a day, about 220,000 a second, nineteen times the fetch rate. And 30% of first-time fetches turn out to be pages we already have. If either check is wrong, we either waste the budget or silently lose pages."

There are two questions with different shapes. Have we seen this URL? is asked before fetching, on every link. Have we seen this content? is asked after fetching, on every body.

Step One: Make Equal URLs Look Equal

Two strings that name the same page must produce the same fingerprint, or the seen-set is useless. So every raw link passes through a normalizer first, built on RFC 3986's normalization rules plus a few crawler-specific ones:

The path stays case-sensitive, because servers treat /About and /about as different resources. Beyond the fixed list, each host can carry a small list of ignorable parameters, learned when varying a parameter never changes the content hash. Session IDs are the usual catch.

rel=canonical isn't a normalization rule: you only see it after fetching, so it belongs to the content step.

Here's the running example. At 09:02 the hourly refetch of https://news.example.com/ yields two links: /2026/10/29/harbor-bridge-closed and the same path with ?utm_source=social. Stripping utm_source collapses both to one normalized URL, one fingerprint, and one lookup.