Section 5: Five Records and a Reversed Hostname - Data Modeling
Executive Summary: The crawler keeps five kinds of record: a URL row, a host row, a frontier entry, page bodies in append-only segments, and the fingerprint indexes. We list the access patterns first and pick a store for each record afterwards. The URL table is an ordered wide-column store keyed by reversed hostname, so one host's URLs sit together for the scheduler. Bodies go into 1 GB WARC-style segments, and the URL row holds a pointer to them. The seen-set and the frontier are rebuildable derived state.
Section 4's contracts named urlFp, etag, segment and offset. Now you say where they live. Don't open with "we'll use Cassandra." A store answers a question, and the questions are the access patterns.
Access Patterns First
You: "Before I pick a database, here's every way the crawler touches its data, with the rate from Section 3."
| Access pattern | Rate (Section 3) | Shape | Where it lands |
|---|---|---|---|
| "Have we seen this URL?" | 220,000/s | Point lookup by fingerprint, batched | Seen-set, in memory |
| Insert a new URL | 4,400/s | Point write | URL table + seen-set |
| Update a URL after a fetch | 11,600/s | Point write by row key | URL table |
| Find a host's URLs that are due | Continuous | Range scan over one host | URL table |
| Read a host's politeness state | Every fetch | Point read, latency-critical | Host row, in memory on the crawl node |
| Refresh robots.txt | ~20M/day ≈ 230/s | Point write | Host table |
| Store a new or changed body | ~550M/day (475M kept and parsed, 75M duplicates dropped later), within the 290 MB/s inbound | Sequential append | WARC segments |
| Read one body for the indexer | ~5,500/s | Byte-range read by pointer | WARC segments |
| "Is this page a near-duplicate?" | Every new or changed body | Lookup by SimHash bits | SimHash tables |
Two things stand out. The hottest pattern needs only one bit of answer: seen or not. And the range scan over one host is the pattern a simple key-value store can't serve. That scan decides the URL table's key.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson