Section 4: A Lease, Not a Pop - API Design (with pushback)

Executive Summary: A crawler has almost no public API. Its contracts are internal: the frontier hands out work, fetchers report back, an event tells the indexer what changed, operators set host policy, and websites send hints. We write each one as a compact request/response shape and walk the running example through them. Then we take three pushbacks: why fetchers lease work instead of dequeuing it, why sites can't simply push everything they publish, and why Enqueue must be idempotent when 220,000 link lookups a second are mostly repeats.

The interviewer turns to a clean part of the board: "Show me the interfaces."

You: "There's no user-facing endpoint, so the API is the seams between components. I'll draw four: the frontier contract, the event the indexer consumes, the admin surface, and the hints websites send us. The first matters most, because how work is handed out decides what happens when a machine dies."

The Frontier Contract: Enqueue, Lease, Complete

Three calls. Anything that discovers a URL calls Enqueue. Fetchers call Lease to get work and Complete to report the result.

Enqueue(urls[], source, priorityHint) -> { accepted, alreadyKnown }
    source       = link | recrawl | sitemap | indexnow | admin | seed
    idempotent by URL fingerprint; a repeat can raise priority, never lower it

Lease(nodeId, max = 500) -> [
    { leaseId, url, hostKey, priority,
      conditional: { etag, lastModified, contentHash } }
        // etag, lastModified: for If-None-Match / If-Modified-Since
        // contentHash: previous SHA-256, to spot an unchanged 200
]                                               // lease TTL = 60 s

Complete(leaseId, outcome, latencyMs) -> ok
    outcome = fetched { status, segment, offset, length, contentHash }
            | not_modified
            | redirect { target }
            | robots_blocked
            | error { class, retryAfter? }

Three details carry the weight.

max = 500 is not arbitrary. Section 3's scale estimate sized a crawl node at 500 concurrent fetches. A node leases up to its free slots and no more, so the frontier never hands out work nobody can start.

The frontier only leases a URL whose host is ready. Politeness lives on the frontier side of the contract. A lease promises that this host's next allowed time has arrived and its connection slot is free, so the fetcher can stay deliberately dumb.

conditional turns a recrawl into a cheap question. The lease carries the ETag and Last-Modified from the previous fetch, so the fetcher sends a conditional GET. It also carries the previous content hash, so a 200 whose body hasn't changed is recognised on the node. Section 3's outcome split says 30% of recrawls, 225M a day, come back 304 Not Modified with no body at all.

latencyMs and the error class in Complete are how politeness learns. A fast 200 nudges a host's allowance up; a 429 or 503 halves it and honors Retry-After. Section 7's frontier and politeness deep dive builds that loop; the contract just carries the signal.

Here's one lease end to end, for the harbor-bridge article: