Section 9: Seven Hundred Fifty Million Choices a Day - Deep Dive: Freshness & the Recrawl Scheduler

Executive Summary: Three quarters of the daily fetch budget goes to pages we already have, and this section decides which ones. We learn each page's change rate from its fetch history and spend the 750M daily recrawls where a fetch buys the most weighted freshness, which means not chasing pages that change faster than we can follow. Conditional GETs make recrawls cheap, a young-page schedule catches early edits, and sitemaps and IndexNow are hints weighted by each host's track record.

Section 8 made sure we never fetch a page twice by accident. Now the harder question:

Interviewer: "You have 750 million recrawls a day and 5 billion pages. How do you choose?"

You: "The frontier decides how fast we may fetch from a host. The scheduler decides what is worth fetching at all. I'll define 'fresh' first, because the policy falls out of the definition."

Freshness and Age: What We Are Optimizing

Two metrics, from Cho and Garcia-Molina's work on keeping a copy of the web in sync:

• Freshness of a page is 1 if our copy matches the live page and 0 if it doesn't. Corpus freshness is the average, weighted by importance.
• Age is 0 while our copy is current, and otherwise how long it has been out of date.

A news front that went stale three minutes ago and one that went stale three hours ago both score 0 on freshness. Age tells them apart. So the corpus is optimized for weighted freshness, and Tier A is also held to an age bound: refetched within an hour.