Section 10: The Slowest Shard Sets the Pace - Deep Dive: Caching & Tail Latency
Executive Summary: Speed has two halves. Caching: five layers, each keyed for its own job, with a result cache keyed on geohash6 (never raw coordinates, never a CDN) that re-filters and re-ranks every hit from the caller's exact point. The tail: a 100 ms per-shard timeout, hedged requests after the ~40 ms p95, honest partial results, and a degradation ladder. Together they hold search at p95 ≤ 150 ms and p99 ≤ 300 ms at 17,400 searches per second.
Section 9 made the index fresh. Now the interviewer wants it fast: "Your budget is 122 ms on a miss. What do you cache, and what happens on the day one shard has a garbage-collection pause?"
Five Caches, Five Jobs
Each layer removes one specific cost; a cache without a job is just another place for stale data to live.
| Layer | Key | Value | Freshness | Size / hit rate |
|---|---|---|---|---|
| Client | last query + cell | last rendered page | until the user moves cells or edits the query, or 60 s | free, per device |
| Result cache (D8) | (geohash6, normalized query, filters, sort) | ≤ 200 ids with coordinates and static score parts | TTL 60 s; kept 10 min for paging | ~40 GB per region, ~35% hits |
| Details cache | placeId | full details payload | read-through, TTL 10 min, CDC invalidation | 40 GB per region (hot 10%) |
| Geocode cache | normalized place text | coordinates | long-lived | 80% hits, 50 ms timeout on a miss |
| OS page cache | index segment files | shard data in memory | always current | 50 GB per node of 128 GB |
Section 3's node count depends on the page-cache row: if shards spill out of memory, p99 shows it first.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson