Section 9: The Index Is Always Slightly Behind - Deep Dive: Ingestion, Freshness & Indexing
Executive Summary: Sections 7 and 8 assumed the index is correct. This section keeps it correct. We follow a change through entity resolution into PlaceDB, then via CDC and Kafka to every region's indexers. External versioning makes reordering and replays harmless, a low-priority lane isolates bulk loads, and a circuit breaker stops bad feeds. A weekly blue/green rebuild with a validation gate keeps the index honest. The target: edits visible in search at p99 ≤ 5 minutes.
The interviewer turns the page: "Menya Kaze has been dinner-only, opening at 17:00. At 12:02 the owner adds lunch hours from noon. When does Sam see it, and what stops your index from slowly drifting away from the truth?" The second half is the real question: derived copies rot unless something forces them back into line.
Where Changes Come From
The volume is small, about 2M changes a day (~23 writes/s), but the sources differ sharply:
| Source | Shape | Trust | Path |
|---|---|---|---|
| Owner portal | Single edits (hours, phone, closures) | High, verified owner | Priority lane, straight to PlaceDB |
| Provider bulk feeds | Millions of rows, hourly or daily | Medium; duplicates and errors | Entity resolution, throttled ≤ 5,000 writes/s |
| User-suggested edits | "This place closed" | Low | Moderation queue first |
| Ops tools | Merges, takedowns | High, audited | Priority lane |
Provider feeds need entity resolution before they touch PlaceDB. When a feed sends "Menya Kaze, W 41st St", we look up places within 50 m via Section 5's geohash7 cell and its neighbours, and match on name similarity plus phone number. A hit updates the existing place_id through place_sources; a miss creates a place. Skip this and Sam sees three Menya Kazes, each with a third of the reviews.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson