Section 8: Flip One Row, Not 1,100 - Deep Dive: Annotation Operations and Atomic Run Visibility
Executive Summary: A re-run replaces a set of annotations, not one row. This design makes the annotation operation, scoped to one (entity, schema, producer), the unit of visibility. A run writes under its own op_id, where no reader looks. It seals with a declared count and waits until Elasticsearch holds exactly that many documents. Then one Cassandra lightweight transaction (LWT) on the scope's pointer row moves current_op from the old run to the new one. Readers resolve the pointer first and filter everything by it, so they see v3 or v4, never a mix. Each flip costs about four round trips, and peak load is ~140 flips/s. Rollback is the same LWT in reverse and takes under a minute, because the old run is kept for 30 days.
Section 7 kept saying "resolve the pointer." The interviewer has noticed.
Interviewer: "char-detect v3 left 1,040 segments on Night Heist. v4 is halfway through writing 1,100. Priya searches right now. What do they see?"
You: "All of v3 and none of v4. Then, at one instant, all of v4 and none of v3. Let me first kill the obvious approaches, because each one breaks a different promise."
Netflix has described "annotation operations" in Marken for this need: an ML pipeline writes a run, and consumers never see it half-written. The mechanism below is this design's.
Four Wrong Answers
| Approach | What Priya sees mid-run | Why it fails |
|---|---|---|
| Delete v3, then insert v4 | Nothing, then a growing part of v4 | A visible gap. On a big title it lasts minutes. |
| Overwrite in place, matched by producer key | Some v3 rows, some v4 rows | A mix. And v3 segments that v4 no longer finds need separate deletes. |
Write v4 hidden, then flip a searchable flag on each ES doc |
A mix while the flags change | 2,140 updates for one title, about 60M (30M on, 30M off) for a catalog-wide run. ES has no multi-document transaction. |
| A new index per run, then swap an alias | A clean switch | Atomic, but one index per entity per run means millions of indices. It's the right tool for a schema migration (Section 11), not for one title. |
Key Insight: Making 1,100 documents change together is expensive. Making one row change atomically is cheap. So visibility should depend on one row: a pointer that names the current run.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson