Section 11: New Layers, Same Pointer - Follow-up Feature Evolution

Executive Summary: Every item on product's wish list attaches at a seam the design already has. A schema major version migrates by expand, backfill, alias swap, contract. Human corrections form their own layer, merged at query time, so they survive re-runs. Release groups put one pointer above the per-entity pointers so a season flips with one LWT. operation-events becomes a change feed, Iceberg answers "what did the model see on date D", and regional reads and annotations on annotations follow. The 202 front door, the envelope, the operation lifecycle and the per-entity pointer stay as shipped.

Interviewer: "It works. Now the character team wants to change their schema, editors want their fixes to stick, marketing wants a season to go live at once, and the London office wants fast searches. What do you rebuild?"

You: "Nothing I've shipped. Each one is a new layer, a new table, or a new optional field. I'll say for each what changes and what doesn't."

Schema Major Versions: Expand, Migrate, Contract

The character team wants character-appearance v3. In v2, character is free text ("Lena Cruz"). In v3 it becomes two keyword ids, character_id and performer_id, so a stunt double is a different performer playing the same character. That's a type change, so the registry refuses it as a minor version (minors are additive only) and it becomes a new major, which means a new Elasticsearch index.

The migration runs in eight steps.

Cassandra doesn't change: it keeps each envelope in the version it was written in, and the translator runs at index time. The backfill goes through the indexers like any write (nothing updates ES in place) and rides the bulk token buckets. One algorithm's catalog run is 30M annotations, so the backfill is at least 30M / 20,000 = 1,500 s, about 25 minutes. With superseded runs, other producers and the human layer, plan for two to three times that: 50 to 75 minutes at 20,000 docs/s, and 70,000 of the cluster's 150,000 docs/s even with a bulk run going. Once producers write v3 envelopes, the v3-to-v2 translator keeps the v2 index current until it's retired.

Interviewer: "In Section 8 you rejected alias swaps."

You: "Per run, yes. A catalog-wide run would be 100,000 per-entity swaps. This is one swap for a whole index, operator-driven, a few times a year. Pointers stay untouched: v3 docs carry the same op_id, so a pointer resolves to the same run in either index. If a team needed a breaking change every week, I'd push back on the schema design, not the migration."

Quick check before moving on: