Annotation Service System Design

The interviewer's prompt sounds like a metadata question: "Our studio runs dozens of ML algorithms and hundreds of human taggers over every film and episode we own. They produce annotations: 'a car is in this box at 01:12:04', 'this scene contains violence', 'Lena Cruz is on screen from 41:10 to 41:52', every subtitle line. Editors, artwork and trailer tools, and ML teams need to search all of it, for one title or across the whole catalog. Design the annotation service."

The first version writes itself: a table of (entity_id, type, start, end, data), an index on the entity, and a search box. It works for one algorithm and a few thousand titles. Then the character-detection team re-runs its new model over the whole catalog. Thirty million annotations arrive in an hour, the trailer tool slows to a crawl, and for forty minutes an editor searching for a character sees the old model's results and the new model's results mixed together, some scenes twice and some not at all. One title holds 3.2 million annotations, most of them subtitles, and its subtitle partition alone is 3.3 GB.

The Trap Built Into a Metadata Question

Two failures come up again and again:

• The Tag Table: designs one table with an index on the entity, adds a full-text search, and calls it done. Every write is a single row, so a model re-run is 30 million independent inserts and deletes, and readers see every half-finished state along the way. Signal: never asked what happens when an algorithm runs again.
• The Search-Engine Purist: puts everything in Elasticsearch because "the problem is search." It works until a schema change needs a reindex and there's no source to reindex from, or until a bulk run's indexing load doubles every editor's query latency. Signal: picked a store for its query features and ignored durability and write isolation.

The passing answer separates three things the tag table blurs. Cassandra is the durable source of truth, partitioned by title, schema, run and media minute so even the largest title spreads into 5.5 MB partitions. Elasticsearch is a derived index, one per schema major version, rebuilt from that source when it must be. And an algorithm run is an annotation operation: it writes in the background, seals with a declared count, and becomes visible in one compare-and-set on a pointer row. Readers see the old run or the new run, never a mix, and a bad model rolls back in under a minute by flipping the pointer back.