Section 6: Two Fleets and a Pointer - High-Level Architecture
Executive Summary: This section draws the boxes and walks one algorithm run through them. Writes enter through a separate Ingestion API fleet, are checked against the Schema Registry, and land on Kafka before anyone gets a 202. Store Writers apply them to Cassandra, the source of truth, then emit index events that Indexers apply to Elasticsearch. The Operation Service owns each run's lifecycle and flips the scope pointer with a Cassandra LWT. Reads go through a separate Query API fleet that resolves pointers first, so a reader never sees a mix of two runs. We follow char-detect v4's 1,100 annotations on Night Heist to Priya's next query, then defend the four choices that shape the board.
The interviewer hands you the marker. "You've got the contracts and the tables. Draw the boxes, then walk one algorithm run from the producer to a search result."
You say the rule before you draw: bulk writes and reads never share a fleet, and nothing becomes visible until its operation flips.
The Boxes, Each With One Job
| Box | Its one job | The number that justifies it |
|---|---|---|
| Ingestion API fleet | Authenticate producers, validate each batch, append to Kafka, return 202 |
100 batch requests/s at peak (500 annotations each) |
| Schema Registry | Hold ~200 versioned schemas; checks compatibility at registration | Cached in every ingestion node, so validation is local |
| Kafka | The front door log for every write, plus index-events and operation-events |
65 MB/s peak in; 3-day retention ≈ 1.8 TB |
| Store Writers | Apply annotations to Cassandra under per-producer token buckets, then emit an index event | 50,000 annotations/s provisioned |
| Cassandra | Annotations, locators, operations, scope pointers | 24 nodes, ~12,500 replica writes/s each |
| Indexers | Apply index events to the right schema-major index | ~6,700 doc writes/s per data node |
| Elasticsearch | Derived search index; dedicated vector nodes for shot embeddings | 15 data nodes + 3 vector nodes |
| Operation Service | Run state machine, leases, seal, count check, pointer flip | ~140 flips/s with 5 runs in flight |
| Query API fleet | DSL compiler, pointer cache, post-filter, hydration | 5,000 point reads + 1,500 searches/s |
| Visibility Reconciler | After a flip, re-indexes both runs' docs through the indexers with the new is_current |
≤ 60 s for catalog-wide queries |
| Drift Reconciler | Compares count plus max version per (entity, op) in Cassandra and ES; re-indexes gaps through the indexers | Catches what a lost event or a lost edit would leave behind |
| Iceberg Exporter | Reads Kafka, never ES; hourly commits to Iceberg for training and analytics | ~3 TB at 10B annotations |
Netflix has described the same broad split in Marken: Cassandra as the source of truth, Elasticsearch for search, Iceberg for offline analytics, and a separate fleet for ingestion so bulk writes don't hurt search latency. The wiring below is this design's.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson