Section 6: One Log, Three Clocks - High-Level Architecture

Executive Summary: This section wires Sections 4 and 5 into one system. Its spine is a single durable log: edge collectors append every hit to Kafka and only then return 204, and everything after the log can be recomputed from it. A streaming run produces the real-time card and provisional hourly cubes. A batch run over the immutable raw store produces final daily cubes that atomically replace them. A query service routes each report to the cheapest store that can answer it. We trace the Lisbon hiker's page_view in, Dana's query out, and name what each failure costs.

The interviewer slides the marker across: "You've got contracts and a data model. Draw me the system." The weak move is narrating twelve boxes left to right. The strong move is stating the organizing idea first.

You: "The organizing idea is one log and three clocks. Collection has a single job: get a hit into a durable, replayable log as fast as possible, and promise nothing else. Everything downstream reads that log and runs on one of three clocks. Seconds for the real-time card. Up to an hour for provisional intraday numbers. Up to twelve hours after a property's day closes for final numbers, which freeze at 72 hours. Because the log and raw store are immutable, every downstream box is disposable. That's what lets me give collection 99.99% and reporting 99.9%."

The Boxes, Left to Right

Each box, with the number that sizes it:

• Edge collectors. About 50K hits/s per node gives 100 nodes of capacity for the 5M hits/s design point. With N+2 per region across ~20 anycast regions, we deploy ~140. A collector validates the batch, looks up geo from the IP, discards the IP, produces with acks=all, and returns 204 inside the 50 ms p99 budget. Geo at the edge means the IP never reaches Kafka, a simpler promise than "we delete it later."
• Regional Kafka. The logical topic has 2,048 partitions keyed by hash(property_id, client_id). That spreads the election-night news property across every partition while keeping each visitor together. Peak is ~875 MB/s compressed, with 72 h retention on 225 TB of broker disk. The retention matches the late-data cutoff: every hit the pipeline may still accept arrives inside it, and a streaming job or raw writer that falls behind can catch up from the log. Restating a whole day reads the raw store, not Kafka, because retention runs on arrival time and a day's earliest hits can be older than 72 hours by the time it freezes.
• Enrichment. Parses user agent and UTM parameters, and flags the ~10% of hits from known bots. Flagged, not deleted, so a corrected bot list can be applied by reprocessing. It also redacts email- or phone-shaped strings from URLs and parameters (Section 11).
• Streaming job. Holds session state (about 300M sessions and ~60 GB at peak, small enough for one streaming cluster), dedups by seq inside that state, and writes two outputs: per-minute HLL sketches to the real-time store and provisional hourly cubes to the cube store.
• Raw writer to object storage. Writes immutable columnar files partitioned by event_date (in the property's time zone) and then property bucket. That's 10 TB/day, or about 4.25 PB logical over 14 months. This is the source of truth.
• Batch finalizer. The same pipeline definition as the streaming job, run in batch mode. It reads a closed property-day's raw files, ~10 TB/day in total across ~24 time-zone waves, and writes the final cube version.

Key Insight: Only three things in this diagram are irreplaceable: hits spooled on a collector's disk during an outage, hits in Kafka that haven't been written to the raw store yet, and the raw store itself. Every other box, including the cube store, can be rebuilt from those. So durability effort goes into collection and the log; downstream gets an honest "we'll recompute it."

Quick check before moving on: