Section 7: Never Lose a Hit You Acknowledged - Deep Dive: The Collection Path

Executive Summary: The interviewer hands over the wheel, and the candidate drives straight to collection, because it is the only part of the system that cannot be recomputed. We take five dives, each told as mechanism, then number, then trade-off: the durable acknowledgement (204 only after acks=all), the partition key (the election-night math that rules out keying by property), deduplication that lives inside session state instead of a 300-billion-entry store, failure handling (an immediate, declared disk-spool downgrade that bridges to steering traffic away from the region, never a silent drop), and bot filtering that flags hits instead of deleting them.

Section 6 ended with the whole board drawn. The interviewer caps the marker: "You've got about twenty minutes. Where do you want to go deep?"

HyperLogLog is clever and late data is subtle, but a Staff candidate picks the place where a mistake can't be undone:

"Collection. Everything downstream of Kafka can be recomputed. A bad cube, a buggy sessionizer or a wrong bot list all get fixed by replaying the raw log. A hit we never stored has no replay. So the path from the visitor's browser to the first durable byte is the one piece of this design that has to be right the first time."

Key Insight: Choose the deep dive by asking which part can't be recomputed. In a log-centric design, that is almost always the write path into the log.

Dive 1: The Durable Ack

The contract from Section 4 is fire-and-forget for the client, but it can't be fire-and-forget for us. The collector holds the request open until the hit is safely replicated:

Here is the mechanism. The producer uses acks=all against topics with replication factor 3 and min.insync.replicas = 2. The leader confirms only once every in-sync replica holds the batch, which is all three when the cluster is healthy and never fewer than two, because below that floor the write is refused. Only then does the collector answer 204. A single broker can die the instant after the ack and the hit still exists, because the three replicas sit in different availability zones (rack-aware placement). Kafka acks from memory, not after an fsync, so the guarantee comes from that spread. That is the whole meaning of "an acknowledged hit is never lost", and Dive 4 names the one declared exception.

That in-region replication is what most of the p99 < 50 ms budget pays for, and the producer's small linger batches many requests into one append. The visitor never feels the wait: navigator.sendBeacon hands the request to the browser and returns at once.

The pushback: "Why not return 204 immediately and append in the background? You'd halve your latency."

"Because nobody is waiting on that latency, and the saving creates a loss window. Each collector carries about 50K hits/s at design load. If it buffers even one second before appending, a crash loses about 50,000 hits that we already told the client were safe. A beacon never retries after a 204, so that loss would be silent. I'd rather spend milliseconds than make a promise I can't keep."

Then the honest boundary: hits blocked by ad blockers, tags that never loaded, and tabs closed before the beacon left never reached us, so they fall outside the promise. sendBeacon narrows that last gap by keeping a queued beacon alive through page unload. The promise says acknowledged, not generated.

Quick check before moving on: