Section 12: The Promises Come Due - Follow-up Feature Evolution

Executive Summary: This section answers the interviewer's classic closing move: six months later, the company wants A/B testing, flags reviewed in git, and typed config through the same pipe. Experiments add a per-experiment salt and a sampled exposure log on the same bucketing; git becomes a front-end syncing into the control-plane database while kills keep their instant lane; dynamic config is JSON variations with schema validation at publish. The reusable lesson: a design is evolvable when a new job means a new consumer of existing primitives, not a parallel system.

The whiteboard is full and the design survives failure. The interviewer leans back for the classic closing move: "Six months later, product wants real A/B testing with statistics, the platform team wants flag changes reviewed in pull requests, and the ML team wants to ship ranking parameters through this thing. What changes?" The question audits your earlier self: Sections 1, 2, 3, 5 and 7 each deferred something here — the stats engine, per-user exposure, git review, sticky assignments. Later is now.

Experiments: A New Salt, Not a New System

You: "Almost nothing changes on the serving path. An experiment is a flag with variations worth comparing, an independent cohort, and a measurement pipeline. The platform already has the first two."

The independent cohort is what D3's flip trigger was written for: in murmur3_32(flagSalt + ":" + unitId) mod 10,000, a new salt gives an independent assignment. When Fabrikam tests a new reply-ranking model, the experiment is its own flag with a fresh salt and a 50/50 split (threshold 5,000). Dana's bucket of 1,234 for smart-reply-v2 predicts nothing about their bucket here — and without that independence, the same unlucky 1% is treatment for every launch and effects confound each other.

Why must experiments share the platform's bucketing at all? Because D12's rejected alternative — a second experimentation system with its own hashing — eventually disagrees with the first, and then Dana gets treatment UI on a control backend: chips render, the endpoint behind them is the old model, and both the experience and the measurement are garbage. It's why smart-reply-chips shares smart-reply-v2's salt and unit, and why experiment assignment is the same hash, not a cousin of it.

What's genuinely new is measurement. Section 9's gates answer "is treatment hurting?" in real time; statistics answers "did treatment win?", which needs to know who was in which cohort. Logging every evaluation is the 50M events/s D9 already refused. The honest unit is the first exposure: one event per (user, experiment), emitted when a user first receives a variation — deduped in the SDK's seen-set and again at the collector, because instances restart and users touch many pods.

You: "Volume check: 100M DAU and a handful of concurrent experiments — call it five. Worst case, an experiment ramps to its full population in a day: 100M first exposures, about 1,200 events a second per experiment, tens of thousands a second across five at the 3× peak. Against the 50M evaluations a second that per-evaluation logging would cost, that's over three orders of magnitude cheaper, and the whole log is tens of gigabytes, not a streaming problem."

Each record is (hashedUnitId, experimentKey, variation, snapshotVersion, timestamp) — hashed IDs, never names or emails (D11). For the very largest experiments the collector also samples on top of the dedup — statistical power saturates long before 100M exposures — which is why the ledger calls the log sampled.

Interviewer: "Section 7 punted sticky assignments to you. Experiments need stickiness — why not store assignments after all?"

You: "Because the exposure log records what was served — a fact about the past, for analysis — while assignment stays computed, never stored. The hash is still the assignment table: zero state, zero lookups, D1 intact at 50 million calls a second. If a mid-experiment rule edit would move users between cohorts, the stats engine excludes the movers or we restart with a new salt; we never put a per-user read on the evaluation path."

Downstream sits an offline stats engine, joining exposures to outcome metrics and computing significance — sequential testing so experiments can be peeked at safely, CUPED to shrink variance; name-drops with a boundary attached, because that engine is its own batch system, owned by a data-science team. Guardrails are already built: an experiment's guardrail metrics are Section 9's gate signals — treatment errors above control + 0.5 pp or p99 above 1.5× control, sustained 5 minutes, pause and page.

Key Insight: The split is harm versus wins. Harm detection must be real-time and lives in the platform's gates; win detection tolerates hours of latency and lives offline. Same bucketing, same exposure truth, different clocks, different owners.

Interviewer: "So is this an experimentation platform now?"

You: "No. Assignment and delivery are shared, because duplicating them creates the disagreement bug. Inference is its own system with its own team: experiment design, metric definitions, statistical rigor. My platform promises deterministic cohorts and honest exposure records. It does not compute p-values."

Quick check before moving on: