The Deploy You Never Ship: Designing a Feature Flag & Progressive Delivery Platform

It's 14:07 on the third day of a rollout. A new reply-suggestion feature is live for a quarter of a mail product's users, and its error rate has just climbed from 0.4% to 3.1%. An on-call engineer presses one button. Four seconds later, 99% of 100,000 servers in ten regions have stopped serving the feature, and nobody built, tested or deployed a line of code. That button is the visible tip of a system that decides millions of times a second, inside every process, who sees what.

Why Interviewers Love This Question

"Feature flags" sounds like an if statement and a database row. The interview is designed to find out whether you can see the distributed system hiding behind it. Three traps sit just under the surface:

• The Remote-Check Trap: The candidate puts isEnabled behind an API. At 50 million checks a second, the flag service becomes the largest service in the company and a hard dependency of every other one. Signal: never did the arithmetic.
• The random() Trap: The candidate rolls out to 25% with Math.random() < 0.25. Users flicker between old and new on every page load, and nothing guarantees the 5% cohort is still inside the 25% one. Signal: no feel for determinism.
• The Slow-Kill Trap: The candidate's kill switch is "set the percentage to zero" through the normal edit path, with its approvals, its queues and its 30-second polling. Signal: hasn't been paged for one.

The passing performance turns each trap into a design decision:

• Evaluation never leaves the process: local snapshots, microseconds, with code defaults when everything else is gone.
• Stable bucketing: hash(salt:unit) mod 10,000, so cohorts only ever grow.
• A kill switch that is its own lane: a priority override pushed over live streams, with every server reached in five seconds.
• Rollouts that police themselves: rings, bake times and health gates that compare treatment against control and stop a bad rollout before a human notices.