Code Deployment System Design

The prompt sounds like a script: "Design the system that deploys code to production for a company running about 2,000 services on 100,000 servers in three regions. An engineer merges a change; the system should get it safely onto every machine that runs that service, catch a bad release before most users see it, and roll back in minutes."

Many candidates hear "deploy" and reach for a loop: SSH into every host, pull the new version, restart. Others draw a CI pipeline and stop at "then it goes to prod". Neither survives the follow-ups. What happens when host 31,402 is unreachable halfway through the loop? How do you know the new version is bad before it reaches every region? How much capacity can you take out of a service at once, and what if a zone fails while you're doing it? When the new version breaks, does "roll back" mean rebuilding the old commit, or something faster? And when the deploy system itself is the thing that's broken, how do you fix it?

The Trap Built Into a Familiar Question

Two failures show up again and again:

• The Script: a central runner pushes the new version to every host and restarts it, maybe in batches. It works for 50 machines. At 100,000 it has no answer for partial failure, no idea what "done" means, and nothing that stops a bad release before it's everywhere. Signal: has deployed code, hasn't owned a deployment system.
• The Throughput Answer: hears "100,000 servers" and designs for scale: sharded queues, a big event bus, a high-QPS control plane. The arithmetic says the control plane handles a few hundred writes a second. Scale was never the hard part. Signal: sized the wrong thing.

The passing performance notices that most outages start with a change, so the deploy system is really a safety system with a speed requirement. It makes every release an immutable artifact built once. It has hosts converge on a declared desired state instead of being pushed to. It rolls out in stages that each limit the blast radius, and judges the canary against a control group running the old version at the same moment. It keeps the previous version on disk so rollback is a fast redeploy, not a rebuild. It does the capacity arithmetic before taking anything out of service. And it designs for its own failure: hosts keep running what they have when the control plane disappears.