Section 3: Most of the Clock Is Waiting - Scale Estimation & Workload Analysis

Executive Summary: 2,000 services on 100,000 hosts means 400,000 instances, 5,000 deploys a day and 400 an hour at peak. A standard rollout takes 192.5 minutes, so about 1,300 are in flight at once. Yet the control plane is small: 22 restarts, 110 state events and 217 health queries a second, plus 10,000 heartbeats. The bytes are large (100 TB a day) but come from zone caches, sized by a re-imaged zone's recovery, not the average. And 1.5 hours of every 3.2-hour rollout is deliberate waiting: the price of catching a bad release.

The interviewer leans back. "How big is it?" The useful answer finds which resource the design spends: requests, bytes, or time.

You: "Inputs first. Correct any that look wrong."

Services                    2,000      (average 200 instances each)
Hosts                     100,000      (3 regions x 3 zones = 9 zones)
Deploys per service          2.5 per working day
Working window              10 h, holding 80% of deploy starts
Change failure rate          5%

From Services to Deploys

Instances:        2,000 services x 200          = 400,000 instances
Per host:         400,000 / 100,000             = 4 instances per host
Per zone:         200 / 9 zones                 = about 22 instances per service per zone
Deploys:          2,000 x 2.5                   = 5,000 deploys/day
Peak starts:      5,000 x 0.8 / 10 h            = 400 deploys/hour (about 6.7 a minute)
Rollbacks:        5,000 x 5%                    = 250 rollbacks/day
Builds:           20,000/day (every merge builds; not every build deploys)

The giant, web-frontend, runs 20,000 instances (5% of all), about 2,222 per zone. Sam's checkout-api runs 1,800, 200 per zone.