Section 8: 420,000 Questions a Second - Deep Dive: Delivering Bytes to Three Billion Devices

Executive Summary: This is the subsystem that carries the load Section 3 found: 420,000 update checks a second at peak, 16.8 million version lookups inside them, and 14.4 Tbps of egress, 77% of it update deltas. Three problems. Deciding: the device brings its installed list, the server holds a 1.8 GB release index in RAM, and the check is a read-only join with no per-device row anywhere. Choosing bytes: device-specific splits, deltas precomputed from the top three prior versions, a full-file fallback, and verification that treats a delta exactly like a full file. Moving bytes: immutable, content-addressed URLs on several CDNs, never invalidated. Then the hot release, and why peer-to-peer isn't the backbone.

Interviewer: "Storefront's done. Where would you lose sleep?"

You: "Here. A storefront page five minutes stale goes unnoticed. Delivery is 99.99%, carries nearly all the bytes, and anything wrong in it reaches every phone. Three parts: how a device learns there's something new, which bytes it gets, and how they travel."

Here's the whole path for Priya's Trailhead update, 4210 to 4300, on a day when release 4300's cohort covered her bucket:

Part A: "Anything New for Me?"

The device sends its installed list, roughly 70 bytes per app: package name, versionCode, signing-cert digest, 2.8 KB for Priya's 40 apps. The server holds the release index from Section 5: every live app's active releases (up to three per app, about 300 bytes each, including the digests of the app's signing lineage), 1.8 GB in total, loaded into memory on every update-check server. It's rebuilt from the catalog DB and refreshed by the change feed within 60 seconds. Each of the 40 entries costs one hash lookup plus a targeting check. Nothing is written, so any server can answer any device and the fleet scales by copying: at about 10,000 checks a second per server, 42 cover the 420,000 peak, and about 100 spread across regions with 2× headroom.

The average check finds a third of an update: 4 billion update downloads a day against 12 billion checks, so most checks come back empty. The index answers whether; only on a yes does the service fetch the file list for (artifact, variant, installed version) from variant_manifests, the immutable, cacheable table Section 9's processing step writes. That keeps index entries small enough for RAM and ties the costlier lookup to downloads (150,000 a second at peak), not to 16.8 million version lookups.

There's no per-device pending-updates table: Sections 3 and 6 showed why 120 billion rows rewritten on every rollout step and halt lose to letting the device bring the state it already holds. Here, moving 4300 from 1% to 5% is one field, rolloutBps 100 → 500.

The device asks; the server sets the schedule. Each response carries nextCheckAfterSec, about 6 hours (four checks a day) plus random jitter. Fixed intervals would synchronize devices into storms after every outage. A server-set interval is also a load-shedding lever: stretching it to 12 hours halves check traffic within one cycle, with no client release. Push is kept for one case. A critical security fix sends a data-free "check now" tickle, and devices answer it with jitter. Pushing every release to every device is the Push Maximalist's design; we'd widen tickles only if ordinary updates ever needed freshness under about an hour.

Interviewer: "The index is up to 60 seconds stale. Section 9 halts a release. What happens in that minute?"

You: "Halt sets the release status to HALTED, and within 60 seconds every index copy offers it to no device. Until then, devices in the 5% cohort that happen to check can still be offered a release already on up to 100,000 devices. Those keep it, since Android doesn't auto-downgrade; the fix ships forward as a higher versionCode. Rollout is gradual precisely so a minute of lag sits inside its margin. Strong consistency would put the catalog DB back on the hot path at 16.8 million lookups a second. Bad trade."

Quick check before moving on: