Section 9: Do the Heavy Work Early, Ship by Changing a Number - Deep Dive: The Publishing Pipeline & Staged Rollouts

Executive Summary: Section 8 moved bytes to 3 billion devices at 14.4 Tbps. Every one of those bytes came out of this pipeline, and a bad byte reaches devices just as fast. The design splits publishing into two halves with very different costs. The pipeline is slow, compute-heavy and low-QPS: 50,000 submissions a day, each scanned, possibly reviewed by a human, split, signed in an HSM and written to content-addressed storage before release. The release is cheap: it changes one row in the release DB, which the release index picks up within 60 seconds. Because a release is only a row, a halt is only a status change: HALTED, offered to no device. We follow Trailhead Maps 4.3.0 from a 92 MB upload to an automatic halt at the 5% step, then 4.3.1 to Priya's phone.

The interviewer turns the marker around. "Trailhead uploads 4.3.0 on Tuesday morning. What happens between that upload and Priya's phone, and what stops it if it's broken?"

You: "Two different machines answer that. The first turns the upload into finished, signed files and can take up to an hour, because nobody is waiting on it. The second decides who gets those files, and it has to be quick to change and quicker to stop. I'll keep them apart, because a release that does any real work at release time can't be stopped cleanly."

The Submission State Machine

At 50,000 submissions a day (about 1.7/s at peak), throughput is irrelevant. But a step can take ten minutes, a human review can take a day, and a worker can die in the middle of either. That is why the pipeline is a durable workflow engine (Temporal class): one persistent state machine per submission, with queue-fed scanner and processor workers doing the steps.

Trailhead's 92 MB bundle arrives over a resumable upload into the object store, inside the edit transaction from Section 4. The first check is whether it's signed with Trailhead's upload key, which proves who sent it. It isn't the key devices trust. Then come the scans:

• Static analysis: declared permissions against the app's category and history, known-malware signatures, and SDK and target-API policy checks.
• Dynamic analysis: the app runs in an emulator that exercises its UI and watches network and file behaviour. At about 10 instance-minutes per submission, that's ~350 concurrent emulators on average and ~1,000 at peak.
• Risk score: about 80% of submissions are auto-approved in under an hour. The other ~20%, about 10,000 a day, go to human review. That includes new developers, sensitive permissions and risky changes such as a reset upload key.

Interviewer: "Why not chain these with cron jobs and queues? It's just a few steps."

You: "Because then nobody can answer 'where is submission 81,233, and why?' When a scanner dies halfway through, something has to know whether to re-run it. The workflow engine records every transition, retries failed steps, and shows the developer one status."