Section 8: Bytes Before Restarts - Deep Dive: Artifact Distribution & Host Convergence

Executive Summary: Section 7 decided when a rollout moves. This section is about what happens on the machines when it does. Two walls carry the weight. Distribution: content-addressed layers mean a release ships only its 100 MB of new bytes through caches nobody has to trust, and a three-tier path (origin, zone cache, host disk) cuts the recovery of a re-imaged zone from 74 minutes to 15. Convergence: a host agent fetches and verifies before it drains, drains properly, applies only newer generations, never decides a rollback on its own, and keeps running when the control plane disappears.

Interviewer: "You've shown me how the orchestrator decides to advance. Now it writes desired state for 20 instances of checkout-api. What actually happens on those hosts?"

You: "Two things have to go right: the bytes arrive fast without one origin serving 100,000 machines, and the swap on each host stays safe even if the agent crashes halfway. Bytes first, because the convergence order depends on them."

Content-Addressed Layers

An image is not one blob. It is a manifest listing layers, and every layer is named by the SHA-256 of its compressed bytes. Release r-1432 is sha256:9f4e…, the digest of its manifest, and that manifest lists its layer digests. That gives three properties the rest of the design leans on:

1. Ship only what changed. The full image is 500 MB compressed, but a typical release changes only the top layers: 100 MB. The 400 MB of base layers are already on the host under the same names, so they never move again.
2. No cache invalidation, ever. A name can't point at different bytes, because the name is the hash. Caches only evict; they never go stale.
3. Caches don't need to be trusted. The agent recomputes each layer's hash and compares it with the manifest, the manifest's hash must equal the digest in desired state, and CI's signature covers that digest. A corrupted or compromised cache can waste time, but it can't make a host run bytes CI didn't sign. (Section 10 owns the signing chain.)

That is why releases are immutable digests, never tags like latest: a tag can move between two hosts' fetches.