Section 6: Four Planes, One Spine, and the Table That Isn't There - High-Level Architecture
Executive Summary: Section 5 split the data into a small, strongly consistent source of truth and several derived read models. This section assigns those tables to services. The system has four planes of traffic: Publishing, Storefront, Delivery and Commerce. They hang off one spine, the Catalog: a relational database plus a change feed that builds every derived store. We walk Priya's first install of Trailhead Maps through the boxes, then Trailhead's 4.3.0 release from upload to downloadable. Then comes the part that gets scored: four components candidates draw by reflex, each refused with a number from the ledger and a trigger that would bring it back.
Most candidates draw one column: phone, API, database, CDN. That hides four kinds of traffic that differ by orders of magnitude. Publishing runs at 1.7 submissions a second but burns about 1,000 emulators at peak. Delivery answers 420,000 update checks a second and pushes 14.4 Tbps. Draw them as one service and you size everything for the worst of each.
The Planes and Their Boxes
| Plane | Box | Its job | Ledger number it serves |
|---|---|---|---|
| Publishing (99.9%) | Developer API + upload service | Edit transactions, resumable uploads | 50,000 submissions/day, 1.7/s peak; 3 TB/day uploaded |
| Workflow engine + scanners | One durable state machine per submission, emulator scanning, routing to human review | ~350 concurrent emulators avg, ~1,000 peak; ≈10,000 human reviews/day | |
| Build/sign service | Splits, universal fallback, top-3 deltas, HSM signing | Outputs 4× the upload: 12 TB/day; 8.2 PB/year raw with erasure coding | |
| Catalog (spine) | Catalog DB | Apps, releases, files, listings: strongly consistent, keyed by package | 2M apps, 80 GB of listings |
| Change feed + builders | Rebuilds the release index, storefront documents and search index from committed rows | Index ≤ 60 s, documents ≤ 5 min | |
| Storefront (99.95%) | Edge cache + storefront API | Serves per-(package, locale) documents, applies the device check per request | 174,000 views/s peak, ~90% hit, ~17,400/s at origin |
| Search service | Replicated inverted index with compatibility filters | 35,000/s peak, p99 < 500 ms | |
| Delivery (99.99%) | Update-check service | In-memory release index, stateless | 420,000 checks/s peak, 16.8M lookups/s, ~100 servers, p99 < 200 ms |
| Install-manifest service | Resolves the split set for a device profile from the immutable variant_manifests, checks entitlement, issues file URLs |
≈10,500 installs/s peak, p99 < 300 ms | |
| Multi-CDN + object origin | Immutable /f/<sha256> files, cached forever |
14.4 Tbps peak, ~600,000 file requests/s | |
| Commerce (99.99%, RPO 0) | Purchase service + ledger | Idempotent purchases via the billing provider | 1,740/s peak, p99 < 3 s; 9 TB/year of ledger |
| Entitlement store + outbox | "What do I own", keyed by userId; events to developers | 25 TB, 250 billion rows |
Ratings sit beside the storefront: 350 writes/s, aggregated into the storefront document as a pre-computed histogram. Commerce is its own plane because its numbers differ completely: 25 TB keyed by user rather than 80 GB keyed by package, RPO 0 rather than "minutes stale is fine".
Key Insight: The Catalog DB is the system's source of truth, yet it is not on the 99.99% path. The update-check service reads an in-memory snapshot. The CDN reads immutable files. The storefront reads derived documents. If the Catalog DB goes down for ten minutes, every existing release keeps installing and updating; what pauses is new rollouts, which belong to the 99.9% tier anyway. Halts still land through the emergency denylist (Section 11), and install manifests come from the precomputed variant manifests, not the catalog. Derived stores keep a strongly consistent core from becoming a single point of failure for 3 billion phones.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson