Section 5: Pages, Not Lists - Data Modeling
Executive Summary: This section turns Section 4's resources into records at rest. We start from the five access patterns and their rates, then model the Topics table, the paged Subscriptions table (partition key topicId#pageNo, at most 1,000 subscriptions per page), the Message Log record, the delivery task and the Retry Scheduler's time buckets. The central idea is the page: it makes small topics one cheap read and spreads app-promotions' 8 million subscriptions evenly across the store's partitions. We defend a partitioned log over a database queue (D2) and a sharded key-value store over a single relational primary (D5), and size everything from the ledger.
The interviewer points below the API you just wrote. "Topics, subscriptions, messages. Where do they live, and how are they keyed?"
You start with who reads what, how often, and by which key.
Access Patterns First
| Who reads | What it needs | Rate at peak | Key it reads by |
|---|---|---|---|
| Publish API | Topic record (exists? policy? quota?) | 150,000/s | topicId (cached) |
| Fan-out Workers | The topic's subscriptions, one page at a time | 150,000 publishes/s, mostly cache hits; 8,000 page reads for one flash-sale publish | topicId#pageNo |
| Control Plane API | One subscription: confirm, patch, delete | low, human and deploy-driven | subscriptionId |
| Delivery Workers | The payload | 600,000/s | inline in the task, or (partition, offset) in the Message Log |
| Retry Scheduler | Every task due now | 6,000 tasks/s in; ~6.1M pending at most (upper bound) | due second |
Key Insight: None of the 150,000-a-second reads should hit the Metadata Store per publish. The Subscription Cache absorbs them (Section 7 covers its staleness). The store's job is to make cache misses, large-topic page reads and control-plane writes cheap and evenly spread. That changes which store wins.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson