Section 5: Store IDs, Not Posts - Data Modeling
Executive Summary: This section turns Section 4's API into records at rest. We list the access patterns and their peak rates first, then pick a store and a key for each: posts in a relational store sharded by author across 8,192 logical shards, with a 64-bit ID that carries both time and shard; the follow graph as two adjacency lists; and the materialized feed as a capped, in-memory list of 16-byte entries. The central idea is that feeds hold IDs, not content. That one choice keeps the feed cache at 8 TB raw instead of about 500 TB, and makes edits, deletes and blocks cheap. We name the rejected alternative for each store and what would flip it.
The interviewer taps the API you just wrote. "Posts, follows, feeds. Where does each one live, and how is it keyed?"
You don't start with databases. You start with who reads what, how often, and by which key.
Access Patterns First
| Access pattern | Who calls it | Rate at peak | Key it reads by |
|---|---|---|---|
| Candidates for a feed open | Feed Service | 175K opens/s | user_id |
| Next page of a feed session | Feed Service | 175K scroll pages/s | snapshot ID |
| Batch-get posts by ID (hydration) | Feed Service | 7M post reads/s, at most 70K/s past the cache | post_id |
| Recent posts of the celebrities I follow | Feed Service | 3.5M key reads/s | author_id |
| One author's posts, newest first | Profile, rebuilds, deep scroll | up to 3M reads/s during a full regional rebuild | author_id |
| Followers of an author, paged | Fan-out Service | 3.5K new posts/s | followee_id |
| Who a user follows | Feed Service, rebuilds | per open (cached), 5.8K changes/s | follower_id |
| Is this viewer blocked, or is this account private? | Hydration | every hydrated post | (viewer_id, author_id) |
| Is this follower active? | Fan-out Service | 700K checks/s | user_id |
Two things stand out. Every hot read names exactly one key, so nothing here needs a join or a secondary index at request time. And the two biggest numbers, 7M post reads/s and 700K inserts/s, both land on caches, not on the source of truth.
Key Insight: Design the keys from this table, not from the entities. "A post has an author" is true but doesn't tell you anything. "The fan-out reads followers by followee, 1,000 at a time" tells you the key, the page size and the shard.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson