Section 5: State With an Expiry Date - Data Modeling

Executive Summary: This section derives the data model from the six access patterns the contract of Section 4 implies, rather than from a schema drawn in advance. Five stores emerge — an ephemeral location cache, a last-seen record that deliberately outlives it, a cached friend adjacency list, a session registry, and per-pair notification state — and each is justified by the pattern it serves and the lifetime it needs. The pivotal idea is that a time-to-live is not a cleanup policy here but a semantic: the absence of a location key means unknown, never "far away" — which is also why the age of that silence has to be recorded somewhere else, since an expired key cannot report how long it has been gone. And the most consequential modeling decision is a store that does not exist, because persisting the update stream would cost about 2.9 terabytes a day of the most sensitive data the company could hold.

Most candidates reach for data modeling by naming technologies — "I'll use Cassandra for locations, Redis for sessions" — and then retrofit queries onto the choice. Invert it. Write down every read and every write the system actually performs, attach a rate and a lifetime to each, and the stores fall out with their engines nearly decided. For this system that exercise is unusually revealing, because it exposes that almost nothing here is worth keeping.

Start With the Questions, Not the Schemas

Six access patterns, and only six:

1. Write my current location. About 333 thousand writes a second at peak, one per active sharer every 30 seconds, each one a blind overwrite — there is no read-modify-write, no merge, no history to append to. The same write touches a second, tiny key: a last-seen timestamp.
2. Read my friends' last known locations. This is the publisher-side geo-prefilter of Section 4's ingest path, and it is by far the hottest read in the system: every update consults roughly 38 online friends' cached positions, on the order of 13 million point lookups a second. Crucially it is not one lookup — it is a scatter-gather, because under user-ID partitioning those 38 friends land on up to 38 different cache shards.
3. Read my friend ID list. About 400 IDs, read once when a client connects, then held for the life of the session.
4. Find which session servers hold a user. Once per session establishment, when that user's socket is subscribed to their channel, and again on a graph-change invalidation. It is not a per-message lookup: once the subscription exists, the broker's own subscription table does the routing, so no component asks "where does this user live?" on the delivery path.
5. Read and write per-pair notification state. Only for pairs that are actually near each other, which is a tiny fraction of the friend graph.
6. "Where was Maya yesterday?" Deliberately unanswerable. This is not an omission; it is a decision, and it is the one with a number attached.

Key Insight: Every one of these five live patterns is built entirely out of point lookups by keys the caller already holds — P2 simply does up to 38 of them at once. Not one of them is a range scan, a join, a secondary-index query, or a geospatial radius search. That single observation is what tells you this is a messaging problem wearing a storage problem's clothes — and the entire live working set is about 1 gigabyte, so it fits in memory on one modest cache tier with room to spare.