Section 6: We Own the Tags, We Borrow the Permissions - High-Level Architecture
Executive Summary: This section puts services around Section 5's tables. We name each component and the number that justifies it, separate the two event streams (tag-events flowing out and content-events flowing in), and draw the one boundary that shapes everything else: the tagging service owns the tag-to-content relation, but the products own the content and decide who may see it. We then defend five things the diagram leaves out on purpose, each with the trigger that would bring it back, and trace Sam's click on checkout-outage from the gateway down to the two items they're allowed to see.
The tables are on the whiteboard. The interviewer taps them and asks: "Okay. Now put services around this." This is the moment candidates start drawing everything they know. Resist that. You've already done the arithmetic: 1,400 writes/s and 3.2 TB. So every box you add has to answer one question: what number, or what boundary, makes it necessary?
The Components, Each With One Job
| Component | Its one job | The number that earns it |
|---|---|---|
| API gateway | Turns the token into (tenant, user). Identity never comes from the path. | Every request, every path |
| Tag API service | A stateless service that handles every read and write, plans tag pages and serves autocomplete | 20,800/s forward lookups at peak, 2,100/s autocomplete |
| Tenant directory | A small global table mapping tenant to shard, cached in every Tag API process | The only structure that spans tenants |
| Postgres shards | 64 shards, one of them dedicated to Meridian, each a primary with 2 replicas | ~50 GB and ≈ 22 writes/s per shard |
| Forward and tag-metadata caches (Redis) | Forward: tag ids (never names) per content ref, 10 min TTL. Tag metadata: (tenant_id, tag_id) → display name, state, alias_of, as an in-process LRU plus Redis, invalidated on rename, delete and merge by a per-tenant pub/sub message, 60 s TTL as the backstop | Forward: 90% hit, so 20,800/s becomes ≈ 2,080/s, using ~20 GB. Metadata: hot tags only, well under 1 GB |
| Container-set cache (Redis) | Each (tenant, user)'s accessible containers: 60 s soft TTL with an asynchronous refresh-ahead, a 10 min hard TTL that serves stale sets to item lists only, plus invalidation | ~4 GB for 2M concurrently active users; refreshes cost at most ≈ 2,300 product calls/s at peak (derived in Section 3) |
| Permission Adapter | Calls each product's checkAccess, accessible-containers and batch hydration APIs | checkAccess p99 30 ms, hydration p99 150 ms |
| Outbox relay | Moves committed outbox rows to Kafka tag-events, keyed by content_ref |
40M events/day, ≈ 16 GB/day in Kafka with 7-day retention; the Postgres outbox itself holds ≈ 12 GB at most (a day of rows) |
| Content-events consumer | Applies moved, deleted, restored and restriction-changed events from the products | ≈ 58/s, applied within ≤ 10 s p99 |
| Bulk job workers | Run merges, deletes, imports and bulk tagging on their own lane | 1,000 rows/s per tenant, 10,000 rows/s per region |
Product search indexers and automation sit outside our boundary. They consume tag-events, and we don't own them.
The forward cache deliberately holds ids, not names. Every read resolves those ids through the tag-metadata cache, drops any tag whose state is deleted, and maps alias_of to the merge target. So a rename of frontend shows immediately for the admin and within seconds everywhere, and a delete hides the tag within seconds, without rewriting 1.2M cache entries.
Interviewer: "That's ten boxes for a join table."
You: "Then let me take each one away and show you what breaks. Drop the forward cache, and 20,800 lookups a second land on shards that are also taking writes. Drop the container-set cache, and every tag page and every autocomplete keystroke has to ask three products who the user is. Drop the bulk workers, and a 17-minute merge shares a pool with a 150 ms p99. The table really is the easy part. The other boxes are all there because we don't own the permissions."
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson