Section 3: Three Terabytes and a Permission Problem - Scale Estimation & Workload Analysis

Executive Summary: This section derives the tagging service's scale step by step, from tenants and items, to assignments, to bytes, to shards, to writes and reads per second. The verdict surprises most candidates: the whole suite's tags fit in about 3.2 TB and peak at about 1,400 writes a second, so raw volume is not the problem. The numbers instead point at three different things. The forward read (tags on every issue, page and PR) is the hottest path at 20,800 lookups a second, so it gets a cache. The tag page is rare at 70 a second, but each one is expensive and bound by permissions. And one skewed tenant, Meridian, matters more than the average of all the others. You'll also see why storage and blast radius, not throughput, set the shard count at 64.

The interviewer caps the marker. "Before you draw a single box, how big is this thing?"

This is where the "it's just a join table" instinct gets tested. If you estimate well, you'll find the table really is easy, and the numbers will tell you exactly where the hard parts are. If you skip the math, you'll end up building a giant write pipeline for a write rate a single database could handle, and treating the tag page as an afterthought.

State the inputs out loud and invite correction. The interviewer agrees to 250,000 active tenants and 10M daily active users across all products.

From Tenants to Assignments

Start with what exists, then with what links it:

Containers (projects, spaces, repos):
  250,000 tenants × 150 per tenant           = 37.5M containers
  largest tenant                             = 20,000 containers

Taggable items (issues, pages, PRs):
  250,000 tenants × 40,000 per tenant        = 10B items
  Meridian Group                             = 200M items
  restricted (item-level permissions)        = 3% of items

Assignments (tag ↔ item):
  10B items × 40% tagged × 2.5 tags each     = 10B assignments
  Meridian: 200M × 0.4 × 2.5                 = 200M assignments

Distinct tags:
  250,000 tenants × 2,000 per tenant         = 500M tag rows
  Meridian                                   = 300,000 tags

Average tag:  10B / 500M                     = 20 assignments
(container, tag) pairs:  500M × ~3 containers = 1.5B pairs
  → 1.5B / 37.5M                             = 40 distinct tags per container

Pause on that "20 assignments per tag", because it's the average that hides the problem. Meridian's frontend tag has 1.2M assignments across 800 containers. That's 1.2M / 20 = 60,000 times the average tag.

You: "If I sized everything for the average tag, every design would work. The design has to survive frontend. So from here on I'll carry two numbers for every read: the average case and Meridian."

Interviewer: "Why does the per-container pair count matter?"

You: "Because counts and suggestions will be aggregated per container, so we never leak a tag name from a project you can't see. That table has 1.5B rows, and I want to know its size before I commit to it."