Section 7: Where Ten Billion Clicks Land - Deep Dive: The Redirect Read Path

Executive Summary: With the architecture on the board, the interviewer hands over the wheel: "Where do you want to go deeper?" The Staff answer is the redirect path, because that is where 99 % of the traffic and the strictest SLOs live. We dive into four walls that bear the weight: the layered cache, the choice of 302 over 301, partitioning under uneven heat, and consistency across caches and regions. Each lands as mechanism, number, and trade-off, and each draws one pushback. The lesson that carries over to other designs: on a read-dominated system, almost every interesting decision is about which layer answers, and how stale that answer is allowed to be.

The interviewer caps the marker. "You've got twenty minutes. Where do you want to go?"

You don't go to code generation first, even though it's the famous part of this problem. Look at the numbers from Section 3: about 1,200 creates per second against about 116,000 redirects per second, peaking near 350,000. The redirect path has the 99.99 % availability target, the p99 < 50 ms latency budget, and every one of Maya's 2 M newsletter subscribers. So you say: "The read path carries the system. Let me take its four walls in order, then we can move to code generation."

Wall One: The Layered Cache

The mechanism is cache-aside in three tiers, with negative caching at the bottom. Here it is as code, because the order of the checks is the design:

def resolve(code):
    hit = local_lru.get(code)                 # in-process, TTL 60 s
    if hit: return hit
    hit = redis.get(code)                     # 6 shards, TTL 24 h
    if hit is NEGATIVE: return NOT_FOUND
    if hit: local_lru.put(code, hit); return hit
    row = kv.get(code)                        # local region replica
    if row is None:
        row = kv_home_region.get(code)        # brand-new link? (Wall Four)
        # on timeout/unreachable: return NOT_FOUND, do not cache
    if row is None:
        redis.put(code, NEGATIVE, ttl=300)    # 5 min
        return NOT_FOUND
    redis.put(code, row, ttl=86400)           # populate on miss
    local_lru.put(code, row)
    return row                                # caller checks status/expires_at -> 302 or 410

The number that justifies the whole stack: at a ≥ 90 % hit rate, the KV store sees ≈ 35 K reads/s at peak (≈ 11.6 K/s average) instead of 350 K/s. The hot set is about 200 M links, or about 100 GB, which fits in 6 Redis shards × 32 GB, each with a replica. The in-process LRU sits above Redis because a link that is hot right now gets clicked far more often than once a minute. With a 60-second TTL, each of the ~50 redirect servers asks Redis about that link at most once a minute, however many clicks arrive.

Negative caching is the quiet hero. A typo or a truncated link in a forwarded email would otherwise reach the KV store every time, and twice if it also triggers the home-region check. A 5-minute negative entry turns repeated misses into Redis hits. It absorbs repeated misses. A bot probing distinct codes never hits it, and Section 10's per-IP 404 limits handle that.

The Pushback: "Why populate on miss? Write-through on create means Maya's first click is already a hit."
Because most links are never hot. We create 100 M links a day; write-through would push 50 GB a day into a 100 GB cache. That's half the cache every day, filled mostly with links that will be clicked a few times or never, and it would evict the real hot set. Populate-on-miss costs exactly one KV read, a few milliseconds, on the first click of each link. That trade is cheap. Flip trigger: if the measured hit rate falls below 80 %, or the KV store's p99 holds at full uncached load, re-size the cache or reconsider it.

Key Insight: Each cache tier exists because of a specific number: the LRU for per-key heat, Redis for the 100 GB hot set, the negative cache for traffic that has no answer. A tier you can't tie to a number is decoration.

Quick check before moving on: