</> MAANG.io
system design · 301

Advanced

Master system design interviews with scalable architecture patterns, distributed systems, and real-world design challenges.

0/480 solved 0% complete

Section 11: Four Teams at the Door - Follow-up Feature Evolution

Executive Summary: The generator ships, and Leo's "boarding now" message gets its ID in well under 2 ms. Then the requests arrive. A trading-style team wants IDs without a network hop. The storage team wants the shard derivable from the ID. A new team insists on 128-bit keys. Product wants public URLs that don't reveal creation time or volume. A fifth ask, per-tenant ID spaces, gets a polite no. This section adds each feature without changing the 64-bit layout every table depends on: embedded mode reuses Section 8's lease protocol, shard-aware IDs get their own namespace, UUIDv7 interop is a one-way widening, and opaque public IDs are a translation at the gateway.

The interviewer's tone changes. The adversarial part is over. "It's live. Four teams are waiting outside my office with feature requests. Walk me through how the design absorbs them, or tell me which ones it shouldn't."

There are two ways to fail here. One is to discover the bit layout has to be rebuilt. The other is to say yes to everything. The answer that scores routes each request through a door the earlier sections deliberately left open: the client library, the spare worker and region bits, the Decode API, and the rule that the generator never leaks outside the company.

Doors left open: the core 64-bit Snowflake design opens four doors (client library, spare worker and region bits, Decode API with timestamp in high bits, internal-only generator) leading to embedded mode, UUIDv7 interop, shard-aware IDs, opaque public IDs, and declined per-tenant ID spaces


Embedded Mode: The Library Becomes the Generator

The first team runs a matching engine. Section 6 put the generator one network hop away, and Section 2 promised a p99 of ≤ 2 ms in-region. For most callers the prefetch buffer hides that hop completely. For this team, a 2 ms tail on a cold buffer is a lot of time, and minting in-process costs well under 1 µs of CPU.

The decisions ledger anticipated this: embedded mode is right when the caller fleet fits in the worker-ID space and sub-millisecond matters. The mechanism is the client library itself. It already knows how to talk to the generators. In embedded mode it is a generator: it acquires a worker-ID lease from the regional etcd, honors the 10 s TTL with keep-alives every 2 s, self-fences after 5 s without a renewal, and waits out the previous owner's reserved_until_ms before minting.

You: "The hard limit is slots, not code. A region has 128 worker IDs. The largest region's service uses 18, so about 110 are free. I'd carve those into a separate embedded range so a misconfigured caller can never take a slot the service needs. If one region needs more, it can borrow one of the 3 unused region codes, which adds another 128 slots. That's where the 'about 500 processes' ceiling in our decisions comes from: 110 plus 3 × 128 is 494, and only one region could claim all three spare codes. Those three codes are the same reserve Section 9 spends on service growth and Section 3 keeps for new regions, so they're allocated by a written policy: new regions first. Every code lent out is a region we can no longer open without a re-layout."

Interviewer: "Then why not make embedded mode the default for everyone?"

You: "Because every embedded caller becomes part of our correctness boundary. Its host's clock discipline, its GC pauses and its lease handling can now produce a duplicate primary key in a table we don't own. With the service, I control about 50 hosts with tuned chrony and alerting on rollback. With embedded mode, I'm trusting hundreds of hosts that belong to other teams. So it's opt-in, it's limited by slot quota, and the library refuses to mint, and falls back to the service, the moment its clock or lease looks unsafe."

Key Insight: Embedded mode doesn't change the algorithm. It changes who runs it. The cost isn't CPU or bits, it's the size of the set of machines that can break uniqueness. That's why it's an exception for a few latency-critical callers and not the default.

Quick check before moving on:


Sign in to continue reading

The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.

Sign in to read the full lesson

Sign in to MAANG.io

Use your Google or Microsoft account — no password to remember.

Continue with Google Continue with Microsoft

Please accept the terms above to continue.