Section 6: Five Boxes and a Clock - High-Level Architecture

Executive Summary: This section puts the ID generator on one whiteboard. In each region there's a thin client library inside every caller, a pool of stateless generator nodes behind client-side balancing, a small etcd cluster that leases worker IDs, a time source the nodes trust, and metrics that watch for clocks and leases going wrong. We trace Leo's chat message from the moment he taps send to the moment it carries ID 104148763141017605. Then we boot a generator node and follow it until it may safely mint its first ID. Finally we name the boxes we left out on purpose, a global sequencer, Kafka, a cache and a database, and say what would bring each one back. The drawing is small because the hard work happens inside one box, and inside one clock.

The interviewer hands you the marker: "Okay. Draw it." Most designs in this course get bigger at this point. This one doesn't. Section 3 sized the fleet at 50 generator nodes worldwide, and Section 5 showed that the only state the system keeps is a handful of etcd keys per region. The whole design fits in one region's worth of boxes, copied five times, with no call that crosses a region on the path of a single ID.

One Region on the Whiteboard

Draw one region and say plainly that the other four are identical copies with a different region ID in their config:

Here is what each box does, and what it doesn't do:

• The client library lives inside every caller process and holds the Section 4 buffer (100 IDs, 1 s discard). Most of the time, getting an ID costs a caller no network hop at all.
• Balancing happens in the client, across the list of healthy nodes from service discovery. A regional L4 load balancer would also work. The thing that matters is the retry rule: when a node answers UNAVAILABLE, the library tries a different node straight away.
• Generator nodes are stateless apart from three values held in memory: the leased worker ID, the last millisecond they minted in, and the sequence within that millisecond. At 25,000 requests/s usable per node, the largest region needs 16 nodes plus 2 spare = 18, which is 18 of its 128 worker slots. That scarcity is also why minting lives in a service rather than an embedded library in every caller: a busy region's caller fleet would need far more leases. Embedded mode only makes sense when a caller fleet fits in the worker-ID space (about 110 processes, the slots the service leaves free in a region, or up to ~500 if that one region also borrows the three spare region codes, which are held for new regions first) and needs sub-millisecond IDs, and Section 11 builds it.
• etcd is the only coordination in the design, and it's off the hot path. Nodes talk to it once at startup and then send a keep-alive every 2 seconds. Minting never waits on it.
• chrony/NTP keeps each clock within 10 ms of the reference time, usually about 1 ms. It's a dependency even though it's barely visible on the drawing, because the timestamp field is only as good as the clock that fills it.
• Metrics count the events that come before a duplicate: clock rollbacks, self-fences, lease churn, and waits for sequence overflow.

You: "Notice what isn't on the request path. A NextIds call reads the clock, does some shifts and ORs on three numbers in memory, and returns. It touches no disk, no lock shared with another node, and no other region. That's why one node can serve tens of thousands of requests a second and why the p99 is set by the network, at ≤ 2 ms in-region."

Interviewer: "And if the whole generator pool in a region disappears?"

You: "Callers keep running on their buffers for up to about a second. After that, minting stops in that region. That's a real outage, and Section 9 walks through failing over to a neighbouring region. What it never does is guess. A missing ID is an outage. A duplicate ID is corruption."

Quick check before moving on: