Section 8: Trust, But Fence - Deep Dive: Clocks, Worker IDs & Never Duplicating

Executive Summary: Section 7 chose Snowflake because it needs no coordination per ID. The price is that it trusts two things it doesn't control: the clock, and the claim that worker 17 is running on exactly one node. This section writes the minting loop and closes each path to a duplicate: clock steps back of 10 ms or less are waited out and larger ones make the node refuse; a full 4,096-ID millisecond waits for the next; worker IDs are leased from etcd (10 s TTL, 2 s keep-alive, self-fence at 5 s); and a time high-water mark stops a new owner from reusing the old owner's timestamps. Leo's worker 17 sees its clock step back 40 ms, and we show the exact duplicate the design refuses to create.

The interviewer leans back: "You've told me Snowflake needs no coordination. So what's left that can go wrong?"

"An ID is unique because its tuple (timestamp, region, worker, sequence) is unique. Region is static config, so three things can break it. The timestamp can repeat because the clock went backwards. The sequence can wrap within one millisecond. Or the worker field can be held by two nodes at once. I'll write the loop first, then close each of those."

The Minting Loop

def next_id(self):
    with self.lock:
        if self.fenced():                        # no lease renewal for 5 s (monotonic timer)
            raise Unavailable("fenced")
        now = self.wall_ms()                     # wall clock, ms since 2026-01-01T00:00Z
        if now < self.last_ms:                   # the clock went backwards
            back = self.last_ms - now
            if back > 10:
                self.mark_unhealthy(back)        # stop minting, leave rotation, alert
                raise Unavailable("clock unsafe")
            now = self.wait_until(self.last_ms)  # small step: spin it out
        if now == self.last_ms:
            seq = (self.seq + 1) & 0xFFF         # 12 bits
            if seq == 0:                         # all 4,096 used in this ms
                now = self.wait_until(self.last_ms + 1)
        else:
            seq = 0
        if now > self.reserved_until_ms:         # time high-water mark
            raise Unavailable("reservation not renewed")
        self.last_ms, self.seq = now, seq
        return (now << 22) | (self.region << 19) | (self.worker << 12) | seq

State is committed only after every check passes, so a refused call changes nothing. The function costs well under 1 µs of CPU; only the rare waits are expensive.