Section 8: One Hand on Every Alarm Clock - Deep Dive: The Trigger Engine

Executive Summary: The trigger engine is where "every run happens, on time" stops being a slogan. Four moves: partitions claimed through etcd leases; a fenced, once-a-second firing transaction; failover that costs ≈10–13 s of lateness, never a run; and a paced herd that drains 300,000 midnight runs in 12 s without one tenant starving the rest.

The interviewer taps "Scheduler nodes" on your Section 6 diagram. "Something fires the settlement run at 07:00:00. Who, exactly? You have sixteen machines and 52 million timers." You restate the job as four constraints:

1. One owner per due timer at any moment. Not "one execution" (Section 7 already gave that up for at-least-once), but one firer.
2. Dispatch lag p99 ≤ 2 s up to the peak-minute rate of 12,000 runs/s.
3. 25,000 runs/s of firing capacity, about 1,600 per node.
4. A node dying costs lateness, never a run.

Ownership: Partitions Are Leased, Not Placed

Every job lives in one of 1,024 logical partitions, hash(job_id) mod 1024, and every table's primary key starts with that partition_id (Section 5). Sixteen nodes split them 64 each. Each node holds an etcd lease with a 10 s TTL, renewed every 3 s, with its partition keys attached, so if the node dies all 64 expire together.

etcd doesn't assign anything; nodes claim. Each node watches the membership prefix, computes its fair share (⌈1,024 / live nodes⌉), and claims free keys with an etcd transaction that succeeds only if the key doesn't exist (create_revision == 0). A node over its share releases keys; one under it claims them. A partition is a unit of ownership, not data: moving partition 417 moves one etcd key and no rows. When a 17th node joins, the share drops to ≈60 and every node hands over about 4.

The subtle part: etcd decides who should fire; the store decides who may. Each partitions row carries owner_epoch, which a new owner bumps on takeover. That number, not the lease, is the authority. (Hardened: use the claim's etcd revision as the epoch, SET owner_epoch = :rev WHERE owner_epoch < :rev, so a new owner that pauses before bumping can't overwrite a newer one.)

The pushback: "Why not let all sixteen nodes race for due rows with SELECT … FOR UPDATE SKIP LOCKED?"

"Below about 2,000 runs/s on one Postgres, that's what I'd build. At 25,000 runs/s we're on distributed SQL, where sixteen nodes scanning the same ranges contend on every hot row at midnight. With partition ownership, a firing transaction has no competitor."

The pushback: "You already run a strongly consistent store. Why a second consensus system? Keep the leases in the partitions table."

"You could. A lease row renewed by conditional update every 3 s is about 340 writes/s for 1,024 partitions: trivial. I keep etcd for watch-driven membership and rebalancing, and because the coordinator stays up while store ranges elect leaders. The fence is identical either way, so if etcd ever costs more than it saves, leases move into the store and nothing else changes."

Quick check before moving on: