Section 9: The Clock Decides, the Closer Only Notices - Deep Dive: Closing a Million Auctions

Executive Summary: By the time anything called a "closer" runs, the hard decision is already made: Section 7's accept transaction, under Section 8's clock rule, compared every bid's server-stamped accepted_at with end_time, so the close is bookkeeping, not adjudication. This section builds the bookkeeper. A closer worker per shard sweeps a (status, end_time) index once a second, claims each due auction with a conditional update, and finalises winner, price and the auction.closed event in one transaction. We size it against the 7 p.m. burst of 1,600 closes a second, show that the real burst is the 10,000 notifications a second behind it, give it exactly-once semantics from at-least-once parts, and bolt on an auditor that replays the ledger to check every winner and price. Two probes test whether you understand why the sweep exists at all.

It is Sunday, 19:00:00.000 PT, and auction #7731 is over. Theo's $12,100 arrived fifty milliseconds late and the accept transaction rejected it under the row lock; Elena's $12,000 lost the tie to Marcus's earlier $12,000. Yet nothing in the database says Marcus won. The status column still reads OPEN, and will for another thirty-one seconds.

The interviewer notices. "So right now, who owns this auction?"

You: "The clock does. The cut-off is accepted_at < end_time, evaluated inside the accept transaction, so no bid can be admitted after 19:00:00.000 whenever the closer runs. The closer never decides anything. It records a result that already exists and tells people, exactly once per auction, 14.3 million times a day."

Key Insight: Separate deciding from noticing. Candidates who let the closer's run time define the cut-off have made correctness depend on scheduler lag. Here the closer can be a minute late and nobody's outcome changes, which is why the close budget is 60 seconds at p99 while the bid budget is 500 milliseconds: nobody is waiting on the close.

Finding What Is Due

Every shard carries an index on (status, end_time), and a closer worker per shard runs this once a second:

SELECT auction_id
FROM auctions
WHERE status = 'OPEN' AND end_time <= now() AND owner_node IS NULL
ORDER BY end_time
LIMIT 500;

With 100 million live auctions that query is cheap only because it range-scans the index from the left: due OPEN rows sit at the front and leave the scan the moment they flip. LIMIT 500 bounds one sweep's work; the one-second cadence bounds lag.

owner_node IS NULL is hot-item mode's share of the close: at end_time a promoted auction's owner stops accepting, runs its final flush and clears owner_node in the same transaction, so the sweep never races a flush still in flight and picks the row up on its next tick, which is how #7731 reaches the closer at 19:00:31; the reaper below hands back a dead owner's row.

Each candidate is claimed, not processed:

UPDATE auctions
SET status = 'CLOSING', closer_epoch = :epoch, closing_at = now()
WHERE auction_id = :id AND status = 'OPEN';
-- rows affected = 1  ->  this worker owns the close
-- rows affected = 0  ->  someone else got it; skip

Then one transaction finalises it from the row the accept transactions maintained all week:

BEGIN;
SELECT high_bidder_id, current_price, closer_epoch FROM auctions
  WHERE auction_id = :id FOR UPDATE;
-- abort if closer_epoch != :epoch (fenced out, see below)
UPDATE auctions SET status = 'CLOSED',
  winner_id = high_bidder_id, final_price = current_price, closed_at = now()
  WHERE auction_id = :id;
INSERT INTO outbox (event, payload) VALUES ('auction.closed', :payload);
COMMIT;

For #7731 the row says Marcus, $12,000. For #7732 it says $41 and the NES console's second bidder. For an auction nobody bid on, high_bidder_id is null and the row goes to CLOSED with winner = NULL and a single unsold event to the seller. Section 12 adds CLOSED_RESERVE_NOT_MET on the same path.

Interviewer: "Why not just schedule one delayed message per auction at creation? Then you have no polling at all."

You: "Because that is 100 million outstanding timers, most of them seven days out, and delay queues cap delays at minutes, not a week. I would end up building a timer store and polling that. A single global cron is the other rejection: one hot scanner over 100 million rows instead of 32 cheap index probes. The sweep is boring, which is the point. It stops being right above roughly 20,000 closes a second or under a sub-second close budget; then I would build a hierarchical timer wheel service. Neither is on the horizon."

Quick check before moving on: