Section 7: Where Promises Meet Clocks - Deep Dive (Candidate-Driven): Run Guarantees and Time

Executive Summary: You pick the four walls where a scheduler's promises crack: duplicate deliveries, silent workers, local times that don't exist, and runs that fire late or on top of each other. Each lands as mechanism, number and trade-off, then survives pushback. The lesson across all four: the platform can't make side effects happen exactly once, but it can make every duplicate recognisable, so the tenant's code turns it into a no-op.

Section 6's box diagram is on the board. The interviewer caps the marker: "Where do you want to go deeper?" Walk to where the Payments team would be paged at 02:05 on a Sunday:

1. Delivery semantics: where duplicates come from, and why they're harmless.
2. Worker liveness: how we know a run is dead, and what happens if it isn't.
3. Time: what 0 2 * * * in New York means on the night 02:00 never happens.
4. Misfire and overlap: what "on time" means after an outage or a slow run.

Section 8 takes the trigger engine on its own. These four are about the run.

Wall One: At-Least-Once, and Where the Duplicates Come From

Saying "at-least-once" is cheap. Naming every path that produces a second delivery is the senior signal:

The last row is the platform's own duplicate, and it dies in the store. The epoch fence rejects the stale owner's transaction, and even past that, run_id = hash(job_id, scheduled_at_utc) carries a unique constraint, so a second fire of the 2 November settlement is an insert conflict on hash(job_7f3a, 2026-11-02T07:00:00Z). The three re-enqueue rows produce duplicate messages, and those die at StartRun, which only one worker per attempt can pass (Wall Two). The LOST retry is a genuine second execution, and no platform trick stops it: a worker can finish the side effect and crash before CompleteRun, and nothing on our side can tell "never ran" from "ran, didn't report".

So the contract is at-least-once delivery, plus a key that never changes: retries keep the same run_id and increment attempt. The Payments job uses it like this:

def settle(ctx):                                   # ctx.run_id is identical on every attempt
    day = business_date(ctx.scheduled_at, "America/New_York")
    if reports.status(day) == "SENT":
        return                                     # duplicate delivery: clean no-op
    report = build_report(day)                     # pure computation, safe to redo
    bank.upload(report, idempotency_key=f"settlement-{day}")   # downstream dedups
    reports.mark_sent(day, ctx.run_id)

That gives effectively-once outcomes. Two deliveries still produce one settlement report.

The Pushback: "Use a queue with built-in deduplication and the problem goes away." It moves. A deduplicating queue (SQS FIFO, five-minute window) stops two enqueues, and conditional StartRun already neutralises duplicate messages. It can't stop two executions: the duplicate this design fears is the LOST retry, after the message was consumed. Dedup has to live where the side effect lives.