Section 11: Every Door Was Left Unlocked - Follow-up Feature Evolution

Executive Summary: The adversarial phase is over and the interviewer asks what comes next. We add six features: workflows (DAGs), webhook targets, managed execution, priority classes, multi-region cells and sub-second precision. Each one plugs into the core the earlier sections built (the timers table, the transactional outbox, per-tenant queues, deterministic run_id, fencing), and three of them are flip triggers named earlier finally firing. We close with how a live scheduler changes shape: expand-migrate-contract for the schema, additive-only changes to the API.

The interviewer puts the marker down. "It's shipped. Three thousand teams use it. Now the requests come in: the Payments team wants settlement, then reconciliation, then an email. Small teams have no workers at all. Someone wants their runs to beat everyone else's at midnight. And the EU wants its own region. Evolve the design."

Two ways to fail: a feature that needs a new trigger engine (Sections 5 to 8 were too narrow), or revealing you'd already built them all (Section 2's scope was theatre). The answer the interviewer wants is that every feature lands as a new producer or consumer of runs, while the firing loop stays the same.

Key Insight: The run is the stable unit: deterministic id, state machine, lease, attempt number. Anything that creates runs or executes runs inherits all of Section 7's guarantees for free.

Feature One: Workflows - Runs That Create Runs

You: "The tempting wrong answer is chaining by clock: reconciliation at 02:30, email at 03:00. Settlement usually takes 4 minutes, but with default retries it can be on its third attempt after 30 s and 2 min of backoff, and on a slow night reconciliation runs against a settlement file that doesn't exist yet."

So dependencies have to be events, not times. I add two tables in the same store, partitioned the same way (hash(workflow_id) mod 1024):

• workflows (partition_id, workflow_id) holds the graph: the nodes (each one a job template: target queue, payload, retry policy) and the edges. The workflow has its own cron expression and time zone, so 0 2 * * * America/New_York now belongs to the workflow rather than to settlement.
• workflow_runs (partition_id, workflow_run_id) holds, for each node, its state and a pending_deps counter.

The trigger engine doesn't change. The workflow's timer fires through Section 8's loop, and instead of one run row the firing transaction inserts the workflow_runs row and a PENDING run for every node with zero dependencies. Each child gets a deterministic id, run_id = hash(workflow_run_id, node_name), so the duplicate-fire protection carries straight over. Child runs live in the workflow's partition, so CompleteRun's decrement-and-create stays a single-partition transaction.

The one new piece of logic lives in CompleteRun. When a node's run SUCCEEDS, the same transaction decrements pending_deps for each child, and any child that reaches zero gets its run row inserted as PENDING. From there the outbox publishes it as before.

Interviewer: "CompleteRun is at-least-once too. What stops a duplicate completion from decrementing the counter twice?"

You: "The worker fencing. The fencing token is the attempt number plus the start token StartRun hands out, and CompleteRun is conditional on both, with the run still RUNNING. The first completion flips it to SUCCEEDED and decrements in the same transaction; a duplicate finds it terminal and gets STALE_ATTEMPT, a no-op. So does a stale worker holding an older start token."

The senior line: "I'd stop at static DAGs with success-only edges. Once teams ask for branching, loops, dynamic fan-out or compensation, we're growing a workflow engine by accident. Then I'd point them at Airflow for data pipelines or Temporal for durable workflows in code, and keep our scheduler as the thing that triggers them."

Quick check before moving on: