Scheduler Agent Supervisor
maang.io System Design Series
The Scheduler Agent Supervisor pattern coordinates a distributed, multi-step job so it either completes or is cleanly remediated. A scheduler orchestrates the steps and records their state; agents do the actual work against unreliable remote services; a supervisor watches for failures and triggers retries or compensating actions. It's the resilient backbone behind order fulfillment, provisioning, and workflow engines like Temporal.
This deep dive comes in three parts:
- Foundations & Core Concepts — the three roles and the durable state store.
- Internals — the workflow state machine, idempotency, and detect→remediate logic, with a worked example.
- Interview — Staff/Principal Q&A: the Saga comparison and build-vs-buy (Temporal, Step Functions).
Scheduler Agent Supervisor — Part 2: State Machines, Leases & The Remediation Loop
maang.io System Design Series
What is this?
Back to the kitchen renovation. In Part 1 we met the three people and the logbook on the wall. Now we're going to stand in the room and watch a day unfold — the foreman crossing steps off, a subcontractor knocking on the plumber's door and getting no answer, the inspector walking in at noon, reading a step that's been "in progress" since 9am, and deciding what to do about it.
The whole pattern lives or dies on one thing: the logbook is the truth, and it's written in a disciplined way. Not "the foreman remembers what he told the plumber." Every intention is written down before it's acted on, and every outcome is written down after. If you get that write discipline right, any component can crash at any instant and a survivor can reconstruct exactly what was happening and finish the job. If you get it wrong, you get double-booked hotels and orphaned charges.
The one-line idea: The durable store holds a state machine per step. The Scheduler advances it, Agents do the work and report the outcome idempotently, and the Supervisor reads the machine looking for steps stuck or failed and drives them back to a clean state. No component ever trusts its own memory — the wall is the only truth, and it's guarded by leases and atomic conditional writes so two components can't fight over the same step.
We'll carry the same running example — booking a trip — and this time we'll break it on purpose and watch the machinery clean up, with real numbers.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson