Distributed Job Scheduler System Design

The prompt sounds like something a single Linux box solved decades ago: "Design a distributed job scheduler for our company. Thousands of internal teams need to run jobs at a specific future time or on a recurring cron schedule, and every scheduled run has to actually happen, on time, even when machines fail."

Cron works perfectly until the machine it runs on dies, and then every job on it silently stops. The interview is about what happens when you take that one reliable-looking line in a crontab and spread it across a fleet. Which machine fires the 2 a.m. settlement report, and what stops two machines from both firing it? If the machine that owned it crashes at 1:59:58, does the report run late, run twice, or not run at all? What does "2 a.m. in New York" even mean on the night the clocks skip from 2:00 to 3:00? And when three hundred thousand schedules all say midnight, what happens in that one second?

The Trap Built Into a Familiar Question

Two failures show up again and again:

• The Big Cron Box: puts a database table of jobs behind one scheduler process that polls every second, and stops there. It works on the whiteboard and fails in the first real incident: one process is one failure domain, and the fix the candidate reaches for ("run two of them") fires every job twice. Signal: hasn't separated deciding a job is due from owning the right to fire it.
• The Exactly-Once Promise: draws Kafka, a distributed lock and a confident box labelled "exactly once". The worker that crashes after charging a card but before acknowledging the run breaks that promise, and no amount of infrastructure fixes it. Signal: claims a guarantee the physics doesn't allow, instead of designing the one it does.

The passing performance is quieter and much harder to fake. It hands each timer to exactly one owner through leases, and backs the leases with a fencing check inside the database so a paused node can't fire twice. It gives every run a deterministic id so a duplicate fire is a harmless no-op. It promises at-least-once delivery and makes the outcome effectively once through idempotency keys. It treats time zones, daylight saving, missed runs and overlapping runs as product decisions with stated defaults. And it reduces everything to one promise the interviewer can test: late, never lost.