Section 2: Late, Never Lost - Requirements (Functional & Non-Functional)

Executive Summary: This section turns Section 1's assumptions into a contract. The functional half lists what tenants and workers can do: schedules, one-off jobs, cancellation, retries, overlap and misfire policies, run history and quotas. The non-functional half puts a number on every promise: dispatch lag, herd drain time, API latency, two availability targets, and durability in-region and across regions. It replaces the impossible "exactly once" with a promise the design can keep: late, never lost, delivered at least once with an idempotency key.

The interviewer caps the marker: "Okay, you've scoped it. What exactly are we promising three thousand teams?" In a scheduler, the guarantees matter far more than the features. A beautiful API that sometimes fails to fire the Payments team's settlement report gives you incidents, not a scheduler. So you write requirements a test can check and an on-call engineer can be paged on. "Jobs run on time" is a wish. "99% of runs are in the tenant's queue within 2 seconds of their scheduled instant" is a promise.

The Functional Contract

Group the functions by caller. The grouping becomes Section 4's API.

Tenant-facing (the scheduling API):

• Recurring schedules: create, update, pause, resume and delete a cron schedule, which always carries an IANA time zone (America/New_York, not "EST"). The platform, never the tenant, computes every next fire time.
• One-off delayed jobs: fire at an absolute time T, or after a delay, up to 30 days out. Checkout's expire-cart-8812 fires 30 minutes after the cart is created.
• Cancel: if the customer checks out, the cart-expiry job must not fire.
• Run now: an ad-hoc run of an existing schedule, for backfills and for checking a fix.
• Policies per job: a retry policy (default 3 attempts total, backoff 30 s, then 2 min, each ±20% jitter, configurable up to 10 attempts), an overlap policy (what happens if the previous run is still going) and a misfire policy (what happens to a fire the platform missed).
• Run history and status: every run's state, attempts, timestamps and error, by job.
• Per-tenant quotas (defaults, raisable): 50,000 active schedules, 5M pending one-offs, dispatch at 1,000 runs/s sustained, burst 20,000.

Worker-facing (the SDK): tenant workers pull runs from their own queue, report start, heartbeat and report completion. The platform never runs tenant code. We rejected a platform-hosted container per run: seconds of cold start for a job whose median is 800 ms, and we would own sandboxing and capacity for every team. Flip trigger: if many small teams have no infrastructure of their own, Section 11 adds managed runners.

Two defaults need defending, because the interviewer will ask:

• Overlap: FORBID. If yesterday's report is still running at 02:00, today's run is recorded SKIPPED_OVERLAP rather than piling up. We rejected ALLOW as the default, which is Kubernetes CronJob's, because it hides slow jobs as silent pile-ups. Tenants whose runs are independent opt into ALLOW.
• Misfire: RUN_ONCE within a 1-hour window. Missed fires coalesce into one run on recovery; anything older is recorded MISSED and the tenant alerted. We rejected RUN_ALL as the default because it turns every recovery into a catch-up storm. Jobs that process each interval's data choose RUN_ALL.

The time-zone rules are functional requirements too. On Sunday 8 March 2026, 02:00 in New York doesn't exist. The rule: a nonexistent local time fires once at the first valid instant after it, so the settlement report fires at 03:00 EDT = 07:00 UTC. An ambiguous local time, like 01:30 on 1 November 2026, fires once, at the first occurrence.

Key Insight: Writing the DST rule into the requirements is the Staff move. Most candidates find the problem during the deep dive, if at all. Naming it here, with the date and the policy, shows you know where schedulers break.

Quick check before moving on: