Section 5: Turning the Clock into a Range Scan - Data Modeling
Executive Summary: Starting from five access patterns, led by "what is due now in partition p?", we derive four tables (jobs, timers, runs, partitions), every key prefixed by partition_id = hash(job_id) mod 1024. We defend hashing over time ordering, model the run lifecycle, make duplicate fires a primary-key conflict, choose a consensus-replicated (Paxos/Raft) distributed SQL store, and age run history out of the hot store.
The interviewer taps the API sketch: "Fine contract. What's behind it?" The tempting move is a jobs table with a next_run column. Start elsewhere.
You: "Before I name a table, let me list what the system asks of storage and how often. The schema should fall out of that list."
Access Patterns Before Tables
| # | Query | Who asks | Rate (from Section 3) |
|---|---|---|---|
| 1 | What is due now in partition p? | Scheduler node, every 1 s per owned partition | 1,024 range queries/s fleet-wide, every second of the day |
| 2 | Get or update a job by id | API (create, cancel, pause, PATCH) | ≈700 writes/s avg, 3,500/s peak |
| 3 | Runs for a job, newest first | API history reads | Part of ≈2,500 reads/s avg, 10,000/s peak |
| 4 | RUNNING runs whose heartbeat lease expired in p | Lease reaper on the owning node | Continuous; heartbeats update leases at 600/s avg, 6,000/s peak |
| 5 | PENDING runs older than 5 s, or QUEUED runs past their dispatch bound, in p | Outbox sweeper | Continuous, normally returns nothing |
Pattern 1 is the heartbeat of the product, so the target is blunt: "what is due now?" must be a bounded range read over a small, ordered index, scoped to one partition. Patterns 4 and 5 have the same shape.
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson