Section 11: Breaking the Calendar on Purpose - Interviewer Challenges: Failure, Recovery & Abuse

Executive Summary: The interviewer breaks things: a region before the Monday reminder burst, the clocks, the timer, the fan-out, a client release, and the all-hands. Each answer points at the decision that already handles it and names what is still at risk. Together they are the reliability and DR plan.

"Let's break it." The weak answer to every challenge is a new box; the strong one points at a decision you already made and says where it stops working.

Challenge 1: The Region Dies at 08:45

"us-east goes dark at 08:45 on a Monday. Priya's 08:50 reminder is due in five minutes."

What they're really asking: do your recovery numbers survive contact with the busiest minute of the week?

"Three things. Viewing keeps working: the device is the primary cache; Priya's phone renders locally and queues edits. We fail over to her region's async DR replica: RPO ≤ 60 s, RTO ≤ 30 min. Reminders don't wait for the RTO. The ~200 GB timer bucket store is derived data, replicated continuously to warm standby shards in the DR region. They take the leases under a new fencing epoch within a minute, so 08:50 fires at most a minute late. The backstop is the device: it schedules a local alarm for each synced reminder, sharing a reminder_id with the server push, and shows whichever arrives first. An outage costs us reminder accuracy, not the reminders."

"Honesty about the 60 seconds: acknowledged writes can be gone, and the replica's per-calendar seq restarts below numbers devices have seen. So sync tokens carry a store epoch that promotion bumps: an old-epoch token gets 410 Gone and a full resync instead of trusting seq numbers about to be reused. Clients keep each mutation until a later sync shows it in the log, not just until the 200, then replay what the replica lost, made safe by client-generated ids and idempotency keys (Section 4). Web sessions without a local queue genuinely lose up to 60 s of edits. Say so."

The senior line: "If the contract demands RPO 0 under region loss, I flip the async DR replica to a multi-region quorum within a continent and pay roughly 30 ms on every write. I wouldn't pay that by default."

Quick check before moving on: