Section 10: A Bad Day at Via Verde - Reliability, Failure & Operations
Executive Summary: This section walks the failure matrix for a system whose worst outage window is a lunch hour: the PSP dies (fail closed — "ordering paused," never a silent queue of unpaid promises), the database dies (ordering spends its 99.9% budget while browsing survives on the CDN), the CDN runs stale (a bounded, 30-second lie), the kitchen display goes dark, the push notifications don't arrive. Then the parts interviews skip and operators live in: the store-open invariant, who may change what, and the reconciliation sweep that guarantees money and order state never disagree silently. It ends with the question that separates candidates who've shipped from candidates who've studied: "What's your worst incident?"
The interviewer leans back. "It's 12:20 on a Tuesday. Something just broke. Convince me Maya still gets lunch — or at least the truth." The right opening stance: components here fail into two postures, and knowing which is which is the design. The browsing path fails soft — stale is acceptable, because a menu one version behind still sells salads. The money path fails closed — wrong is unacceptable, because a charge without an order, or an order without a kitchen, is a debt the brand pays with interest.
The Failure Matrix: Soft on Lettuce, Closed on Money
Take the components one at a time, answering two questions each: what still works, and what does the customer see?
| Component down | What stops | What keeps working | The honest behavior |
|---|---|---|---|
| PSP (auth ~500 ms–1 s normally) | Checkout | Browsing, carts, the kitchen finishing in-flight orders | Fail closed. No "we'll charge you later" queue — an unpaid order is a promise the restaurant may eat. The app says "ordering is paused, back shortly"; the menu stays up. Retries reuse the same idempotency key (D9), so a flaky PSP can time out, recover, and be retried without ever double-charging Maya's $17.45 |
| Database | Ordering, the 86 board, order status | Menu browsing, served from the CDN copy of v47 + the last overlay | This is what the 99.9% budget is for — about 8.8 hours a year, and the design spends it only on the ordering path. Browsing degrading to a stale cached menu instead of a blank page is D6's two-document split paying off: the thing that changes weekly survives the outage of the thing that changes per-order |
| CDN stale | Menu freshness | Everything | A menu one version behind is fine — prices are enforced server-side at checkout anyway (the reprice-and-confirm from Section 7 catches drift). The availability overlay's short TTL bounds the worst case: a customer can believe avocado exists for at most ~30 s after Marco's 86, and the order-time check (D7) catches the stragglers |
| Kitchen display | The tablet's view | The queue itself — it lives in order_events, not the tablet |
One line, because Section 9's Challenge 4 already earned it: the tablet is a view, never the truth. Reconnect and replay; printer fallback meanwhile |
| Notifications | READY pushes | The order, the kitchen, the counter | Best-effort by design. If the push fails, Maya's order still turns READY on the status screen and her name still gets called. A notification system that can block an order pipeline has its dependencies backwards |
Key Insight: Every row is one of two verbs. The browsing path degrades — staleness bounded, truth enforced later at order time. The money path halts — visibly, with an apology, never silently queuing obligations. A system that fails closed on payments and soft on menus has its values in the right order; one that does the reverse is a lawsuit with a loading spinner.
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson