Section 11: Proving the Books Every Morning - Reconciliation, Reliability & Disaster Recovery
Executive Summary: Section 10's controls stop bad writes; this section proves none got through. We check invariants at three cadences, run a three-way reconciliation (ledger, PSP, bank) that closes by 06:00 UTC with $0 unexplained, and alert on dollars at risk. Then we break things. An AZ loss costs nothing (RPO 0). A region loss costs at most 5 seconds of writes, about $345k at peak, rebuilt from the PSP's record and our own logs while payouts stay frozen.
The interviewer folds their arms. "Your write path validates everything. So how do you know the books are right?" Validation shows what the code meant to do. It can't show what the disk holds, what the PSP did, or what reached the bank. Those need independent evidence.
Invariants at Three Cadences
Section 1's invariants, enforced inline by Section 5's write path, get checked at three speeds, each catching what the faster layer can't see:
| Cadence | Check | Cost | What it catches |
|---|---|---|---|
| Inline (every write) | Postings sum to zero; accounts exist; one currency; sufficient available balance | Microseconds, inside the shard transaction | Buggy callers, malformed requests |
| Nearline (stream, seconds) | account_balance = sum of postings per account; the 64 house sub-accounts add up to the reported house total |
A consumer on the outbox stream | A balance cache that drifted from its postings |
| Offline (daily) | Global trial balance across all 64 shards = 0; each shard's hash chain recomputed and compared to the day's anchored head (Section 10) | A batch job over replicas and the archive | Writes that bypassed the write path, a bad restore or migration, tampering |
Interviewer: "If every transaction sums to zero inline, why would the global trial balance ever be nonzero?"
You: "Inline checks only cover writes that went through the Ledger service. A migration script or a botched restore never passes that check. The trial balance and hash chain test the stored data itself, and either failing pages someone. There's no 'warning' level for an unbalanced book."
Quick check before moving on:
Sign in to continue reading
The rest of this lesson is available with a free account. Signing in with Google or Microsoft is free.
Sign in to read the full lesson