Distributed Transactions — Part 1: Foundations & Core Concepts
maang.io System Design Series
What is this?
Imagine you're booking a honeymoon package through a travel agent. You need three things and you need all of them: a flight, a hotel room, and a rental car. If you get the flight and the hotel but the car falls through, the whole trip is broken — you're stranded at the airport. So the agent doesn't just book them one by one and hope. She calls each vendor and asks a careful question first: "Can you hold this for me — are you ready to commit if I say go?" Only when all three vendors say "yes, held and ready" does she call each one back and say "book it." If even one says "sorry, sold out," she rings the others and says "cancel the hold." Either you get the whole trip, or you get none of it. Never half.
That travel agent is running a distributed transaction. The flight system, the hotel system, and the car system are separate databases owned by separate companies — no shared lock, no shared log, no shared clock. Yet you still want the old, comforting guarantee from the 101 ACID chapter: atomicity — all-or-nothing — but now stretched across machines that can each crash or vanish independently. That is a genuinely hard problem, and this topic is the toolbox for it.
The one-line idea: A distributed transaction makes several independent systems agree to commit together or not at all, even though any of them — and the coordinator itself — can crash at the worst possible moment. Because there's no shared everything, you can't get this for free; every technique here is a different way to buy atomicity, and each charges a different price in latency, availability, or isolation.
We'll keep the travel agent with us through all three parts. She's about to teach you Two-Phase Commit, and then — when she faints mid-booking holding everyone's reservations hostage — she'll teach you why the industry mostly moved on to sagas.
1. Why this exists — the dual-write problem
Inside a single database, atomicity is a solved problem. You wrote BEGIN … COMMIT, and the database's write-ahead log (the Write-Ahead Log (WAL) topic in this tier covers this) guarantees that either every change lands or none does, even across a crash. One log, one lock manager, one commit point. Easy.
The moment your write has to touch two systems, that guarantee evaporates. This is the dual-write problem, and it's everywhere:
- Charge a credit card (Payments DB) and decrement stock (Inventory DB) for one order.
- Update a row in Postgres and publish an "order placed" event to Kafka.
- Move money out of account A at Bank 1 and into account B at Bank 2.
Naively you'd just do them one after another:
paymentsDb.charge(order); // ✅ succeeds
inventoryDb.decrement(order); // 💥 process crashes right here
Now you've charged the customer for an item you never reserved. There is no COMMIT that spans both databases — they don't share a log. Whatever you do, there is a window between the two writes where a crash leaves the world inconsistent. Distributed transactions exist to close that window: to give you a single logical "all-or-nothing" outcome across systems that were never designed to trust each other.
⚠️ The dual-write problem is subtle because the naive code looks correct and works 99.9% of the time. It fails only on the crash, the timeout, the deploy that restarts the pod mid-request. That rare failure is exactly what a Staff engineer is paid to see coming.
2. The core vocabulary
Every technique in this topic is built from the same handful of roles and ideas. Learn these once:
- Coordinator (a.k.a. transaction manager) — the travel agent. The one component driving the transaction to a global decision: commit or abort. It remembers the outcome durably.
- Participant (a.k.a. resource manager) — each vendor. A local system that can commit or roll back its own piece, and takes orders from the coordinator about the global outcome.
- Atomic commit — the goal: all participants commit, or all abort. Never a mix.
- The commit point — the single, durable instant when the decision becomes irreversible. Before it, the transaction can still abort; after it, it must eventually complete. Pinning down exactly where this instant lives is the whole game.
- In-doubt / prepared — the dangerous middle state where a participant has promised it can commit (and is holding locks to keep that promise) but doesn't yet know the global verdict. A participant stuck in-doubt is the source of most distributed-transaction pain.
There are two broad families of solution, and knowing which family you're in is the first thing an interviewer wants to hear:
- Blocking atomic commit — hold everything still until everyone agrees, then flip together. This is Two-Phase Commit (2PC) and its relatives. Strong guarantees (real atomicity and isolation), but participants hold locks and can block if the coordinator dies.
- Eventual atomicity by compensation — let each step commit immediately, and if a later step fails, run "undo" steps to walk it back. This is the Saga pattern (and TCC, a hybrid). Always available, never blocks — but there's no isolation, so half-finished states are briefly visible to the outside world.
Part 2 dissects the machinery of both. For now, hold the mental split: lock-and-agree versus do-and-compensate.
3. How you actually use it — a first taste
You rarely hand-roll 2PC; you either lean on a database/broker that speaks it (the XA standard, PREPARE/COMMIT) or — far more common today — you build a saga. Here's the shape of a saga for placing an order, just enough to see the idea:
Saga: PlaceOrder
Step 1 T1: Payments.charge(order) → C1: Payments.refund(order)
Step 2 T2: Inventory.reserve(order) → C2: Inventory.release(order)
Step 3 T3: Shipping.schedule(order) → C3: Shipping.cancel(order)
Rule: run T1, T2, T3 in order.
If Tk fails, run C(k-1) … C1 in REVERSE to undo the committed steps.
Each Tk is a normal, fully-committed local transaction in its own service. Each Ck is its compensating action — a semantic undo (you can't un-charge a card, but you can issue a refund). If Inventory.reserve fails, you don't magically roll back the payment — you explicitly Payments.refund. The atomicity is real, but it's achieved forward through compensations, not by holding a global lock.
And the single most useful building block underneath all of this — the transactional outbox — is just a table:
-- Written in the SAME local transaction as the business change,
-- so the event can never be lost or double-committed relative to the data.
CREATE TABLE outbox (
id uuid PRIMARY KEY,
aggregate_id uuid,
event_type text,
payload jsonb,
created_at timestamptz DEFAULT now(),
published boolean DEFAULT false
);
We'll see in Part 2 how a background relay drains that table into Kafka to defeat the dual-write problem — with a beautiful side effect: it forces you to make your consumers idempotent, which turns out to be the real secret ingredient of every reliable distributed system.
4. When to reach for it — and when to run away
The most senior instinct in this entire topic is: avoid distributed transactions if you possibly can. They're expensive, they complicate failure handling, and half the time the need for one is a sign your service boundaries are drawn in the wrong place. So the decision tree starts with "can I make this not be distributed?"
| Situation | Reach for | Why |
|---|---|---|
| The data all lives in / can move into one database | A plain single-node ACID transaction | No distribution, no problem. The best distributed transaction is the one you deleted. |
| Short, low-latency, same-organization services that need strong isolation (e.g. bank ledger transfer within one company) | 2PC / TCC | You can afford brief locks and you control every participant. |
| Long-running, cross-service, cross-org business flows that must stay available (order fulfillment, travel booking, loan origination) | Saga (Part 2) | No global locks; each service commits independently; failures handled by compensation. |
| You "only" need to update a DB and emit an event/message reliably | Transactional outbox + idempotent consumer | Sidesteps 2PC entirely — one local transaction plus at-least-once delivery. |
| You just need the effect to eventually converge and can tolerate staleness | Idempotent retries + eventual consistency | Often no transaction is needed at all — just retryable, idempotent operations. |
🚫 The anti-pattern to name out loud: wrapping a chatty microservice call graph in 2PC to paper over bad boundaries. If two services must always commit together, that's often a hint they should be one service (or share one database). Reaching for a distributed transaction should feel like a small defeat, not a default.
💡 Rule of thumb: strong consistency and short-lived → 2PC/TCC; available and long-lived → saga; "DB + message" → outbox. And before any of them, ask whether you can redesign the need away.
5. Key takeaways
- A distributed transaction extends ACID atomicity — all-or-nothing — across independent systems that share no log, no lock, and no clock. It exists to kill the dual-write problem, where a crash between two writes leaves the world half-updated.
- Learn the roles cold: a coordinator drives the outcome; participants commit their own slice; the commit point is the one durable instant the decision becomes irreversible; a participant that's voted yes but doesn't yet know the verdict is in-doubt.
- There are two families: blocking atomic commit (2PC/3PC/TCC — strong isolation, but can block) and eventual atomicity by compensation (sagas — always available, but no isolation and intermediate states leak).
- The everyday workhorse isn't 2PC — it's the transactional outbox + idempotent consumers, which reliably turns "update the DB and publish an event" into a single local transaction plus safe retries.
- The Staff-level move is to avoid distributed transactions when you can: collapse the data into one database, or redesign the boundary, before reaching for coordination machinery.
Next, Part 2 gets under the hood: the exact 2PC prepare/commit protocol and its infamous blocking problem, why 3PC tried to fix it and mostly failed, how sagas and TCC actually run, and how the outbox relay defeats dual writes — all with worked numbers.