Write-Ahead Log (WAL) — Part 1: Foundations & Core Concepts
maang.io System Design Series
What is this?
Picture a busy restaurant kitchen at the height of the dinner rush. A waiter takes an order and could, in theory, sprint straight to the kitchen and shout it at whichever cook is free. But no good restaurant works that way. The waiter first writes the order on a ticket and clips it to the rail — before a single pan is touched. Only then does cooking begin. Why the extra step? Because if the waiter trips, forgets, or the cook wanders off mid-dish, the ticket is still there. Anyone can walk up to the rail, read what was ordered, and either finish the dish or start it over. The ticket rail is the single source of truth for "what was promised." The plates are just the result of replaying those tickets.
That ticket rail is a Write-Ahead Log. A database that wants to never lose your data does exactly what the restaurant does: before it changes the actual data — the "plates," which live in pages on disk — it first appends a small note describing the change to a durable, append-only log. The note goes down first. That's the whole idea, and it's the single most important durability mechanism in every serious database on Earth: PostgreSQL, MySQL/InnoDB, SQLite, Oracle, SQL Server, RocksDB, and — as you saw in the Apache Cassandra topic — Cassandra's commit log is exactly this. We'll carry the kitchen ticket rail analogy through all three parts.
The one-line idea: Never change the data until the intention to change it is safely on durable storage. Write the log record first, force it to disk, and then you're allowed to touch the actual data — because if you crash at any moment, the log tells you how to finish the job. Log-before-data is the rule the entire technique is named after.
1. Why WAL exists — the problem before it
To feel why WAL matters, imagine a database without one. You run:
UPDATE accounts SET balance = balance - 100 WHERE id = 'alice';
UPDATE accounts SET balance = balance + 100 WHERE id = 'bob';
A money transfer. Two rows change. Now suppose the machine loses power right after Alice's row is written to disk but before Bob's. Alice is down $100, Bob never got it, and $100 has simply vanished. That's an atomicity violation — the "all-or-nothing" promise from the ACID chapter, broken. Worse, some databases buffer changes in RAM and write them out lazily; a crash could lose an "acknowledged" write entirely — a durability violation.
The naive fixes are all bad:
- "Write every change straight to its data page on disk, immediately." Now every tiny update is a random disk seek — jump to Alice's page, jump to Bob's page (they could be anywhere on disk). Random I/O is brutally slow, and you still aren't atomic, because you can't write two far-apart pages in one indivisible step.
- "Keep everything in memory and hope." Fast, but a crash loses acknowledged data. Non-starter.
WAL threads the needle. Instead of chasing scattered data pages on the hot path, it writes one thing — a compact log record describing the change — to a file that only ever grows at the end. Appending to the end of a file is sequential I/O: the disk head (or SSD controller) never has to seek. Sequential writes are often 100× faster than random ones. So WAL gives you both durability (the change is safely on disk before you ack) and speed (one cheap sequential append instead of several expensive random writes). The actual data pages get updated later, lazily, in the background — and if a crash interrupts that, the log lets you redo the lost work.
💡 The mental unlock: WAL decouples "the change is safe" from "the change is applied to its final resting place." The log makes the change durable cheaply and instantly; moving it into the real data pages happens on a relaxed schedule. This is the same decoupling you saw power Cassandra's LSM-tree write path, and it's why WAL and log-structured storage are cousins.
2. Core concepts & vocabulary
Four terms carry the whole topic. Learn these and the internals in Part 2 will click.
Log record. One entry in the log describing one change. A record typically says: which transaction made the change, which data page and byte range it touched, and enough information to redo it (the after-image: "byte 40 of page 7 became value X") and often to undo it (the before-image: "byte 40 of page 7 was value W"). Records are appended, never modified. The log is append-only, exactly like Cassandra's SSTables were immutable.
LSN — Log Sequence Number. Every log record gets a unique, monotonically increasing number: its position in the log. LSN 1000, 1001, 1002… Think of it as the ticket number stamped on each kitchen ticket. LSNs give the database a total order over all changes and a way to say precise things like "this data page reflects all changes up to LSN 1050" or "start recovery from LSN 900." Almost every recovery decision is an LSN comparison. Each data page even stores the LSN of the last log record that modified it (its pageLSN) — a fact that becomes the linchpin of recovery in Part 2.
Redo information. The "how to reapply this change" content — the after-image. If a committed change hadn't yet made it into the data pages when we crashed, recovery redoes it from the log: re-stamp the ticket's result onto the plate.
Undo information. The "how to roll this change back" content — the before-image. If a transaction was still in-flight (uncommitted) when we crashed, its half-finished changes must be undone so we don't leave garbage behind: scrape the half-cooked dish and restore the plate to how it was.
💡 A clean way to remember it: redo carries you forward (finish committed work the crash interrupted), undo carries you backward (erase uncommitted work the crash left behind). A crash-recovery pass usually needs both, and Part 2's ARIES algorithm does them in a strict order.
3. How you actually use it
Here's the reassuring part: as an application developer you almost never touch the WAL directly — it's the invisible machinery under COMMIT. But you configure and observe it constantly, and knowing the knobs is what separates a senior operator from someone cargo-culting defaults. In PostgreSQL, for instance:
-- The single most important durability knob in Postgres.
-- ON = fsync the WAL to disk before COMMIT returns (durable, default)
-- OFF = ack the commit before the WAL is safely on disk (fast, DANGEROUS)
SHOW synchronous_commit;
-- How much WAL to keep for crash recovery + how often to checkpoint (Part 2).
SHOW max_wal_size; -- e.g. 1GB
SHOW checkpoint_timeout; -- e.g. 5min
-- Where the WAL physically lives (each file is a 16MB segment by default).
-- SELECT * FROM pg_ls_waldir(); -- lists the segment files
The one line worth burning into memory is synchronous_commit. With it on, when COMMIT returns, your change is guaranteed on durable storage — the ticket is physically clipped to the rail before the waiter says "got it." Turn it off and commits get faster because the database acks before the WAL hits disk — but a crash in that window silently loses acknowledged transactions. That's a real, deliberate trade-off (durability for latency) that a Staff engineer makes with eyes open, never by accident.
⚠️
synchronous_commit = offis not the same as disabling the WAL or turning offfsyncentirely. With it off you can still lose recent, acknowledged commits on a crash, but the database stays consistent (no corruption). Turning offfsyncaltogether risks actual corruption. People conflate these constantly — don't.
4. When to reach for it — and when someone else already did
WAL isn't usually a thing you build; it's a thing you rely on and reason about. You "reach for it" in two situations:
- You're choosing or tuning a datastore and need to reason about its durability guarantees, commit latency, and recovery time. Every question like "will I lose data if this box loses power?" or "why did failover take 40 seconds?" routes back to WAL behavior.
- You're building a system that itself needs crash-safe, atomic, append-first state — a storage engine, a message broker, a ledger, a metadata service. Then you implement (or adopt) the pattern directly.
| Situation | Is WAL the right lens? |
|---|---|
| Any relational/transactional DB (Postgres, MySQL, SQLite, Oracle) | Yes — it's already there; tune and understand it. |
| LSM-tree stores (Cassandra, RocksDB, HBase) | Yes — the "commit log"/WAL is the durability layer under the memtable. |
| Replication & failover (Postgres streaming replicas, Kafka) | Yes — the log is the replication stream (Part 3). |
| Building a ledger / event store / broker from scratch | Yes — append-only WAL is the core primitive. |
| Pure in-memory cache where loss is acceptable (Redis as a cache) | No — if you can afford to lose it on restart, durability machinery is wasted cost. |
| Stateless service with no local durable state | No — nothing to log; push durability to the datastore behind you. |
The alternatives-at-a-glance: some systems chase durability with shadow paging (write new pages elsewhere, then atomically flip a pointer — used by LMDB), or copy-on-write B-trees. These avoid a separate log but pay in write amplification and fragmentation. WAL won the industry because sequential-append-then-lazy-apply is simply the best all-round bargain — and, as Part 3 shows, the log turns out to be reusable as a replication and change-data-capture stream, which the alternatives don't give you for free.
Key takeaways
- Log-before-data is the rule. Before changing the actual data pages, append a record describing the change to a durable, append-only log and force it to disk. If you crash, the log tells you how to finish or unwind — that's how WAL delivers ACID durability and atomicity.
- It's fast because it's a log. Appending sequentially to the end of a file is ~100× cheaper than the random disk seeks you'd need to update scattered data pages directly. WAL buys durability and throughput by deferring the expensive random work.
- Four words unlock everything: a log record (one change), stamped with an LSN (monotonic sequence number = total order), carrying redo info (after-image, to reapply committed work) and undo info (before-image, to roll back uncommitted work).
- You mostly tune and reason about it, rarely build it.
synchronous_commitin Postgres is the canonical durability-vs-latency dial; understanding it is a senior-vs-junior tell. - It's the same decoupling — "make it durable now, apply it to final storage later" — that powers Cassandra's commit log and the whole LSM family. WAL is the shared foundation the ACID guarantees, the storage engines, and (Part 3) replication all stand on.
Next, Part 2 goes under the hood: the exact anatomy of a log record, how the buffer pool and the WAL cooperate on the write path, checkpoints that bound recovery time, and the famous ARIES three-pass recovery algorithm that turns a pile of log records back into a correct database.