Replication — Part 1: Foundations & Core Concepts
maang.io System Design Series
What is this?
Picture a bank before computers. The head office keeps the one authoritative ledger — the single book where every deposit and withdrawal is posted. But customers don't all live next to head office, and if that one book ever burns, the bank is finished. So the bank keeps copies of the ledger at every branch. Any teller, anywhere, can answer "what's my balance?" by reading the local copy. And if head office burns down, a branch copy can be promoted to become the new master. Nobody loses their money.
That's replication: keeping the same data on multiple machines connected by a network. Every serious database, cache, message queue, and file store does it. The whole subject boils down to two questions that sound simple and turn out to be deep:
- Who is allowed to accept a change (a write)?
- How, and how quickly, do the copies find out about it?
Answer those two differently and you get every replication design that exists — from PostgreSQL streaming to a follower, to MySQL rings spanning two continents, to Cassandra's leaderless quorums (the store we dissected in the Cassandra topic).
The one-line idea: Replication keeps redundant copies of your data on several nodes so the system survives machine death, serves reads close to users, and scales read throughput — and the entire hard part is keeping those copies in agreement while the network, and the machines, keep failing.
We'll carry the branch-ledger picture through all three chapters: head office = the machine that owns the truth, branches = the copies, "posting an entry" = a write, "checking a balance" = a read.
1. Why replicate at all?
Before you learn the mechanics, be crystal clear on why you'd pay this cost — because replication is not free. You're now maintaining N copies that can disagree. Three motivations, and you should always know which one you're actually buying:
- Availability / durability. If one node dies, another copy keeps serving. A branch burns; the bank survives. This is the reason people who say "replication" most often mean.
- Read scalability. Reads can be answered by any copy. Ten branches answer ten times as many balance inquiries as one head office could. You spread read load across replicas.
- Latency (locality). Put a copy near the user. A customer in Tokyo reads a Tokyo branch instead of round-tripping to a New York head office. Physics — the speed of light — is the constraint you're buying around.
⚠️ Notice what replication does not buy you for free: write scalability. In most designs there's still one place that accepts writes for a given piece of data, so replication multiplies your read capacity, not your write capacity. Scaling writes is what Partitioning (Sharding) is for — the sibling topic. The two are orthogonal and almost always used together: partition to split the write load, replicate each partition for safety. Keep them separate in your head.
2. The core vocabulary
A few words you must be able to use precisely, because interviewers listen for them:
- Replica — one copy of the data (one branch's ledger). A node can hold many replicas of many partitions.
- Leader (a.k.a. primary, master, source) — the replica that accepts writes for a piece of data. Head office.
- Follower (a.k.a. replica, secondary, standby, hot standby) — a copy that receives changes from the leader and typically serves reads only. A branch.
- Replication log — the ordered stream of changes the leader ships to followers so they can replay them (Part 2 goes deep on the formats).
- Replication lag — how far behind a follower is. The gap, in time or in bytes, between "posted at head office" and "visible at this branch." Lag is the source of nearly every replication bug you'll ever chase.
- Replication factor (RF) — how many copies you keep. RF=3 means three ledgers.
3. The three topologies (the mental model)
There are exactly three ways to answer "who can accept a write?" Everything else is a variation. This is the single most important diagram in the topic — memorize the shapes.
3.1 Single-leader (leader/follower)
One replica is the leader; every write goes only to it. The leader applies the change locally, then streams it down its replication log to the followers, which replay it. Reads can go to the leader or any follower.
This is the default of the relational world — PostgreSQL streaming replication, MySQL binlog replication, MongoDB replica sets, and the leader-of-a-partition model inside Apache Kafka. It's the simplest to reason about because there are no write conflicts: with one place accepting writes, every change has an unambiguous order. Head office's ledger is the truth; branches just copy it.
- 💡 When to reach for it: almost always, as your first answer. If a single leader can absorb your write volume, single-leader replication gives you durability and read scaling with the least conceptual pain. The overwhelming majority of systems live here.
3.2 Multi-leader
Now you have two or more leaders, each accepting writes, each replicating its changes to the others. The classic reason is multi-region: a leader in the US and a leader in the EU so that writes are fast on both continents (you write to your local leader instead of crossing an ocean). Other cases: multiple datacenters for disaster tolerance, or offline-capable clients (your phone's calendar is effectively a leader that syncs later).
The price is brutal and unavoidable: write conflicts. If a US customer and an EU customer both change the same record at the same second, both leaders accept it locally, and now the two ledgers disagree. Somebody has to decide who wins — and that conflict resolution is the hard, bug-prone heart of multi-leader (Part 2).
- 💡 When to reach for it: when you genuinely need low-latency writes in multiple regions, or offline writes, and you can define a sane conflict-resolution rule. If you can avoid it, avoid it.
3.3 Leaderless (Dynamo-style)
Throw out the leader entirely. Any replica accepts a write directly from the client, and the client (or a coordinator on its behalf) writes to several replicas at once and reads from several at once. Correctness comes not from a boss but from overlap: if you write to enough copies and read from enough copies, the two sets are guaranteed to share a node holding the newest value. This is the quorum idea — the R + W > RF rule from the Cassandra topic.
This is Amazon DynamoDB, Apache Cassandra, and Riak. Every branch can post an entry; the branches agree by majority vote rather than by deferring to a head office.
- 💡 When to reach for it: when you need to stay writable through node and even datacenter failures with no failover step, and you're willing to manage consistency yourself with quorums and background repair. It's the "always-on, high-write" shape from the CAP theorem discussion — leaning A and P.
4. Synchronous vs asynchronous — the other big dial
Independent of topology, there's a second choice: when the leader ships a change to a follower, does it wait for the follower to confirm before telling the client "done"?
- Synchronous — the leader waits for the follower's acknowledgement before confirming the write to the client. Guarantee: the follower is guaranteed up-to-date, so if the leader dies you can promote that follower with zero data loss. Cost: the write is only as fast as the slowest waited-on follower, and if that follower is down or slow, the write blocks. A fully-synchronous system where every follower must ack is fragile — any one sick replica stalls all writes.
- Asynchronous — the leader applies the write locally, replies to the client immediately, and streams the change to followers in the background. Fast and highly available (a dead follower never blocks a write), but if the leader dies before a change reaches the followers, that acknowledged write is lost. This is where replication lag lives.
4.1 Semi-synchronous — the pragmatic middle
Almost nobody runs fully synchronous. The standard production compromise is semi-synchronous: make one follower synchronous and the rest asynchronous. Now you always have (at least) one guaranteed-current copy for safe failover, but a single slow follower can't stall you because the others are async. If the sync follower falls over, another async one is promoted to be the sync one. MySQL and PostgreSQL both offer exactly this knob.
💡 Rule of thumb: asynchronous is the default (fast, available, but can lose the last few writes on failover); semi-synchronous is what you use when losing acknowledged writes is unacceptable; fully synchronous is almost always a mistake outside of consensus systems (Part 2). The quorum writes of leaderless systems are a third answer — "synchronous to a majority" — which is why they tolerate failure so gracefully.
5. How you actually use it — a concrete taste
You rarely hand-code replication; you configure it. Here's PostgreSQL, single-leader, streaming to a follower, set to semi-synchronous (wait for one replica named replica1):
# On the LEADER (postgresql.conf)
wal_level = replica # emit a Write-Ahead Log rich enough to replicate
max_wal_senders = 10 # allow up to 10 followers to stream
synchronous_standby_names = 'ANY 1 (replica1, replica2)' # semi-sync: wait for any 1
-- On the FOLLOWER, once streaming, route read-only traffic here:
SELECT balance FROM accounts WHERE id = 42; -- served from the replica, offloads the leader
Two things worth noticing. First, PostgreSQL replicates by shipping its Write-Ahead Log — the exact same WAL you met in the Write-Ahead Log (WAL) topic, now doing double duty: crash recovery and replication. Second, synchronous_standby_names is the entire sync/async/semi-sync decision expressed as one config line. That's typical — the concepts are deep, but the knobs are few.
6. When to reach for it — and when not
| You need… | Reach for | Because |
|---|---|---|
| Survive a node dying; scale reads | Single-leader | Simplest, no conflicts, the default |
| Low-latency writes in several regions, or offline writes | Multi-leader | Writes stay local; accept conflict-resolution cost |
| Always-writable, high write volume, no failover step | Leaderless (quorum) | No leader to fail over; managed via R + W > RF |
| Zero data loss on failover | Semi-synchronous | One guaranteed-current follower to promote |
| Fastest writes, can tolerate losing the last few on failover | Asynchronous | Never blocks on a slow/dead follower |
And when not to replicate for a given concern: don't reach for replication to scale writes (that's partitioning), and don't reach for multi-leader when a single leader would do — you'd be signing up for conflict resolution you didn't need. The senior instinct is to start single-leader-async and add complexity only when a specific requirement forces it.
7. Key takeaways
- Replication = redundant copies for availability, read-scaling, and locality. It does not scale writes — that's partitioning, and the two are used together, not interchangeably.
- Three topologies, decided by "who accepts writes": single-leader (one writer, no conflicts, the default), multi-leader (many writers, must resolve conflicts, for multi-region/offline), leaderless (any replica writes, agreement by quorum, always-on).
- Second dial — sync vs async: synchronous = safe failover but blocks on slow followers; asynchronous = fast and available but can lose the last writes; semi-synchronous (one sync follower, rest async) is the pragmatic default when data loss is unacceptable.
- Replication lag — how far a follower trails the leader — is the root of most replication bugs, and every guarantee in Part 2 exists to tame it.
- Start simple: single-leader, async, add a semi-sync follower if you can't lose writes. Add multi-leader or leaderless only when a concrete requirement (multi-region writes, always-on) demands it.
Next, Part 2 opens the machine: how the replication log is actually built (statement vs WAL vs logical), the precise consistency guarantees lag forces you to choose (read-after-write, monotonic reads, consistent prefix), and the scariest operational moment in all of replication — failover, split-brain, and how leader election stops two branches from both thinking they're head office.