Section 12: Growing Up Without Starting Over - Follow-up Feature Evolution: Replication, Transactions & Beyond

Executive Summary: Four evolutions arrive, each at a trigger the decisions ledger already named. Cross-cell replication is asynchronous, per row and last-writer-wins by timestamp, with a lag target of p99 under 10 s. Routing policies decide whether a client gets read-your-writes or failover, and a replica cell finally takes the batch scans off the serving cell. Cross-row transactions come from Percolator, which builds snapshot isolation on top of single-row atomicity and timestamps without changing the tablet server. When per-row consistency inside one cell stops being enough, Megastore and then Spanner add Paxos and TrueTime. Underneath, Colossus replaces GFS. Through all four, the core stays the same: a sorted map, atomic single-row mutations, one server per tablet, and bytes kept in a shared file system.

The interviewer closes the ledger. "It's a year later. Profiles wants to survive losing a datacenter, the indexing team wants a cross-row transaction, and someone asks whether this is just Spanner with the hard parts left out. Where does the design go?"

You: "Each of those is a layer we can add without rebuilding what's under it, and each one's trigger is already in the ledger. I'll take them in the order they'd show up. Replication comes first, then transactions, then the point where we stop layering and change the replication model, and last what changes underneath us."

Cross-Cell Replication: Copy the Mutations, Not the Files

Trigger: a whole cell is a single failure domain. Section 11 got RPO to zero inside a cell, but losing the datacenter loses Profiles. And Section 10's noisy neighbour, the 14 GB/s nightly Webtable scan pushing Profiles past its 10 ms p99, was still waiting for a replica cell (D14).

This happened in the original system too. The Bigtable paper says Personalized Search replicated its data across several clusters, first with a client-side mechanism that ensured eventual consistency and later with a replication subsystem built into the servers. We take the destination directly: each cell tails its own committed mutations and ships them to its peers. Every replicated value keeps its original timestamp, so when two cells disagree, the higher timestamp wins (D10).

Interviewer: "Why not synchronous? Then a lost cell loses nothing."

You: "Because every write would then wait on a wide-area round trip, and Maya's write has a 20 ms p99 inside one datacenter. Async gives a cell-loss RPO of seconds, set by the lag target, and I'd state that on the record. The flip trigger is a tenant that needs strong consistency across cells. At that point I wouldn't bolt sync onto this. That's what Spanner exists for."

The routing policy is where the trade shows up for the client, and Cloud Bigtable exposes the same choice through app profiles (its clusters are our cells). Single-cluster routing sends every request from a profile to one cell. Maya's 08:14 write lands in cell A, her next request reads cell A, and she sees her own search. It has no automatic failover: if cell A is lost, an operator or the application repoints the profile to cell B. Multi-cluster routing sends each request to the nearest available cell. It survives an outage with no human involved, but her next request might hit cell B up to 10 s before the write arrives there. So the Profiles front-end uses single-cluster routing and accepts a manual failover, and the indexer gets its own profile pinned to cell B. The batch problem goes away because batch work now runs in a different datacenter.

Critical Insight: Last-writer-wins is safe for Set and unsafe for increments. If a "searches today" counter is incremented in both cells at once, each cell runs ReadModifyWrite locally and replicates the result. The higher timestamp wins and one increment disappears. The fix is a policy, not a mechanism: route every ReadModifyWrite and CheckAndMutate for a row through one cell. Cloud Bigtable does this by allowing single-row transactions only in single-cluster app profiles.

Quick check before moving on: