Apache Cassandra — Part 1: Foundations & The Masterless Ring

maang.io System Design Series

What is this?

Picture a co-op of identical corner shops arranged in a big circle — no head office, no manager, no "main" store. Every shop follows the exact same rule for who stocks what: take an item's name, run it through an agreed formula, and that number tells you which shop on the circle owns it. Because every shopkeeper knows the same rule, you can walk into any shop and they'll either serve you or point you straight to the one that can. And every item is quietly copied to the next couple of shops going clockwise, so if one shop burns down, nobody loses their groceries.

Apache Cassandra is that co-op, for data. It's a distributed database with no leader and no single point of failure. Every node is a peer that knows the same placement rule, so any node can accept your read or write. Data is spread across the circle — the ring — by hashing, and copied to a handful of neighbours for safety. Add more shops to the circle and you get more capacity, linearly, without redesigning anything.

The one-line idea: Cassandra trades the single-leader database's easy strong consistency for a masterless ring that's always writable and scales linearly — you add capacity by adding plain, interchangeable nodes, and you dial how much consistency you want per query.

That trade — give up the single boss, gain availability and effortless scale — is the whole personality of Cassandra. Let's unpack why anyone would want it.

1. Why Cassandra exists — the pain before it

The relational database (Postgres, MySQL) is a masterpiece, but it has one leader per shard, and that leader is a bottleneck and a liability. Three problems bite hard once you're big:

Writes don't scale sideways. A single leader serializes writes. You can add read replicas all day, but write throughput is capped by one machine. Sharding by hand helps, but now you own a painful, manual resharding problem.
The leader is a single point of failure. When it dies, writes stop until a new leader is elected. There's a window of "sorry, we're down."
Partitions force an ugly choice. When the network splits (and at scale it will — recall the CAP theorem from the 101 course), a strongly-consistent leader-based system has to refuse writes on the minority side to stay correct. For a shopping cart or a "like" button, refusing the write is often the wrong business call.

Cassandra was born (at Facebook, for inbox search) from two famous papers stitched together: Amazon's Dynamo (masterless replication, consistent hashing, "always writable") and Google's Bigtable (the wide-column data model and the log-structured storage engine). That heritage is the fastest way to remember what Cassandra is:

💡 Cassandra ≈ Dynamo's distribution + Bigtable's storage. Dynamo says how data is spread and kept available; Bigtable says how each node stores and lays out data on disk. Almost every design choice traces back to one of those two.

The result is a database that leans hard toward Availability and Partition-tolerance in CAP terms: it stays up and writable even when nodes and networks misbehave, and it lets you decide, per query, how much consistency to trade back in.

2. The core mental model: the ring

Everything starts with the ring. Imagine the entire space of possible hash values — a huge range of numbers — bent into a circle. Each node is assigned positions on that circle called tokens. A node owns the slice of the ring from the previous token up to its own.

To place a row, Cassandra takes its partition key, hashes it (the default hash is Murmur3), and lands on a point on the ring. Walk clockwise to the first node — that node is the primary owner of the row. Simple, deterministic, and every node can compute it without asking anyone.

Chalkboard flow of the Cassandra ring mental model: hash the partition key and walk clockwise to the owning node

Why a ring instead of a plain hash-to-server map? Because a plain hash % N map reshuffles almost everything when N changes (add one server and nearly every key moves). This is consistent hashing, the same idea behind the caching and load-balancer chapters: when a node joins or leaves, only the keys in its immediate arc move, not the whole dataset. Adding a node is a local event, not a global reshuffle.

2.1 Virtual nodes (vnodes) — smoothing the ring

Early Cassandra gave each node one token, which caused two headaches: lumpy load (some nodes owned bigger arcs) and slow, uneven rebuilds when a node died. Modern Cassandra gives each physical node many small tokens — virtual nodes, or vnodes (configured by num_tokens). Instead of owning one fat arc, a node owns dozens of tiny arcs scattered around the ring.

Two payoffs: load evens out automatically, and when a node dies, its little arcs are rebuilt in parallel from many other nodes at once, instead of streaming from one overloaded neighbour. This is the co-op deciding that each shop looks after lots of small aisles spread across the circle, not one big contiguous section.

3. Replication: copies for safety

Owning a slice isn't enough — you need copies. The replication factor (RF) says how many nodes hold each row. RF = 3 means every row lives on three nodes: the primary owner plus the next distinct nodes walking clockwise.

Chalkboard replication chain placing copies on the primary and next clockwise nodes

The rule that picks which nodes is the replication strategy. In any real deployment you use NetworkTopologyStrategy, which is rack- and datacenter-aware: it places the RF replicas on distinct racks (so one rack losing power doesn't take out all your copies) and lets you set a different RF per datacenter — e.g. three copies in us-east and three in eu-west for a globally-distributed, low-latency deployment. How does a node know the rack and datacenter layout? A component called the snitch tells it. (We'll go deeper on placement in Part 2.)

💡 RF is not a consistency setting — it's a durability and availability setting. It says how many copies exist. How many of those copies you talk to on each read or write is a separate dial called the consistency level, and that separation is the single most important thing to understand about Cassandra. It's coming up in Part 3.

4. The vocabulary you'll actually use

Cassandra reuses SQL-ish words but means subtly different things. Pin these down now:

Term	What it means
Keyspace	The top-level container (like a "database" in SQL). Holds the RF and replication strategy.
Table	A collection of rows sharing a schema (Cassandra tables were once called "column families").
Partition key	The part of the primary key that decides which node stores the row. This is the single most important modeling decision.
Clustering key	The part of the primary key that decides *how rows are sorted within* a partition** on disk.
Partition	All the rows sharing one partition key — stored together, on the same nodes, sorted by clustering key. The fundamental unit of storage and access.
Coordinator	Whichever node you connected to for a given request. It doesn't have to own the data — it just orchestrates talking to the replicas that do.
Consistency level (CL)	Per-query dial: how many replicas must respond before the query is considered a success.

The mental leap from SQL is this: in Cassandra, a table is a set of partitions, and a partition is a mini, sorted table keyed by clustering columns. You almost always want to read one partition at a time.

5. How you actually use it — CQL by example

You talk to Cassandra with CQL (Cassandra Query Language), which looks like SQL on purpose but bans the expensive stuff (no joins, no arbitrary WHERE). Here's a full, small example.

First, a keyspace with datacenter-aware replication:

CREATE KEYSPACE chat
  WITH replication = {
    'class': 'NetworkTopologyStrategy',
    'us_east': 3          -- three copies in the us_east datacenter
  };

Now a table for chat messages. Read the primary key definition carefully — it's the whole ballgame:

CREATE TABLE chat.messages (
    conversation_id  uuid,        -- partition key: groups a conversation together
    message_id       timeuuid,    -- clustering key: sorts messages by time
    sender_id        uuid,
    body             text,
    PRIMARY KEY ((conversation_id), message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);   -- newest first

That PRIMARY KEY ((conversation_id), message_id) says: store all messages for one conversation together on the same nodes (partition key conversation_id), sorted by message_id descending (clustering key). Now the natural query is blazing fast because it hits exactly one partition, already sorted:

-- Fetch the 50 newest messages in a conversation — one partition, pre-sorted, one seek.
SELECT sender_id, body
FROM chat.messages
WHERE conversation_id = 8a7f...        -- partition key is REQUIRED
LIMIT 50;

Here's the discipline that trips up newcomers. This query is illegal by default:

-- 🚫 Cassandra refuses this: no partition key, so it would scan the whole ring.
SELECT * FROM chat.messages WHERE sender_id = 3c2d...;

Cassandra won't let you run a query that would have to visit every node, because that doesn't scale. Which leads to the golden rule of using Cassandra well:

💡 Model your tables around your queries, not around your data. In SQL you normalize once and query freely. In Cassandra you decide the queries first, then build a table (often a denormalized copy) for each one. Want messages by conversation and messages by sender? That's two tables, each written on every message. Storage is cheap; scans are not.

We'll turn this rule into a full worked data model in Part 4.

6. When to reach for Cassandra — and when not

Cassandra is a specialist. It is spectacular at exactly the shape it was built for, and genuinely bad outside it.

Reach for it when you have:

Huge write volume — telemetry, IoT, event logs, activity feeds, messaging. Cassandra's write path (Part 2) is extraordinarily fast.
Always-on availability across regions — no maintenance windows, survive a whole datacenter, multi-region active-active.
Predictable, key-based access — you know how you'll look data up ("give me this user's feed", "this conversation's messages").
Linear scale — you want to double capacity by roughly doubling nodes, with no re-architecture.

Reach for something else when you need:

Ad-hoc queries, joins, or aggregations — Cassandra has none of these. That's an analytics database's job (or Postgres).
Strong transactions across many rows — Cassandra offers only limited single-partition compare-and-set (lightweight transactions, Part 3), not general multi-row ACID.
Strong read-after-write by default — you can get it (tune the consistency levels), but it costs latency; if every read must be perfectly fresh cheaply, a leader-based store fits better.
Small data — under, say, a few hundred GB on one box, a single Postgres instance is simpler, cheaper, and more flexible. Cassandra's power only pays off at scale.

A quick lay of the land: DynamoDB is the managed, Dynamo-lineage cousin (same distribution ideas, AWS runs it for you). HBase is the Bigtable-lineage cousin (strong consistency, but a master and a HDFS dependency). ScyllaDB is a C++ reimplementation of Cassandra chasing lower latency. Knowing where Cassandra sits among these is a senior-level talking point we'll sharpen in Part 4.

7. Key takeaways

Cassandra is a masterless, peer-to-peer ring — every node is an equal peer, so there's no single point of failure and any node can serve any request. It marries Dynamo's distribution to Bigtable's storage.
Data is placed by consistent hashing on the ring: hash the partition key → walk clockwise → that's the owner. Vnodes spread each node's ownership into many small arcs for even load and fast rebuilds.
The replication factor (RF) sets how many copies exist; NetworkTopologyStrategy places them across racks and datacenters. RF is about durability — a separate dial (consistency level) controls how many copies you talk to.
A partition (all rows with the same partition key, sorted by clustering key) is the atom of storage and access. Query one partition at a time.
Model around your queries, not your data — denormalize, one table per access pattern. CQL looks like SQL but forbids joins and any query that would scan the ring.

Next up, Part 2 cracks open a single node: the write path (commit log → memtable → SSTable), the LSM-tree storage engine that makes writes so cheap, and how compaction keeps it all tidy.

Intermediate