Chapter 10: Storage Scaling – Evolving from Hot to Cold

Storage is the vast archive of conversations, a place where petabytes of data live, accessed, and aged gracefully from the hot fervor of recent chats to the cold slumber of archives. In this chapter, we scale the WhatsApp-like messaging system's storage layer for 100M+ daily active users, handling billions of messages, trillions of historical entries, and petabytes of media. We'll design independent subsystems— NoSQL for sequenced messages, Redis for ephemeral presence, object storage for media—with partitioning strategies, tiering from hot to cold, and capacity planning to ensure reliability, low latency, and cost efficiency as the system evolves.

2. Storage Components Overview

Each is scaled independently based on access patterns.

3. Message Storage Scaling

Messages arrive at a rate of 5 billion per day, generated by 100 million daily active users each sending an average of 50 messages, with peak throughput reaching 175,000 writes per second per region. We select Cassandra or DynamoDB as our storage engine, chosen for their horizontal scaling capabilities, partitioned architecture that distributes load, fast write performance, tunable consistency levels, and high availability guarantees.

The table schema uses conversation_id as the partition key and message_sequence as the sort key, optimizing for queries like fetching the last N messages, fetching messages from a sequence greater than X, and handling sequential writes efficiently.

Partition Key:   conversation_id
Sort Key:        message_sequence

Optimized for:

• "Fetch last N messages"
• "Fetch from sequence > X"
• Sequential writes

3.1 Partitioning Strategy

Goal: Avoid hot partitions.

Large groups or high-volume conversations can overload a single partition.

Solutions:

1. Prefix hashing (advanced)

   partition_key = hash(conversation_id) % 32
   

2. Limit group size (WhatsApp constraint = ≤256 members)
3. Tiered fanout & sequencer sharding

WhatsApp solves this problem operationally by limiting group size rather than complicating partitioning.