Chapter 9: Multi-Region Architecture – Global Scalability

Messaging should feel instantaneous across the globe, but the Earth is large, and regions falter. To deliver WhatsApp-scale reliability, we deploy an active-active multi-region architecture that routes users locally, anchors conversations to home regions for ordering, and fails over gracefully like a city's seismic resilience. In this chapter, we explore how to design a system supporting billions of daily messages with strict ordering, zero data loss, and seamless disaster recovery. We'll navigate user routing, cross-region fanout, and failover strategies, ensuring users never feel the distance or the outages.

---

2. Multi-Region Architecture Goals

A production-grade global messaging system must achieve low latency by routing users to the nearest healthy region, high availability by preventing region outages from halting messaging, conversation ordering by preserving strict per-conversation sequences, geo-affinity by keeping traffic and storage local to minimize costs, scalable fanout by having local workers deliver to local users, efficient replication through asynchronous methods for data durability, and disaster recovery with fast failover, near-zero RPO, and RTO under minutes. These goals guide the entire global system design.

---

3. Regional Architecture Model

We assume three example regions:

• US-East
• EU-West
• APAC-East

Each region contains:

• WebSocket clusters
• Presence store (Redis)
• Sequencer service
• Message Store (Cassandra/DynamoDB)
• Ack Store
• Kafka cluster
• Media upload endpoints
• Push Notification gateway

Each region operates independently for its own local users.

---

4. User Routing & Region Affinity

Users are routed to the nearest region based on:

• IP geolocation
• Mobile carrier hints
• Device locale
• Historical nearest healthy region

This ensures:

• Lower latency
• Localized presence tracking
• Minimal cross-region network cost

A user stays in the same region unless:

• Traveling
• Region becomes unhealthy
• Automatic rebalancing occurs

---

5. Conversation Home Region Model

To maintain strict ordering:

Every conversation is anchored to a single home region.

conversation_id hash → region

All conversation operations occur in that home region:

• Sequencing
• Message persistence
• Fanout
• Ack processing

Why?

Sequencing across regions would require:

• Cross-region locks
• High-latency coordination
• Complex failure handling

By assigning each conversation a single ordering authority, we avoid all distributed sequencing issues.

---

6. Multi-Region Message Flow

Example: User in APAC sends message to a conversation whose home region is EU-West.

Steps:

1. Client → APAC WebSocket
2. APAC forwards message to EU-West sequencer
3. Sequencer assigns message_sequence
4. Message written to EU-West Message Store
5. EU-West Kafka publishes event
6. Fanout Worker delivers:
   • Local EU-West recipients
   • Remote recipients via inter-region routing

Cross-region fanout uses:

• Internal message bus
• gRPC service mesh
• Kafka MirrorMaker / Pulsar Geo-Replication

---

7. Cross-Region Replication

The message stores replicate data asynchronously:

• Cassandra multi-region replication
• DynamoDB global tables
• Event-stream replication

Purposes:

• Disaster recovery
• Global search (if implemented)
• Multi-device user syncing after travel
• Region failover safety

Replication is not used for real-time delivery—only durability and cross-region availability.

---

8. Presence Across Regions

Presence is stored only within a region, not replicated.

Why?

• Presence heartbeats are extremely high volume (millions/sec)
• Region-local presence is sufficient for fanout correctness
• WebSocket routing table in fanout pipeline is authoritative

If user is connected in APAC but conversation is in EU:

• EU fanout worker pushes through APAC routing service
• Presence lookup is region-local

No global synchronization needed.

---

9. Read Receipts Across Regions

Read receipts originate in the region where the user is connected.

Flow:

1. APAC receives read receipt for conversation in EU
2. APAC forwards read receipt to EU
3. EU updates Ack Store
4. EU emits read receipt events to other members

This is identical to message send flow but in reverse direction.

---

10. Media Distribution

Media uses object storage:

• Multi-region S3 buckets
• GCS multi-region storage

Delivered through CDN:

https://cdn.example.com/media_id

Benefits:

• Transparent global caching
• No cross-region load on messaging infra
• Low latency media playback/download

---

11. Region Failure / Failover Model

What happens if a home region fails?

Example: EU-West goes offline.

We must:

• Promote a backup region
• Preserve message ordering
• Avoid losing messages

Two proven approaches:

---

11.1 Hot Standby with Sequencer Promotion (preferred / WhatsApp-like)

Each region maintains:

• Replicated Message Store
• Replicated Ack Store
• Replicated Sequencer State

If EU fails:

1. US-East sequencer for affected conversation becomes active
2. Routing layer remaps conversation → US-East
3. Users reconnect to new region
4. Fanout resumes normally

Failover time: seconds to minutes

RPO: Near-zero (writes replicated asynchronously but quickly)

---