Chapter 8: Media Pipeline – Handling Rich Content at Scale

Media elevates conversations from text to vibrant stories, but it brings challenges of bandwidth, storage, and security that could overwhelm traditional chat systems. Envision a pipeline that elegantly offloads heavy files to object storage, then delivers them globally via CDN—like a digital warehouse feeding the world instantly. In this chapter, we explore how a WhatsApp-like backend manages images, videos, audio, and documents outside the core chat flow, ensuring high performance, cost efficiency, and compatibility with end-to-end encryption. We'll trace the journey from client upload requests to global delivery, handling billions of media objects while keeping chat servers lightweight and responsive.

2. High-Level Media Flow

The media journey unfolds in a streamlined sequence that offloads heavy lifting from chat servers. It begins when the client requests a pre-signed upload URL from the backend, then uploads content directly to S3, GCS, or Azure Blob. The backend stores metadata and returns a media_id. The client sends a chat message referencing the media_id. Recipients fetch media from CDN via signed URLs. Optionally, asynchronous scanning for antivirus, NSFW, and abuse detection occurs. Multiple quality tiers are supported, from thumbnails to previews to full resolution. This approach keeps chat servers lightweight and ensures global scalability.

1. Client requests pre-signed upload URL from the backend
2. Client uploads content directly to S3/GCS/Azure Blob
3. Backend stores metadata and returns media_id
4. Client sends chat message referencing the media_id
5. Recipients fetch media from CDN via signed URLs
6. Optional asynchronous antivirus, NSFW, and abuse scanning
7. Multiple quality tiers (thumbnail → preview → full resolution)

This keeps chat servers lightweight and ensures global scalability.

3. Upload Pipeline (Step-by-Step)

3.1 Client Requests Upload Credentials

Request:

POST /media/upload-request
{
  "mime_type": "image/jpeg",
  "size_bytes": 523000
}

Backend validates:

• Allowed MIME type
• Size limits
• Rate limits the sender

Generates:

• media_id
• Pre-signed URL for object storage upload
• CDN URL template for downloads

Example response:

{
  "media_id": "m-12345abcd",
  "upload_url": "...signed_S3_url...",
  "cdn_url": "https://cdn.example.com/m-12345abcd"
}