Design WhatsApp

WhatsApp serves over 2 billion users, delivering 100 billion messages per day. Its defining properties — real-time delivery, end-to-end encryption, offline message queuing, and lightweight client design — make it a canonical system design problem. Building it touches WebSocket management, message ordering, delivery guarantees, distributed storage, and the specific challenges of group messaging at scale.

Requirements

Functional:

• One-to-one messaging with real-time delivery.
• Group messaging (up to 1,024 members).
• Message delivery status: sent, delivered, read (single and double blue tick).
• Media sharing: photos, videos, audio, documents.
• Online presence and “last seen” indicators.
• Push notifications for offline users.
• End-to-end encryption: the server cannot read message content.

Non-functional:

• Low latency: messages delivered within seconds on good networks.
• High availability: 99.99% uptime.
• Durability: messages must not be lost even if the server crashes mid-delivery.
• Scale: 2 billion users, 100 billion messages/day.

Capacity Estimation

Daily active users: 500M
Messages per day: 100B
Messages per second: ~1.15M msg/s
Average message size: 100 bytes (text only)
Text message throughput: ~115 MB/s

Media messages: ~20% of total = 20B/day
Average media size: 500 KB
Media storage per day: 20B × 500KB = ~10 PB/day (compressed + deduplicated: ~1 PB/day)

WebSocket connections: 500M concurrent connections at peak
Chat servers needed: at 100K connections per server = 5,000 servers

Messaging Architecture

The core of a messaging system is the connection layer — maintaining persistent connections with clients and routing messages between them.

WebSocket gateway (chat servers): Each client maintains a persistent WebSocket connection to a chat server. A client sends a message by writing to its WebSocket; the server routes it to the recipient’s chat server; that server delivers to the recipient’s WebSocket.

Connection routing: With 5,000 chat servers, how does server A know which server holds the recipient’s WebSocket? A service registry (Redis or ZooKeeper) maps user ID → chat server. On connect, the client’s server registers the mapping; on disconnect, it deregisters.

Message flow:

1. Sender writes message to their WebSocket connection (chat server A).
2. Chat server A persists the message to the message store with status sent.
3. Chat server A looks up the recipient’s server (chat server B) from the registry.
4. Chat server A sends the message to chat server B via an internal RPC or message queue.
5. Chat server B delivers the message to the recipient’s WebSocket and updates status to delivered.
6. Chat server B sends a delivery acknowledgment back to the sender’s server, which notifies the sender’s client (single tick → double tick).

Delivery Guarantees

WhatsApp guarantees at-least-once delivery — a message is stored on the server until the recipient acknowledges receipt. The challenge is handling offline recipients.

Offline message queuing: If the recipient is offline (no WebSocket connection), the message is stored in a per-user message queue (backed by a database like Cassandra). When the user reconnects, the server delivers all queued messages and waits for client acknowledgment before deleting them.

Message IDs and ordering: Each message gets a globally unique ID (Snowflake-style: timestamp + server + sequence). Within a conversation, messages are ordered by client timestamp (with server timestamp as tiebreaker). Clients display messages in order; out-of-order delivery is resolved client-side.

Push notifications: For users who haven’t opened the app in a while, their device OS has killed the WebSocket connection. A push notification (APNs for iOS, FCM for Android) wakes the app, which reconnects and pulls queued messages. Push notifications carry a notification text but not the message content (which is end-to-end encrypted and can’t be read by the server).

Group Chat

Group chats are fundamentally harder than 1:1 chats because one message must be delivered to N recipients.

Fan-out on write: When a message is sent to a group, the server creates N copies of the message (one per recipient) and places each in the recipient’s message queue. Delivery proceeds as normal per-user delivery. Simple and predictable, but expensive for large groups — a message in a 1,024-member group creates 1,024 writes.

Fan-out on read: Store the message once in a group message log. Each recipient tracks a cursor (last-read message ID). On reconnect, they pull new messages since their cursor. More storage-efficient but requires a more complex read path and can’t leverage per-user push efficiently.

WhatsApp uses fan-out on write for smaller groups (lower latency, simpler delivery tracking) and fan-out on read / hybrid approaches for very large groups. Group message storage uses a Cassandra-style model with (group_id, message_id) as the partition + clustering key — efficient range scans for message history.

Delivery receipts in groups: A message is “delivered” when all members have received it and “read” when any member has read it. Tracking per-member read status for 1,024 members per message is expensive. WhatsApp maintains a per-message delivery bitmap — one bit per group member, set when delivered and again when read.

Media Storage

Media (photos, videos, documents) is not sent through the chat server WebSocket — it’s too large and would clog the message pipe.

Upload flow:

1. Client requests an upload URL from the media service.
2. Client uploads the file directly to object storage (S3-equivalent) via the pre-signed URL.
3. Client sends a text message with the media URL and metadata (MIME type, dimensions, duration) to the chat server.
4. Recipient downloads the media from object storage when they open the message.

Deduplication: The same photo sent to multiple recipients is stored once. Content-addressed storage (SHA-256 hash of the file → storage key) deduplicates automatically. If the hash already exists in storage, the upload is skipped — only the metadata message is sent.

Media expiry: WhatsApp stores media for 30 days on its servers. After the recipient downloads it, the client stores it locally. This limits storage costs — media that’s never opened (group chat media, spam) doesn’t accumulate indefinitely.

Presence & Read Receipts

Presence (“Online” / “Last seen 5 minutes ago”) is a high-write, high-read feature that must be carefully isolated from the message path to avoid it becoming a bottleneck.

When a client connects or sends a heartbeat, its server updates a Redis key: presence:{user_id} = {timestamp, server_id} with a short TTL (60 seconds). Heartbeats refresh the TTL. If the TTL expires, the user is considered offline.

Presence data is fanned out to contacts who are online. Rather than pushing presence updates to all contacts immediately (expensive at scale), WhatsApp uses a subscription model: a client subscribes to presence updates for its contacts. The presence service publishes updates to subscribers. At 500M DAU, full presence fan-out would require billions of writes per second — WhatsApp limits presence visibility and updates to mitigate this.

End-to-End Encryption

WhatsApp uses the Signal Protocol for end-to-end encryption. Key properties:

• The server never sees plaintext. Messages are encrypted on the sender’s device and decrypted on the recipient’s device. The server routes ciphertext it cannot read.
• Double Ratchet Algorithm: Provides forward secrecy (past messages can’t be decrypted if current keys are compromised) and break-in recovery (future messages are secure after a key compromise is resolved). Each message uses a fresh key derived from a ratchet chain.
• Key distribution: Each device uploads a batch of one-time pre-keys to the WhatsApp server. When Alice wants to message Bob for the first time, she fetches one of Bob’s pre-keys and initiates the key exchange. The key exchange is asynchronous — Bob doesn’t need to be online.
• Group encryption: Each group member generates a Sender Key and distributes it (encrypted individually) to all group members. Group messages are encrypted once with the Sender Key (not N times), making group encryption O(1) per message after key distribution.

Scaling Considerations

• Horizontal scaling of chat servers. Chat servers are stateful (they hold WebSocket connections), so scaling them requires connection-aware load balancing. A client must connect to the same server for the duration of its session. Consistent hashing on user ID distributes users across servers; when a server is added or removed, only a fraction of connections migrate.
• Cassandra for message storage. Messages are append-heavy and need efficient range scans by conversation (partition key = conversation ID, clustering key = message ID). Cassandra’s wide-row model maps perfectly. Replication factor of 3 across data centers provides durability.
• Message queue TTL. Undelivered messages queued for offline users have a TTL (30 days on WhatsApp). Messages older than TTL are dropped — the sender is notified that the message was not delivered. This bounds storage for inactive users.
• Zookeeper / etcd for chat server coordination. Chat servers register themselves and their connection counts. The load balancer directs new connections to the server with available capacity. On server failure, its connections migrate to other servers (clients reconnect automatically via exponential backoff).