Design a chat system

"Design WhatsApp / Slack / Messenger" is the prompt that makes you confront stateful connections at scale — everything else in this stage has been request/response, and chat is not. It pulls together the WebSockets, sockets/C10K, and load-balancing lessons into one design, and it happens to be the one system in this stage that Fable actually ships, so you can check the theory against a real build.

Requirements

Functional: 1:1 and group messaging; messages delivered in near-real-time; message history; online/presence indicators; delivery/read receipts. Defer: calls, media specifics, encryption details.

Non-functional: low-latency delivery (messages feel instant), massive concurrency (millions of simultaneously-connected clients — the C10K/C10M problem), reliable delivery (a message must never be silently lost, even across drops and reconnects), and ordering within a conversation (messages appear in a sensible, consistent order). The concurrency and reliability requirements are what make this hard.

Estimates

The distinctive number here isn't requests/sec — it's concurrent connections. Say 10M users online at once: that's 10M open, mostly-idle WebSocket connections that must be held, which is a memory-and-file-descriptor problem, not a CPU one (the sockets lesson's C10K, grown up). Each connection server holds some slice — if one server holds ~65k connections, 10M users need ~150+ connection servers. Messages/sec is comparatively modest; it's the held connections that size the fleet.

High-level design

• Connection layer — a fleet of WebSocket gateway servers, each holding a slice of the live connections in memory. This layer is stateful — a connection lives on one specific server.
• Message store — a database that persists every message (the source of truth).
• A backplane — pub/sub (Redis/Kafka) or a routing service so a message can get from the sender's gateway to the recipient's gateway.
• Presence service — tracks who's online.
• Flow: client holds a WebSocket to a gateway → sends a message → gateway persists it, then routes it to the recipient's gateway → that gateway pushes it down the recipient's socket.

Deep-dive 1: routing across a stateful fleet

Here's the problem request/response never has. Alice is connected to gateway 7; Bob to gateway 34. Alice sends Bob a message. Gateway 7 has Bob's message but not Bob's connection — that's on gateway 34. How does it get there? Two approaches:

• Pub/sub backplane — every gateway subscribes to a channel (per-user or per-conversation) on a shared bus (Redis/Kafka); the sender's gateway publishes, and the bus delivers to whichever gateway holds the recipient. Simple, decoupled — this is exactly Socket.IO's Redis adapter.
• Routing service / connection registry — a service that maps user → which gateway holds them, so the sender's gateway looks up Bob's gateway and forwards directly. More targeted at very large scale, more moving parts.

Either way, the core fact is the load-balancing lesson's sticky-session reality: a connection is state pinned to one server, so the LB must keep each client on its gateway, and reaching a user means reaching their gateway. This is the defining structural difference from stateless REST.

Deep-dive 2: the socket is delivery, the database is truth

The reliability requirement forces the single most important decision, and it's the one Fable's chat war story is built on: persist the message before you deliver it. Write the message as a durable row first, then push it over the socket. Because:

• A successful emit is not a delivered message — the recipient may have just dropped (a tunnel, a sleep, a reconnect), and the push evaporates. If the socket were the source of truth, that message is gone.
• With the message durably stored first, a lost push costs only latency: the recipient, on reconnect, fetches everything since their last-seen message via a cursor-paginated history query and catches up. The socket's job shrinks to "notify promptly"; the database is what's always caught up to.

So delivery is at-least-once (push, and also reconcilable via history), and messages carry a client-generated id so a resend after a flaky-network drop dedupes instead of double-posting — exactly-once effects, again. Delivery/read receipts are their own small state machine (sent → delivered → read) layered on top.

Deep-dive 3: ordering and presence

• Ordering within a conversation is provided by a server-assigned sequence (or a sortable id) so all participants render messages in the same order regardless of network jitter — you don't trust client clocks. Global ordering across all conversations is neither needed nor affordable; per-conversation is what matters.
• Presence (online/last-seen) is surprisingly expensive at scale — naively, every status change fans out to everyone who might care. Real systems make it approximate (heartbeats with a timeout, batched/debounced updates, "last seen within a few minutes" rather than to-the-second) because exact presence isn't worth the fan-out.

Go deeper

• WhatsApp system design walkthrough — A full treatment of the connection fleet, message routing, delivery receipts, and presence at billions-of-users scale — the large-scale version of Fable's architecture.
• Real-time messaging & pub/sub backplanes — Background on why stateful connection layers need a pub/sub backplane and how message routing across gateways actually works — deep-dive 1 in depth.
• Ably — WebSockets, delivery guarantees & reconnection — A clear treatment of why realtime transports are best-effort and why durable storage + reconnection reconciliation is the correct pattern — the persist-then-deliver spine, generalized.

Check yourself

Answer out loud, as if an interviewer asked. If you hand-wave, reread that section.

1. What is the distinctive scaling number for a chat system (vs a request/response service), and why does it size the fleet as a memory problem rather than a CPU one?
2. Alice on gateway 7 messages Bob on gateway 34. Explain why gateway 7 cannot deliver directly, and the two ways (pub/sub backplane vs routing service) to get the message to Bob.
3. Why must a message be persisted before it is delivered over the socket? Explain what an emit-only design loses, and how reconnect reconciliation makes a lost push cost only latency.
4. How is message ordering within a conversation achieved without trusting client clocks, and why is exact presence made approximate at scale?
5. Describe the reconnect thundering herd unique to stateful connection fleets, and the defenses (backoff+jitter, spare capacity, bounded history queries) borrowed from the retry-storm playbook.
6. Fable ships this design at small scale. Map each of the three deep-dives to a concrete Fable mechanism, and name the scale-only pieces Fable does not yet need but is structured to add.