Queues & message brokers

Picture a request that adds an expense to a group. The obvious implementation does everything inline: write the expense, recompute balances, send a push notification to each of the other members, maybe update a search index. But the user is waiting for all of that, and worse, the request's success is now hostage to the slowest and flakiest part — if the push-notification service is down, does adding an expense fail? It shouldn't. The expense is saved; the push is a consequence that can happen slightly later.

That instinct — "this work is real, but it doesn't have to happen now, inside this request" — is what a queue is for. The producer (your request handler) drops a message onto the queue and returns immediately; a consumer (a worker) picks it up and does the work afterward. The thing in the middle that stores and routes messages is a message broker. This is asynchronous, decoupled work, and it's one of the most important building blocks you have.

What a broker actually buys you

Four distinct wins, and naming them separately tells you when you need one:

• Responsiveness. The request returns as soon as the message is enqueued, not when the work finishes. The user isn't waiting on the push, the image resize, or the third-party call.
• Decoupling in time. The producer and consumer don't have to be up at the same instant. If the worker is down or slow, messages wait in the queue; when it recovers, it catches up. The broker is a buffer between two things that would otherwise have to be simultaneously healthy.
• Load-leveling. A traffic spike that would overwhelm a synchronous system instead just makes the queue longer. Producers enqueue at the spike rate; consumers drain at their own steady rate. The queue absorbs the burst — the same backpressure idea from the sockets lesson, applied to work instead of bytes.
• Work distribution. Many workers can pull from one queue, each taking different messages. That's horizontal scaling of processing — need more throughput, add more consumers — with no coordination beyond the broker handing each message to one worker.

Work queue vs pub/sub: one message, one consumer — or all of them

There are two fundamentally different delivery shapes, and mixing them up causes real bugs:

• Work queue (competing consumers). Each message is delivered to exactly one consumer out of the pool. Ten workers on a queue means the messages are divided among them — this is the work-distribution case. Use it for tasks: "resize this image" should run once, not ten times.
• Publish/subscribe (fan-out). Each message is delivered to every subscriber. Publishing "expense created" to a topic with three subscribers means all three get it — one updates balances, one sends pushes, one updates search. Use it for events that multiple independent systems care about.

The distinction is "who needs to see this?" — one worker to do a job, or every interested party to react.

Publish a few messages under each shape. A work queue spreads them across the consumers (one copy each — total deliveries = messages); Pub/Sub hands every consumer its own copy (deliveries = messages × subscribers). The "copies per message" meter is the whole difference.

(Kafka blurs the line usefully: a topic is a fan-out log, but a consumer group splits its partitions across group members, giving you competing-consumers behavior within a group and fan-out across groups.)

The brokers, briefly

You don't need to memorize these, but the shape of the trade-offs is worth carrying:

• RabbitMQ — a traditional "smart broker" — rich routing (exchanges, bindings), per-message acknowledgements, and the broker tracks what's been consumed. Messages are typically gone once acked. Great when you want the broker to do the routing thinking.
• Kafka — a distributed, append-only log — a "dumb broker, smart consumer." Messages are retained (for a time window or size) even after being read, so consumers track their own position (offset) and can replay history. Ordered within a partition, enormous throughput. Great for event streams, CDC, and anything you want to re-read.
• SQS (and cloud equivalents) — a managed queue, minimal to operate, at-least-once with simple semantics. Great when you don't want to run a broker at all.
• Redis (lists / streams) — lightweight, already in your stack if you cache with it; fine for modest queues without adding infrastructure, though it's less durable and feature-rich than a dedicated broker.

Delivery guarantees: the part everyone gets wrong

Here is the crux of queueing, and the source of its subtlest bugs. When a broker hands a message to a consumer and the consumer crashes mid-processing — did it finish? The broker can't tell. Its choice of what to do defines the delivery guarantee:

• At-most-once — send it, don't wait for confirmation. If the consumer dies, the message is lost. Never duplicated, sometimes dropped. Acceptable only when losing a message is fine (a non-critical metric ping).
• At-least-once — keep the message until the consumer acknowledges success; if no ack arrives (crash, timeout), redeliver it. Nothing is lost — but a consumer that finished the work and crashed before acking will get the message again, so you can get duplicates. This is the default of most real systems, because losing messages is usually worse than occasionally repeating one.
• Exactly-once — the holy grail, and mostly a myth as usually imagined. Across independent systems (broker + your database + a third-party API), there is no general way to guarantee a message is processed once and only once — the classic proof is that "do the work" and "record that you did it" can't be made atomic across two systems. (This connects to the delivery-guarantee myths in the distributed-patterns stage.)

So what do you actually do? You accept at-least-once delivery and make your consumers idempotent. An idempotent consumer produces the same result whether it processes a message once or five times — because it checks "have I already handled this message id?" before acting, or its action is naturally repeat-safe. At-least-once delivery + idempotent consumers = exactly-once effects, which is what people actually want when they say "exactly-once." This is the single most important sentence in the lesson: you don't prevent duplicate delivery, you make duplicate delivery harmless.

Ordering and backpressure

Two more realities. Ordering: most queues do not guarantee global order — messages can be processed out of sequence, especially with multiple consumers. If order matters (apply these balance changes in sequence), you need a broker that preserves it per key (Kafka's per-partition ordering) and you must route related messages to the same partition. Don't assume order you didn't explicitly arrange for.

Backpressure: the queue absorbs spikes, but if consumers are persistently slower than producers, the backlog grows without bound — and an unbounded backlog means unbounded latency (a message enqueued now won't be processed for hours). The queue depth is a vital sign: monitor it, scale consumers when it climbs, and if it can't be drained, shed load at the producer rather than letting the backlog grow forever. A queue is a shock absorber, not infinite storage.

Go deeper

• Confluent — "Exactly-once semantics are possible: here's how Kafka does it" — The best explanation of why exactly-once is subtle and what it actually requires (idempotent producers + transactional writes) — read it right after the delivery-guarantees section to see the myth handled rigorously.
• AWS Builders' Library — "Avoiding insurmountable queue backlogs" — A production treatment of backpressure, poison messages, and DLQs from a team that runs queues at massive scale — the failure-mode callout in operational depth.
• Enterprise Integration Patterns — messaging patterns catalog — The canonical vocabulary (competing consumers, dead letter channel, publish-subscribe, message ordering) — the reference that names every shape in this lesson precisely.

Check yourself

Answer out loud, as if an interviewer asked. If you hand-wave, reread that section.

1. Adding an expense should not fail just because the push-notification service is down. Explain, in terms of coupling and responsiveness, why moving the push onto a queue fixes that, and name the four benefits a broker provides.
2. Distinguish a work queue from publish/subscribe by the question "who needs to see this message?" Give a task that must use one and an event that must use the other, and explain what a Kafka consumer group does across the two.
3. A consumer records a payment, then crashes before acknowledging. Under at-least-once delivery, what happens next, and why is the resulting double-payment a consumer bug rather than a broker bug?
4. Explain why exactly-once delivery is mostly a myth, and what "at-least-once + idempotent consumers = exactly-once effects" means. Why is that the real-world answer?
5. Define a poison message and describe how it can stall an entire ordered queue. What does a dead-letter queue with a max-retry policy change?
6. A queue is a shock absorber, not infinite storage. Explain what unbounded backlog does to latency, why queue depth is a vital sign, and what you do when consumers simply cannot keep up.