Event-driven architecture & the outbox pattern

As a system grows past one service, the components have to tell each other things: an expense was added, a user signed up, a payment confirmed. There are two ways to say it, and the difference shapes everything. A command is "do this" — a directed request to a specific service (POST /send-email). An event is "this happened" — a statement of fact the emitter broadcasts without knowing or caring who listens (ExpenseCreated). Event-driven architecture is building around the second: services emit events about what they did, and other services react, with no direct coupling between them.

The payoff is decoupling in the strongest sense. The service that creates an expense doesn't call the notification service, the search-indexer, and the analytics pipeline — it doesn't even know they exist. It emits ExpenseCreated and moves on; whoever cares subscribes. You can add a fifth consumer later without touching the emitter. It's the publish/subscribe shape from the queues lesson elevated to an architectural principle: the event log is the integration point, and services are independently deployable reactors around it. But event-driven systems have one problem at their very core that you must solve or everything built on top is quietly broken.

The dual-write problem

Here it is, and it's deceptively simple. When a service handles "add an expense," it must do two things: write the expense row to its database, and publish an ExpenseCreated event so everyone else can react. Two writes, to two different systems (a database and a message broker), and — as the distributed-transactions discussion established — you cannot make two writes to two systems atomic without an expensive, fragile distributed-commit protocol. So one of them can fail while the other succeeds, and both orderings are broken:

• Write DB first, then publish — the row commits, then the broker publish fails (broker down, network blip, process crash in between). The expense exists but no event was emitted: the notification never sends, the search index never updates, the analytics never counts it. A lost event, and the system is now silently inconsistent — the expense is real to the database and invisible to everyone downstream.
• Publish first, then write DB — the event goes out, then the database commit fails and rolls back. Now there's an event for an expense that doesn't exist: consumers send "new expense" notifications for a phantom, the search index points at nothing. A ghost event, arguably worse because the lie propagates.

There is no ordering of two independent writes that's safe, because the failure can always land between them. This is the dual-write problem, and it's the central hazard of event-driven systems. Naive event-driven code — await db.save(); await broker.publish(); — has this bug, always; it just hides until the broker hiccups at the wrong millisecond.

See both patterns fail and heal. In Naive mode, turn the broker DOWN and add an expense — the DB commits but the event is lost forever. Switch to Outbox: the event lands in the DB in the same transaction, and when you bring the broker back UP the relay drains the backlog with nothing lost.

The outbox pattern: one atomic write, then relay

The fix is elegant once you see it: don't do two writes. Do one, atomically, to the database — and derive the publish from it. Instead of publishing to the broker directly, the service writes the event into an outbox table in the same database transaction as the business change:

BEGIN;
  INSERT INTO expenses (...);              -- the business write
  INSERT INTO outbox (event, payload, ...); -- the event, same transaction
COMMIT;                                     -- both, or neither

Now the expense row and its ExpenseCreated event commit together or not at all — it's a single local ACID transaction, no cross-system atomicity needed. The dual-write problem is gone because there is no longer a dual write: there's one transactional write to one database.

Then a separate relay (or "message relay" / "dispatcher") reads unpublished rows from the outbox and publishes them to the broker, marking each as sent once the broker acknowledges. Two ways to run the relay:

• Polling — the relay periodically SELECTs unpublished outbox rows and publishes them. Simple, works anywhere.
• Change data capture (CDC) — the relay tails the database's replication log / WAL (via a tool like Debezium) and publishes new outbox rows as they're committed. No polling lag, no query load, but more infrastructure. This is logical replication put to work.

Crucially, the relay guarantees at-least-once publishing: if it crashes after publishing but before marking the row sent, it'll republish on restart. So consumers can receive duplicates — which is fine, because (as every lesson in this stage keeps insisting) consumers are idempotent and dedupe on the event id. The outbox converts "atomically write-and-publish" (impossible) into "atomically write, then reliably relay with at-least-once delivery to idempotent consumers" (entirely possible).