Stage 4

Distributed patterns

Event-driven systems, outbox, sagas, consensus, the exactly-once myth.

1. Event-driven architecture & the outbox pattern

   The instant one service has to tell another that something happened, you hit a problem that looks trivial and isn't: you need to commit a database change and publish an event, and there is no way to do both atomically across two systems. Do it naively and you either publish events for writes that rolled back, or lose events for writes that committed. This lesson explains event-driven architecture (events vs commands, and the decoupling it buys), the dual-write problem at its core, and the outbox pattern — the standard fix that turns two unreliable writes into one atomic database transaction plus a relay.

   14 min
2. Sagas: orchestration vs choreography

   When a single business action spans several services — book flight, charge card, reserve hotel — you can't wrap it in one transaction, so you use a saga: a sequence of local transactions, each with a compensating action that undoes it if a later step fails. This lesson goes past 'what a saga is' into how you actually coordinate one: choreography (services react to each other's events, no central brain) versus orchestration (one coordinator explicitly drives the steps), the real trade-offs between them, why compensations aren't rollbacks, and the failure that has no clean answer — the compensation that itself fails.

   14 min
3. Consensus: how a group of nodes agree (Raft)

   The hardest problem in distributed systems, stated simply: get a group of machines to agree on one value — or one ordered log of values — even though nodes crash and messages get lost, delayed, and reordered. This is what elects a leader, stores the configuration a whole cluster trusts, and backs distributed locks. This lesson explains why agreement is so hard (no shared clock, no reliable messages, the FLP result), then walks Raft — leader election, terms, log replication, and the majority-quorum rule that makes split-brain impossible — and why you keep consensus off your request hot path.

   15 min
4. Distributed locks, leader election & fencing tokens

   A lock inside one database is easy — one node arbitrates. A lock across machines is one of the most dangerous things you can build, because the mechanism that makes it safe against a crashed holder (a TTL) is exactly what lets two nodes believe they hold the same lock at once. This lesson explains why distributed locks are hard, the process-pause failure that breaks the naive Redis lock, the fencing token that actually fixes it, how leader election is just distributed locking in disguise, and the crucial distinction between locks you hold for efficiency and locks you hold for correctness.

   14 min
5. The delivery-guarantee myths: exactly-once & the two generals

   Every messaging vendor advertises 'exactly-once,' and the phrase is, taken literally, a lie — exactly-once delivery over an unreliable network is provably impossible, and the two generals problem is why. But there's a real thing underneath the marketing worth understanding precisely: exactly-once delivery can't exist, yet exactly-once effects can, and the bridge between them is the single most important idea in the whole curriculum — at-least-once delivery plus idempotent processing. This capstone lesson proves the impossibility, dismantles the myth, and shows how every reliability pattern you've met is the same answer to it.

   13 min