Idempotency, retries & safe writes

Here is the problem that makes writes genuinely hard, and it has nothing to do with your code being wrong. A mobile client sends POST /expenses to add a ₹500 dinner. The server receives it, creates the expense, commits it — and then, as it sends the 201 Created back, the user walks into an elevator and the connection drops. The client waited, timed out, and now faces a question it cannot answer from its own side: did the write happen or not? "Request failed" and "request succeeded but I never got the reply" look identical to the client. If it assumes failure and retries, it creates the expense twice. If it assumes success and doesn't, it might have lost the write. This ambiguity is fundamental — the unreliable network guarantees it — and every robust write path has to have an answer for it.

Safe and idempotent methods

HTTP's method semantics are the first line of defense, and they're not arbitrary — they're a contract about what's safe to repeat:

• Safe methods — GET, HEAD — have no side effects. They only read. Retrying a GET a hundred times changes nothing, so they're trivially safe to repeat and to cache.
• Idempotent methods — GET, HEAD, PUT, DELETE — produce the same effect whether called once or many times. PUT /expenses/e1 with a full body sets that resource to a state, so doing it twice leaves the same state as doing it once. DELETE /expenses/e1 removes it; deleting an already-deleted thing is still "it's gone." So a client can safely retry these after a lost response — worst case, it re-applies a state that was already applied.
• Not idempotent — POST — is the dangerous one. POST /expenses creates a new resource each time, so retrying it after a lost response creates a second expense. This is why the elevator problem specifically bites POST.

The practical rule that falls out: model writes as idempotent operations where you can (a PUT to a client-chosen id instead of a POST that allocates one), and where you genuinely need POST semantics, make it retry-safe explicitly — which is the idempotency key.

Idempotency keys: making POST safe to retry

The pattern (popularized by Stripe, now standard for any serious write API) is simple and powerful:

1. The client generates a unique key for each logical operation — a fresh UUID/ULID per "add this expense" action — and sends it in a header (Idempotency-Key).
2. The server, before processing, checks whether it has seen that key before.
   • Never seen → process the request, and record the key together with the response it produced.
   • Seen before → do not process again; return the stored response from the first time.
3. Crucially, a retry reuses the same key (it's the same logical action), while a genuinely new operation gets a new key.

Now the elevator problem dissolves. The client retries with the same key; the server recognizes it already created that expense and replays the original 201 instead of creating a second one. The write happened exactly once, and the client got its answer. This is exactly-once effects — the same phrase from the queues lesson, because it's the same idea one layer up: you don't prevent the duplicate request, you make the duplicate harmless by recognizing and deduplicating it.

Run the double-charge yourself. With the key OFF, send a charge and hit Retry (a lost response) — every retry books another ₹500 and the "actually charged" total runs past what the user meant, with nothing ever erroring. Flip the key ON and retry the same way: the server recognises the key and replays the original 201, so the charge happens exactly once.

A few details separate a real implementation from a toy one:

• Scope the key to the user and endpoint (e.g. hash user_id + key + path), so one user's key can't collide with another's, and the "same key" only means "same operation."
• Detect body mismatches. If the same key arrives with a different body, that's a client bug (a key reused for a different operation) — reject it (409) rather than silently replaying the wrong response.
• Store with a TTL. Keys don't need to live forever — a day or so covers realistic retries — so they go in a fast store like Redis with expiry, not permanently in your database.

Retry discipline: don't turn recovery into an outage

The other half is how the client retries. Naive retries are their own disaster:

• Only retry what's safe. Retry idempotent methods and keyed operations freely; do not blindly retry a non-idempotent POST without an idempotency key — you'll double-submit.
• Back off exponentially, with jitter. Don't retry immediately or on a fixed interval. Wait 1s, 2s, 4s, 8s… and add randomness (jitter) so that a thousand clients whose requests all failed at the same instant don't all retry in the same next instant. Backoff spaces out one client; jitter de-synchronizes the herd.
• Cap the retries, then surface a real error. Infinite retries just extend the pain.
• Respect Retry-After. If the server (or a rate limiter) said 429/503 with a Retry-After, honor it.