Under the Hood
Apis

Webhooks & backpressure

Two loosely-related realities of talking to other systems. Webhooks flip the direction of an API call — instead of you polling 'anything new yet?', the other service calls you when something happens — which is efficient and instant but makes you a server receiving untrusted, duplicated, unordered calls you must verify and ack fast. Backpressure is the more general problem underneath: what a fast producer does when the consumer can't keep up. This lesson covers how to receive a webhook safely and how the four backpressure mechanisms (flow control, buffering, load shedding, pull) keep a fast sender from burying a slow receiver — and why unbounded buffering is the trap.

Webhooks & backpressure

Two topics that seem unrelated until you build a webhook receiver and discover it's really a backpressure problem in disguise. The first is about direction — letting another service call you instead of you polling it. The second is about flow — what happens when whoever's sending (a webhook provider, a queue producer, a fast client) outpaces whoever's receiving. They belong in one lesson because receiving webhooks well is mostly about handling their flow without drowning.

Webhooks: inverting the API call

Normally you call an API: "Stripe, did this payment succeed yet?" To find out promptly you'd have to poll — ask again and again on a short interval — which is wasteful (the vast majority of polls return "nothing new") and laggy (your latency is the poll interval; poll every 30s and you learn about events up to 30s late). Polling is asking a sleeping friend "are you awake yet?" every minute.

A webhook inverts this: you register a URL, and the other service POSTs to you the instant something happens — "payment pi_123 succeeded." It's a server-to-server callback: push instead of poll, so it's instant and efficient (no empty checks). The catch is that inversion makes you the server, receiving calls from outside, and that role has four hazards you must design for — and each one is a callback to an earlier lesson:

  • It's unauthenticated by default — verify the signature. Your webhook URL is a public endpoint; anyone can POST to it, including an attacker forging "payment succeeded." So real webhook providers sign each payload: they compute an HMAC of the body with a secret only you and they share, and send it in a header. You recompute the HMAC and compare before trusting a single byte. An unverified webhook is untrusted input wearing a trusted costume — never act on one you haven't verified.
  • Delivery is at-least-once — be idempotent. Providers retry if they don't get a prompt success, so you will receive duplicates. Your handler must be idempotent — dedupe on the event id so processing "payment succeeded" twice doesn't credit an account twice. The exact discipline from the idempotency lesson, now on the receiving end.
  • Order is not guaranteed — a "created" and an "updated" for the same object can arrive out of order. Don't assume sequence; reconcile against current state.
  • Respond fast, process later. The sender times out if you're slow — and then retries, giving you duplicates and a pileup (see backpressure, below). So do the minimum synchronously — verify the signature, dedupe, and enqueue — then return 2xx immediately and do the real work in the background off a queue. A webhook receiver that does heavy processing inline is a slow receiver, and slow receivers get hammered.

Notice how the building blocks compose: a good webhook endpoint is verify (crypto) → dedupe (idempotency) → enqueue (queue) → ack fast → process async (worker). Every piece is a previous lesson.

Backpressure: what a fast producer does to a slow consumer

That "respond fast or get buried" problem is a specific case of the general one. Backpressure is what a system does when data or work arrives faster than it can be processed. You've already met it in three places — TCP flow control (the receive window telling a sender to slow down), queue backlogs (work piling up faster than consumers drain it), and a WebSocket's send buffer filling when a client reads slowly. It's the same problem each time, and there are exactly four responses:

Run a fast producer into a slow consumer and pick a policy. Unbounded climbs until it crashes — a temporary overload becomes a hard outage. Load shed rejects the excess fast (a 429); flow control slows the producer. Both keep the system alive; the crash is the one that pretends the problem isn't there.

Producer rate
When the buffer is under pressure
producer 5/tick
capacity 20
consumer 2/tick
0buffer depth
0processed
0growing…

When work arrives faster than you can process it, a buffer only delays the reckoning — every buffer must be bounded. Run the producer fast with Unbounded and watch it climb to a crash. Then switch to Load shed (reject the excess fast — a 429) or Flow control (slow the producer) and the system stays alive. You must decide in advance which — that decision is the whole of backpressure.

  • Flow control (slow the producer down). Make the producer wait until the consumer is ready — TCP's receive window is this: the receiver advertises how much buffer it has, and the sender may not exceed it. Clean when you control both ends and the producer can be slowed. Doesn't work when the producer won't or can't wait (a webhook provider won't block for you).
  • Buffering (absorb the burst). Put a queue between them so a spike becomes a longer queue rather than dropped work. Essential for smoothing bursts — but a buffer must be bounded, which sets up the trap below.
  • Load shedding (drop or reject the excess). When you can't keep up and can't slow the producer, reject the overflow fast — return 429/503 with Retry-After, or drop lower-priority work — so you stay healthy serving what you can instead of collapsing under everything. A rate limiter is load shedding at the front door. Failing fast is a feature: a quick "try later" beats a slow death.
  • Pull-based flow (let the consumer ask). Invert control so the consumer requests N items only when it has capacity, rather than having them pushed. This is what "reactive streams" and gRPC/HTTP/2 flow control do — demand flows upstream, so the producer structurally can't outrun the consumer.

Go deeper

Check yourself

Answer out loud, as if an interviewer asked. If you hand-wave, reread that section.

  1. Explain why polling a third-party API is both wasteful and laggy, and how a webhook fixes both by inverting the call. What new role does that inversion put you in?
  2. A webhook endpoint is public. Walk through the four hazards of receiving webhooks (authenticity, duplicates, ordering, latency) and name the earlier lesson each hazard's fix comes from.
  3. Why must a webhook receiver respond in milliseconds and process later? Describe what happens if it does heavy work inline, and how that becomes a retry storm.
  4. Backpressure appeared in TCP, in queues, and in WebSocket buffers. State the four responses to a fast producer / slow consumer, and give the situation where flow control works and where you must shed load instead.
  5. Explain precisely why "just buffer it" with an unbounded buffer is a deferred crash rather than a solution, and what two policies a bounded buffer must choose between when full.
  6. Fable sheds load at the rate limiter, bounds Socket.IO send buffers, and runs no queue. Explain how each is a backpressure decision, and why "no queue yet" forces inline work to stay fast.