Load balancing: L4 vs L7

Every lesson so far has assumed one server on the other end of your request. That's how most apps start and, honestly, how a lot of them should stay longer than their authors think. But at some point one machine can't keep up — or you can't tolerate that one machine being a single point of failure — and you put several servers behind one address. The thing that decides which server each request lands on is a load balancer, and there are two fundamentally different kinds. Which one you reach for depends entirely on how much it needs to understand about the traffic it's steering.

You've already met a load balancer without knowing it. In what happens when you fetch(), DNS turned api.hisaab.measdev.me into an IP. That IP usually isn't a server — it's the front door, and something behind it fans your request out to one of many identical backends.

First, the uncomfortable truth: you probably don't need this yet

Architecture diagrams seduce people into distributing systems that would run happily on one box. So before any of it: vertical scaling goes much further than folklore admits. "Vertical" means making the one machine bigger — more cores, more RAM — instead of adding machines ("horizontal"). A single modern server with a few dozen cores and a hundred-plus gigabytes of RAM serves enormous traffic; Stack Overflow ran a top-100 website for years on a handful of web servers you could count on one hand. Your side project is not bigger than Stack Overflow.

The trigger to go horizontal isn't always throughput. There are two, and it's worth knowing which you're hitting:

• You've run out of vertical headroom — the biggest single instance you can rent still can't keep up. Rarer than people think, and usually arrives late.
• You can't tolerate a single point of failure — one machine means one power supply, one kernel panic, one bad deploy between you and total downtime. Two behind a balancer means one can die and you stay up. This trigger often arrives first, before any capacity wall.

Both lead to the same place: more than one server, and something in front deciding where requests go. Everything below is about that "something."

Two layers, two entirely different jobs

The names come from the OSI layer model, but you only need the two numbers. L4 operates at the transport layer: TCP/UDP, IP addresses and ports. L7 operates at the application layer: HTTP itself — methods, paths, headers, cookies, TLS. The difference in what they can see is the whole story.

An L4 load balancer works at the level of the TCP connection. When a client's SYN arrives, the balancer picks a backend and from then on shovels that connection's packets to it — usually by rewriting the destination IP (NAT, Network Address Translation) or, in higher-end setups, by DSR (Direct Server Return, where the backend replies straight to the client and skips the balancer on the way back). Crucially, an L4 balancer never looks inside the bytes. Your HTTPS request is an opaque encrypted stream to it — it can't read the path or a header, has no idea whether you asked for /groups or /login. It just keeps a connection glued to whichever backend it first chose. That ignorance is a feature: no TLS to terminate and no HTTP to parse means an L4 balancer is fast, cheap, and can push staggering packet volumes on modest hardware.

An L7 load balancer does the opposite: it terminates the connection. It completes the TLS handshake itself, decrypts, and parses the actual HTTP request — so it can see everything and route on content: /api/* to one fleet and /images/* to another, pin a tenant's traffic to a pool, retry a failed idempotent request against a second backend, add or strip headers, compress responses, enforce rate limits. It's a full HTTP intermediary — nginx, HAProxy in HTTP mode, Envoy, every cloud "application load balancer." The cost is exactly that work: TLS termination and an HTTP parse on every request, plus re-encrypting (or not) on the leg to the backend. Smarter, but not free.

A common production shape uses both: an L4 balancer at the edge spreading raw connections across a fleet of L7 balancers, which do the smart HTTP routing. L4 for volume and failover, L7 for brains.

How it actually picks a backend

Once a balancer decides to send a request somewhere, which backend? The algorithm matters more than it looks.

Round-robin — hand out backends in rotation: 1, 2, 3, 1, 2, 3. Dead simple, fine when every request costs about the same on equally powerful backends. Its blind spot: it counts requests, not work. If request #1 kicks off a 30-second export and #2 is a health check, round-robin keeps piling new work on the backend that's already drowning, because as far as it knows everyone got the same number of requests.

Least-connections — send the next request to the backend with the fewest open connections right now. This adapts: a backend stuck on slow requests accumulates open connections, so it stops getting new ones automatically. For the uneven request costs of most real workloads this is usually the better default — and an L4 balancer can do it, since counting connections needs no reading.

Consistent hashing — hash some key (client IP, a user ID, a cache key) to a backend so that the same key always lands on the same backend. The naive way, hash(key) % N where N is the number of backends, is a trap: the moment N changes (a node dies, or you add one), almost every key remaps to a different backend. Consistent hashing is a specific construction — picture the backends and keys placed around a ring — where adding or removing one node only remaps the keys near that node, roughly 1/N of them, and leaves everyone else put.

That "minimal remapping" property is why consistent hashing isn't a general-purpose balancing algorithm — it's a specialist, and it earns its keep in exactly two places:

• Cache shards. If you have a fleet of cache nodes and you want a given key to always hit the same one (so it's actually cached and not duplicated N times), consistent hashing means losing or adding a cache node only invalidates its slice of keys — not the entire cache. Plain modulo hashing would cold-start your whole cache every time you scaled the cluster.
• Stateful socket servers. If a specific user's realtime connection needs to keep landing on the specific server that holds their session state, consistent hashing gets them there with minimal disruption when the fleet changes. Hold that thought — it's exactly the WebSocket-scaling problem in the next lesson.

For plain stateless HTTP backends, you don't want consistent hashing — any backend can serve any request, so least-connections spreads load better. Reach for consistent hashing only when where a request lands has to be stable.

Health checks: how the balancer knows a backend is dead

A load balancer is only as good as its picture of which backends are alive. That picture comes from health checks, and there are two philosophies.

Active health checks — the balancer pokes each backend on a schedule: an HTTP GET /health every few seconds, expecting a 200 within a timeout. Miss a few in a row and the backend is marked down and pulled from rotation; pass again and it's brought back. This catches a sick backend before a real request hits it. The cost is a steady drizzle of check traffic and a /health endpoint that has to mean something — a naive one returning 200 as long as the process is up will keep a backend "healthy" while its database connection is dead and every real request 500s.

Passive health checks — the balancer watches real traffic and infers health: if requests to a backend start timing out or getting connection-refused, mark it down. No extra traffic, reacts to real failures — but it only learns a backend is dead by feeding it real users first, so someone eats the errors first. Good setups run both: passive to react instantly, active to test recovery before sending real traffic back.

Sticky sessions: a smell, and the one place it isn't

Sticky sessions (a.k.a. session affinity) mean the balancer pins a given client to the same backend for all their requests — usually via a cookie the L7 balancer sets, or by hashing the client IP at L4. It sounds convenient, and for classic HTTP it's almost always a smell — a sign of a design problem you should fix instead of paper over.

Why a smell? Because stickiness only matters if a backend is holding per-user state in its own memory — a login session, a cart, an upload in progress. Storing that in one server's RAM is exactly what breaks horizontal scaling: that user is now hostage to that one machine. It dies, their session evaporates; you can't deploy without kicking everyone off; load spreads unevenly because you can't move users freely. The fix is to make backends stateless — push session state into a shared store (Redis, a database, or a signed token the client carries) so any backend can serve any request. Then you don't need stickiness at all, and every problem above disappears. Stateless backends are the whole reason horizontal scaling works.

So when is it not a smell? When the connection itself is the state and can't be moved — most importantly, long-lived WebSocket connections to a stateful realtime server. A WebSocket isn't a series of independent requests you can spray across a fleet; it's one connection, held open for the session, living in one server's memory. You can't round-robin its frames. There, affinity isn't a workaround — it's a requirement, and consistent hashing is how you honor it while tolerating fleet changes. That's exactly the next lesson.

Go deeper

• Matt Klein — "Introduction to modern network load balancing and proxying" — Written by the author of Envoy; the single best industry survey of L4 vs L7, health checking, and modern proxy topologies — read this one first.
• High Performance Browser Networking — the connection-management chapters — Grounds why terminating TLS and reusing connections at the balancer matters, tying load balancing back to the round-trip costs from the earlier lessons.
• nginx — ngx_http_upstream_module docs — The concrete knobs behind everything here: round-robin vs least_conn vs ip_hash, active/passive health checks, and how a real L7 balancer is actually configured.

Check yourself

Answer out loud, as if an interviewer asked. If you hand-wave, reread that section.

1. A colleague wants to put a load balancer in front of a service that gets 50 requests a second. What two questions should you ask before agreeing, and why might the honest answer be "make the one box bigger"?
2. An L4 balancer literally cannot route "/api/* to fleet A, /images/* to fleet B." Explain exactly what it is unable to see, and what an L7 balancer does to its connection that makes that routing possible.
3. You are load-balancing a set of cache nodes and want each key to keep hitting the same node even as you add and remove nodes. Why is hash(key) % N a disaster here, and what does consistent hashing change?
4. Your /health endpoint returns 200 as long as the Node process is running. The backend passes every active health check while returning 500 to real users because its database is down. What is wrong with the health check, and how would a passive check have behaved differently?
5. Five backends run at 70% CPU. One dies and within a minute all five are marked down and the site is fully offline, even though nothing else failed. Walk through the mechanism, and state the capacity rule that would have prevented it.
6. Sticky sessions are called a smell for ordinary HTTP backends but a requirement for a WebSocket server. Reconcile these: what is different about the WebSocket case, and what design change removes the need for stickiness in the HTTP case?