TCP: the connection under everything

In the fetch lesson we said "TCP gives you an ordered, reliable byte stream" and moved on. That one sentence is doing an enormous amount of work, and this lesson is about what's underneath it. Because here's the uncomfortable truth: the network your packets actually travel on — IP, the Internet Protocol — guarantees nothing. It will lose packets, deliver them out of order, duplicate them, and corrupt them, and it will never tell you it did. IP's only job is "best effort, good luck."

Yet await res.json() gives you clean bytes in the right order every time. Every network promise you've awaited sat on top of a system that quietly detected the loss, re-requested the missing pieces, discarded duplicates, and reassembled everything in order before your code saw a byte. That system is TCP, and it runs in your operating system's kernel, not your app. This lesson is how it pulls that off — and why the machinery has a cost you can feel.

The shape of the whole lesson

Before the details, here's the map. TCP has to answer four questions, and each one becomes a section below:

1. How do I know what arrived? → sequence numbers and acknowledgements.
2. Who am I even talking to? → the handshake, and what a "connection" actually is (spoiler: not a wire).
3. How fast am I allowed to send? → two separate speed limits, flow control and congestion control.
4. What happens when something is lost? → retransmission, and the stall it causes for everything behind it.

Two terms you'll need throughout. A round trip time (RTT) is how long it takes one message to reach the other side and its reply to come back — roughly 5ms to a nearby CDN edge, 70ms across a region, 150ms+ on a bad mobile link. And data is in flight when you've sent it but haven't yet been told it arrived. Almost every performance idea in this lesson is really a statement about one of those two.

Reliability built on top of unreliability

IP moves packets between IP addresses. A packet is a self-contained chunk with a source address, a destination address, and a payload. Each is routed independently — two packets sent back to back can take different paths, arrive out of order, or not arrive at all. There is no connection at the IP layer; just individual packets hurled toward an address.

TCP's whole job is to build a reliable, ordered conversation out of that. It does it with three ideas, and everything else is detail:

• Sequence numbers — every byte gets a number, so the receiver can put bytes back in order and spot gaps.
• Acknowledgements (ACKs) — the receiver tells the sender "I have everything up through byte N." Anything unacknowledged is assumed lost and re-sent.
• A connection — both sides agree on starting sequence numbers up front and keep per-connection state, so those numbers mean something.

Notice what's not in that list: any help from the network. The routers in between never tell TCP that a packet died. TCP infers everything from what comes back and what doesn't. Silence is the only error message it gets, and you'll see that shape repeat all lesson — it's why loss doubles as the congestion signal, and why a dead connection can look alive for minutes.

None of this exists on the wire as a physical thing. A TCP connection is bookkeeping — a shared agreement between two kernels about what's been sent and received. Hold onto that; it matters more than it sounds.

The three-way handshake, byte by byte

Before any data flows, the two sides synchronize their sequence numbers. This is the three-way handshake: SYN → SYN-ACK → ACK. "SYN" is short for synchronize — it carries an ISN (Initial Sequence Number), the random starting point each side will count its bytes from. (Random, not zero, for security — a predictable ISN lets an attacker forge packets into your stream.)

  Client                                              Server
    │                                                    │
    │  SYN   seq=1000                                    │   "let's talk. I'll number
    ├───────────────────────────────────────────────────►   my bytes starting at 1000"
    │                                                    │
    │                     SYN-ACK  seq=5000, ack=1001    │   "ok. I start at 5000, and
    │◄───────────────────────────────────────────────────   I've got your 1000 — send 1001 next"
    │                                                    │
    │  ACK   seq=1001, ack=5001                          │   "got your 5000. we're synced."
    ├───────────────────────────────────────────────────►
    │                                                    │
    │  ── connection ESTABLISHED, first data can flow ── │

Read the numbers, they're the whole point. The client picks ISN 1000 and sends SYN. The server picks its own ISN 5000, acknowledges the client's with ack=1001, and sends its own SYN — both in one packet, which is why it's three messages and not four. The client ACKs that. Now each side knows where the other is counting from, so every future "I got byte N" means the same thing to both of them.

The one detail that trips everyone: an ACK names the next byte it wants, not the last byte it got. ack=1001 means "everything through 1000 is safely here, send me 1001." Once you read ACKs that way, the rest of TCP's numbering stops being confusing.

And now the thing to burn into memory: the handshake is one full round trip before a single byte of your request goes out. The client can piggyback data on that third ACK, but in practice your HTTP request waits for the handshake to finish. If the server is 70ms away, that's ~70ms of pure setup where nothing useful moves — before TLS, before HTTP, before your query even reaches a database. This is the cost the fetch lesson kept pointing at, and it's the first half of why a warm, reused connection is so much cheaper than a cold one. (The second half is slow start, two sections down.)

What "a connection" physically is

Ask "where is the connection?" and the honest answer is: nowhere, and in two places at once. There is no wire held open, no circuit reserved, nothing a network engineer could point at. A TCP connection is identified entirely by a 4-tuple — four numbers both sides agree on:

(source IP, source port, destination IP, destination port)
   192.0.2.5 : 51514  →  203.0.113.10 : 443

That's it. Any two packets sharing all four values belong to the same connection. Each kernel keeps a block of state for that tuple — the sequence numbers, what's been ACKed, the window sizes, buffers of data waiting to be sent or reassembled. If you've ever held a piece of UI state in a store keyed by an ID, this is the same idea: the connection is that entry in a table, and the packets are just messages that carry the key. "Establishing a connection" means creating that entry on both ends. "Closing" it means tearing it down — a FIN/ACK exchange, plus a TIME_WAIT period where the tuple lingers to catch stray late packets.

Two consequences fall out of this, and both show up in production.

The source port is what lets you have many connections at once. Your browser can hold six connections to the same server IP on the same port 443, because each one gets a different ephemeral source port. Same three values, different fourth value, different tuple, different conversation. It's also why a machine has a finite connection ceiling — you only get ~64k ports.

A connection can be alive on one side and gone on the other. Because it's just state in memory, nothing forces the two copies to agree. If a user's phone dies, or they walk into a lift, the client's state vanishes and your server's state does not. Your server still believes the connection is ESTABLISHED and will keep believing it until something forces the issue — it tries to send data and gets no reply, a timeout fires, or a keepalive probe comes back empty. There is no event that tells you "the other end is gone," because there's no wire to go slack. This is the root of half-open connections, phantom "active users," and pools full of dead sockets — and it's exactly why keep-alive and pooling need explicit health checks rather than trusting that an open socket is a working one.

Two separate speed limits

Once you're connected, how fast can you send? Two entirely different brakes apply, and people constantly conflate them. They're easier to keep straight if you name who each one is protecting.

Flow control is the simple one, because it's explicit. Every ACK carries a receive window (rwnd) — literally "I have this many bytes of buffer free right now." The sender may not have more than that in flight unacknowledged. If your app reads from the socket slowly, the kernel's buffer behind it fills up, the advertised window shrinks toward zero, and the sender is told to stop. That's it. It's backpressure, negotiated in the open, and it's purely about not overrunning the other end's memory.

Congestion control is the hard one, because nobody tells TCP anything. The routers and links between you and the server are shared by strangers and owned by no one, and none of them will report how loaded they are. So TCP maintains a second, hidden limit — the congestion window (cwnd) — that is nothing but its own current guess about how much the path can absorb. The amount TCP may actually have in flight is the minimum of cwnd and the receiver's rwnd: it obeys both the brake it was told about and the one it invented.

And since nothing reports congestion, TCP has to infer it from the only signal available: a lost packet means the network is full. That's the entire feedback mechanism. It's also why TCP historically behaves oddly on mobile — a packet lost to radio interference isn't congestion at all, but TCP can't tell the difference and slows down anyway.

Why a cold connection is slow

Here's where the congestion window explains something you've actually felt. A brand-new connection has no idea what the path can handle, so it refuses to assume. It starts timid and ramps up:

• Slow start. cwnd begins tiny — typically 10 packets, about 14 KB — and doubles every round trip as long as ACKs keep arriving. That's exponential growth, but from almost nothing. A fresh connection literally cannot send fast, no matter how much bandwidth you're paying for. It hasn't earned the trust yet.
• AIMD after that. Once cwnd crosses a threshold or hits trouble, TCP switches to Additive Increase, Multiplicative Decrease: grow by roughly one packet per round trip (cautious probing upward), but on a loss, halve it immediately. Slow to climb, quick to back off.
• Loss resets the climb. Every drop knocks the window down and the ramp starts over from a lower point.

Put slow start and round trips together with real numbers and the punchline lands hard. Say you're downloading a 1 MB JSON response over a fresh connection from a server 70ms away. Ignoring rwnd and assuming no loss, here's how much you're allowed to have in flight each round trip:

Seven round trips. At 70ms each, that's ~490ms to move 1 MB — an effective throughput of about 2 MB/s. Your connection might be provisioned for 200 Mbps; it doesn't matter, because you never got to use it. The download was never bandwidth-bound. It was round-trip-bound, and it finished before slow start had finished opening the throttle.

Play with the three knobs that actually move that number. Watch the window double each round trip, then switch the connection to Warm to see the climb vanish (keep-alive), or drop the RTT from 70 ms to 5 ms to see a CDN shrink every rung at once. Notice what never appears in the math: your bandwidth.

That single fact reframes a lot of advice you've heard as boilerplate:

• Keep-alive wins because a reused connection's cwnd is already large. You skip the handshake round trip and you skip the ramp — you start at full speed instead of at 14 KB.
• CDNs win because they shrink the RTT itself. The same seven round trips at 5ms is 35ms instead of 490ms. The ramp doesn't get shorter; each rung of it does.
• Small payloads over cold connections are almost pure overhead. If your response fits in the first 14 KB, the connection is torn down having never learned anything about the network at all.

None of these are shaving CPU. They're all removing round trips.

Retransmission and head-of-line blocking

When a packet is lost, the receiver keeps ACKing the last in-order byte it got — it can't ACK past the gap, because ACKs mean "everything through here," and everything through there isn't true. The sender sees the same ACK number repeating (or a timeout fires) and re-sends the missing segment. Correctness: preserved. But look at what it costs everything behind the gap.

TCP guarantees in-order delivery. So if segments 3, 4, 5, 6 are sent and segment 3 is lost, segments 4, 5, and 6 may have already arrived perfectly and be sitting in the kernel's receive buffer — but TCP will not hand any of them to your application until 3 is retransmitted and arrives to fill the hole. The later data is there, complete, correct, and unusable. Your app doesn't see a partial read; it sees nothing at all, and then suddenly everything at once when the gap fills.

This is head-of-line (HOL) blocking, and if you want an intuition for it, it's the same shape as blocking the main thread with one synchronous call: the work behind it is ready to go, nothing is wrong with it, and it waits anyway because the thing in front hasn't finished. One lost packet stalls a stream that is otherwise entirely intact.

This matters enormously for what comes next. When you multiplex several HTTP requests over one TCP connection — which is exactly what HTTP/2 does — a single lost packet blocks all of them, because they share one ordered byte stream. Six independent requests, one dropped packet, everyone waits. HTTP/2 fixed HTTP/1's connection-level queueing and then hit this floor, because the floor belongs to TCP, not to HTTP. Escaping it is the entire reason HTTP/3 abandoned TCP altogether, and that's the payoff we set up in HTTP/1, HTTP/2, HTTP/3. For now, hold the shape: TCP's ordering guarantee, the thing that makes it useful, is also a source of stalls.

MSS and MTU: why big writes get chopped

You can write() a megabyte to a socket in one call, but a megabyte does not travel as one packet. The physical network has a maximum packet size, the MTU (Maximum Transmission Unit) — on standard Ethernet, 1500 bytes. Subtract the IP header (20 bytes) and the TCP header (20 bytes) and you get the MSS (Maximum Segment Size), the largest chunk of your data per packet: ~1460 bytes on a 1500-byte-MTU link.

So TCP takes your byte stream and slices it into MSS-sized segments to hand to IP. Your one-megabyte write becomes ~700 packets, numbered by their sequence bytes, ACKed and reassembled at the far end into the exact stream you wrote. You never see the seams — that's the abstraction working. (It's also why "one write, one packet" is never a safe assumption, and why a TCP receiver has to frame its own messages: the stream is bytes, not the chunks you sent.)

MTU only becomes visible when it's wrong. Modern TCP marks its packets "don't fragment" so it can discover the real limit along the path. If a segment is too big for some link, that link drops it and is supposed to send back an ICMP "packet too big" message telling the sender to shrink. When a misconfigured firewall silently eats those ICMP messages — depressingly common — you get a PMTUD (Path MTU Discovery) blackhole: the sender keeps sending oversized packets, they keep vanishing, and no one ever explains why. The symptom is unmistakable once you've seen it: small requests work perfectly, large responses hang forever with no error. Connection established, headers fine, body never arrives. If that fingerprint ever shows up, suspect MTU before you suspect your code.