// pattern debugger≡ menu

stack>how the network works / ports_sockets

// Ports & Sockets

A port is an integer in a header, not a thing. How the kernel uses the four-tuple to pick which socket gets a packet, why a listening socket and a connected socket are different objects, the ephemeral range, the accept queue, and where TIME_WAIT and port exhaustion come from.

the ground floor

  • Packet, header, payload, best-effort delivery are built from nothing in what a network actually is. This page assumes all of them.
  • A header is fields at fixed offsets, and each layer wraps the layer above it — the OSI model walks one request down the stack and back up.
  • An IP address is 32 bits identifying an interface, and it is the other half of everything on this page. IP addresses, subnets and routing owns it — along with NAT, which rewrites the port half too, and is why the port a server sees is frequently not the port the client picked.
  • TCP and UDP are the two transport protocols whose headers carry the field this page is about. What they do with a connection — the handshake, sequence numbers, retransmission — belongs to TCP and UDP; this page is only about how the kernel decides which socket a segment belongs to.
  • The kernel, a syscall, and a handle table are built in processes, threads and the kernel. A socket is an object in that table, and every send and receive on it crosses that boundary.

core idea

A port is a 16-bit unsigned integer in the TCP or UDP header — two bytes at a known offset, carrying a number from 0 to 65535. That is the entire definition. It is not a resource the OS hands out, not a pipe, not a thing a process owns. The IP header has no port field at all, which is why ICMP, which rides directly on IP, has no ports and never will.

Everything else falls out of one design decision: the kernel picks which socket gets an arriving segment using the four-tuple — source address, source port, destination address, destination port — keyed separately per protocol. Because the client’s half of that tuple is part of the key, one listening port distinguishes an unbounded number of simultaneous connections without needing a second port for any of them. Ports are not what a busy server runs out of. Its clients are the ones spending a finite range, one entry at a time, and that is where the incidents come from.

how it actually works

a port is sixteen bits at a known offset

Both transport protocols put the same two fields in the same place: the first four bytes of the transport header are the source port and the destination port, in that order, big-endian.

  TCP header — offsets from the start of the TCP header

  byte    0        1        2        3
        ┌─────────────────┬─────────────────┐
        │  source port    │ destination port│   two 16-bit unsigned fields
        │    16 bits      │    16 bits      │   range 0–65535, big-endian
        └─────────────────┴─────────────────┘
  byte    4    5    6    7
        ┌─────────────────────────────────────┐
        │        sequence number (32)         │
        └─────────────────────────────────────┘
        ... 20 bytes in total, without options


  UDP header — all eight bytes of it

  byte    0        1        2        3
        ┌─────────────────┬─────────────────┐
   0–3  │  source port    │ destination port│   the same two fields,
        └─────────────────┴─────────────────┘   at the same two offsets
  byte    4        5        6        7
        ┌─────────────────┬─────────────────┐
   4–7  │     length      │    checksum     │
        └─────────────────┴─────────────────┘


  IPv4 header — 20 bytes without options, and not one of them is a port

        offset  9  protocol   (6 = TCP, 17 = UDP, 1 = ICMP)
        offset 12  source address       (4 bytes)
        offset 16  destination address  (4 bytes)

Three consequences worth having straight before anything else on this page.

Ports live above IP, so a router does not need to read them. Forwarding a packet is a destination-address decision. A device that filters by port — a security group, an ACL, an L4 load balancer — is deliberately reaching one layer up past the header it needs, into a payload it was not required to parse.

TCP port 53 and UDP port 53 are different endpoints. The lookup key includes the protocol number, and the two protocols have entirely separate tables. Two processes can hold “port 53” simultaneously as long as one is TCP and the other UDP. DNS actually uses both.

ICMP has no ports. So “ping the port” is not a thing you can do, and a rule that permits “ICMP on port 8” is a rule someone wrote from muscle memory — type 8 is echo request, a type, not a port. It also means ping succeeding proves a host is reachable and proves nothing whatsoever about whether your service is listening; that distinction is the second rung of the ladder in seeing the network.

the four-tuple is the demultiplexing key

A segment arrives at the network interface. The kernel has, potentially, thousands of sockets open. It picks one by building a key out of the protocol and the four addresses in the two headers, and looking it up.

Here are four connections arriving at one server, all to the same port, all from the same client machine. Nothing distinguishes them except the source port:

protocol source address source port destination address destination port delivered to
TCP 198.51.100.7 41102 203.0.113.10 443 connected socket A
TCP 198.51.100.7 52377 203.0.113.10 443 connected socket B
TCP 198.51.100.7 33914 203.0.113.10 443 connected socket C
TCP 198.51.100.7 49265 203.0.113.10 443 connected socket D

Four distinct keys, four distinct sockets, one destination port. The server did not allocate four ports, and there is no mechanism by which it could have needed to. Its port appears in every key and therefore distinguishes nothing; the client’s port is what does the work.

That is the whole trick, and it is the fact most people are missing when they ask “how many connections can a port handle”. The answer is: as many as the box has file descriptors and memory for. The port is not the scarce thing.

four-tuple, five-tuple, same thing

People say five-tuple when they want to include the protocol number explicitly, and four-tuple when the protocol is understood from context. They are the same key. An L4 load balancer that “hashes the 5-tuple” is hashing exactly this.

two kinds of socket, and the order they are matched in

A listening socket and a connected socket are two different kernel objects with two different shapes.

A listening socket has only a local half. It is created by bind() — which fills in the local address and port — followed by listen(), which tells the kernel to start accepting handshakes for it. Its remote half is a wildcard, because it does not have a peer. No application data ever flows through a listening socket. It is a factory.

A connected socket has all four fields filled in. On the server it is produced by accept(), which returns a new socket and leaves the listener exactly as it was. On the client it is produced by connect(), which fills in the remote half from what you asked for and the local half from whatever the kernel picks.

The lookup runs in that order — specific first, wildcard last:

  a segment arrives:  proto=TCP  src=198.51.100.7:41102  dst=203.0.113.10:443

  1. build the key  (TCP, 198.51.100.7, 41102, 203.0.113.10, 443)

  2. look for a connected socket with EXACTLY that four-tuple
        hit  → append the payload to that socket's receive buffer. done.
        miss ↓

  3. look for a listening socket matching only the local half —
     203.0.113.10:443, or the wildcard 0.0.0.0:443
        hit,  and the segment is a SYN  → begin a handshake (see the queues below)
        hit,  and it is not a SYN       → RST: there is no such connection
        miss ↓

  4. nothing is bound here at all → RST, which the client's connect() reports
     as "connection refused"

Step 4 is the single most useful diagnostic fact on this page. An RST means something was there and said no. Silence until a timeout means the packet was dropped and nobody said anything — a security group, a route that goes nowhere, a wrong address. Those are different diagnoses with different fixes, and conflating them sends you down the wrong path entirely.

the port ranges, and which of them you may bind

range name who uses it the rule
0 reserved nobody, on the wire in bind() it means “kernel, pick one for me”
1–1023 well known the classic services: 22, 25, 53, 80, 443 on Unix, binding one requires privilege
1024–49151 registered vendor-assigned: 1433, 3306, 5432, 6379 bind freely
49152–65535 dynamic / private IANA’s suggested ephemeral range bind freely; most kernels ignore this suggestion

The privileged-bind rule is Unix-specific and it is a capability, not a check for uid 0: on Linux a process needs CAP_NET_BIND_SERVICE to bind below 1024, which root has and your container’s app user almost certainly does not. That is the entire reason Kestrel in a container is usually configured on 8080 rather than 80. The threshold itself is tunable through net.ipv4.ip_unprivileged_port_start, which is how some base images let an unprivileged process bind 80 anyway. Windows has no equivalent restriction.

The ephemeral range is the pool a kernel draws from when it has to invent a local port — either because you called connect() without binding, or because you asked for port 0 explicitly. On the Linux box these pages were written on, that pool is:

  /proc/sys/net/ipv4/ip_local_port_range
    32768   60999

which is 28,232 ports, and it is not the IANA range above. Linux picked a wider, lower window years ago and never moved it. Two practical consequences: the ephemeral ports you see in a packet capture or an ss listing will usually be 32768-and-up rather than 49152-and-up, and any service you deliberately bind inside that window is at risk of an ephemeral allocation getting there first. If you must run a fixed service on, say, 40000, either move it below 32768 or exclude it with ip_local_reserved_ports.

binding to 127.0.0.1 versus 0.0.0.0

The address you pass to bind() is not decoration. It is half of the lookup key in step 3 above, and it decides which arriving packets can match your listener at all.

  • 127.0.0.1 — match only packets whose destination address is the loopback address. Those only ever come from this same machine, because loopback is not reachable from anywhere else.
  • 0.0.0.0 — the wildcard: match packets to any address this host has, on any interface, including interfaces that appear after you bound. This is a bind-time-only meaning of 0.0.0.0; in a routing table the same four bytes are the default route instead — IP addresses, subnets and routing has both readings side by side.
  • A specific interface address, say 203.0.113.10 — match only packets addressed to that interface. Useful when a box is multi-homed and you want a management service on one NIC only.
  • :: with dual-mode — an IPv6 wildcard socket that also accepts IPv4-mapped connections. In .NET that is IPAddress.IPv6Any with Socket.DualMode set to true.

This is the most common self-inflicted outage in the whole section. The service starts, the log line says it is listening, health checks from inside the container pass, and every request from outside gets connection refused. The refusal is the tell: the host was reachable and its kernel answered, which means the packet arrived and step 3 found no listening socket for that destination address. The listener was bound to loopback. Nothing about the network is broken.

  ANNOTATED SCHEMATIC — not a captured session. This is the shape `ss -tan` prints
  (`-a` because LISTEN and TIME-WAIT are omitted by default), with placeholder
  addresses from the documentation ranges.

  State       Local Address:Port      Peer Address:Port
  LISTEN      0.0.0.0:443             0.0.0.0:*        ← listening socket: local half only,
                                                          peer is a wildcard, no data flows here
  LISTEN      127.0.0.1:5000          0.0.0.0:*        ← the outage. Loopback only. Nothing
                                                          from another host can ever match it
  ESTAB       203.0.113.10:443        198.51.100.7:41102
  ESTAB       203.0.113.10:443        198.51.100.7:52377  ← same local half as the row above,
                                                             different peer, different socket
  TIME-WAIT   203.0.113.10:41230      198.51.100.20:9042  ← an outbound connection this host
                                                             closed first; see below

Bare ss -tn prints only established connections — its default state filter drops LISTEN and TIME-WAIT — so reach for -a when you are looking for a listener or counting TIME-WAIT, and -l when you want listeners alone.

In ASP.NET Core the knob is the URL you bind: http://localhost:5000 is loopback-only and http://0.0.0.0:8080 or http://+:8080 is the wildcard. The default is the loopback one, which is correct for a developer laptop and wrong for every container, and that mismatch is the whole bug.

the two queues behind listen()

listen(backlog) does not mean “accept up to backlog connections”. It sizes a queue, and there are actually two of them, holding connections at two different stages of the handshake.

  client                     kernel, server side                    your process

  SYN ─────────────────────→ create a half-open entry
                             in the SYN QUEUE          ← sized by tcp_max_syn_backlog
       ←──────── SYN-ACK

  ACK ─────────────────────→ handshake complete: move
                             the entry to the
                             ACCEPT QUEUE              ← sized by listen(backlog),
                                    │                    capped by net.core.somaxconn

                                    └──── accept() ───→ a NEW connected socket, four-tuple
                                                         filled in. The listener is untouched
                                                         and is still listening on the same port

  what happens when the ACCEPT QUEUE is full and the client's ACK arrives anyway:

      Linux default (net.ipv4.tcp_abort_on_overflow = 0)
        the ACK is silently DROPPED. But the client sent that ACK, so as far as the
        client is concerned the connection is ESTABLISHED — connect() returned, and
        it can write into a socket nobody will ever read. The server retransmits its
        SYN-ACK and hopes somebody calls accept() in the meantime.

      with tcp_abort_on_overflow = 1
        RST instead: the client fails fast with "connection reset", which is a much
        better failure and a much worse thing to enable under a load spike, because
        every retryable blip becomes a hard error.

The important half is the client’s view. An overloaded accept queue does not look like a network problem to the client; it looks like your service being silent. And a full accept queue is almost never a networking condition — it means the process is not calling accept() fast enough, which means its threads are busy or blocked. That is a scheduling story, and it is the one in processes, threads and the kernel: a thread pool full of threads parked on synchronous calls stops draining this queue exactly the way it stops draining any other.

The SYN queue is the other one, and it has its own escape hatch. When it fills — classically under a SYN flood, where an attacker sends SYNs and never completes the handshake — Linux can switch to SYN cookies, encoding the connection state into the initial sequence number it sends back so it does not have to remember anything until the client’s ACK returns it.

TIME_WAIT, and the ports it holds

When a TCP connection closes, the end that sends the first FIN — the active close — holds that socket’s four-tuple in TIME_WAIT afterwards. Not both ends. Not the server by definition. Whichever one closed first.

It sits there for 2×MSL — twice the maximum segment lifetime, the design assumption about how long a segment can wander the network before it must be gone. Two jobs, and both of them matter:

  1. Absorb delayed duplicates. A segment from the old connection could still be in flight somewhere. If the same four-tuple were immediately reusable, that stray segment would be delivered into a brand-new connection’s byte stream as though it belonged there. Holding the tuple makes that impossible.
  2. Be able to retransmit the final ACK. If the peer’s FIN is lost, the peer retransmits it, and something has to be there to answer. A tuple that has been forgotten answers with an RST, which the peer reports as an error on a connection that closed perfectly cleanly.

On Linux the duration is a compile-time constant rather than a sysctl, which surprises everyone who goes looking for a knob. The knob they find, net.ipv4.tcp_fin_timeout, governs FIN_WAIT_2 — how long to wait for the peer’s FIN — and turning it down does not shorten TIME_WAIT by one bit.

Which end pays depends entirely on who closes:

shape who actively closes who accumulates TIME_WAIT what it costs them
client opens a connection per request and closes it the client the client one ephemeral port, per destination, held after the work is done
server closes after each response, no keep-alive the server the server table entries and memory — its port is fixed, so no port is consumed
pooled connections reused for many requests nobody, most of the time nobody this is the fix

Ephemeral port exhaustion is the incident. A client opening a fresh connection per outbound call spends one ephemeral port per call and gets it back only after TIME_WAIT expires. Once the connection rate outruns that, connect() has nothing left to allocate. On Linux it fails with EADDRNOTAVAIL — surfaced in .NET as a SocketException with SocketError.AddressNotAvailable, whose message reads “Cannot assign requested address” and which says nothing at all about ports, so nobody recognises it the first time.

One detail that decides how bad this gets: the constraint is only that the four-tuple be unique, so the same local port may legitimately serve many different destinations. Linux’s allocator does exactly that, so the practical ceiling is roughly one ephemeral port per concurrent plus-recently-closed connection to the same destination endpoint, not per connection overall. Whether any given kernel is that clever is allocator policy, not protocol — do not port the assumption to another platform without checking it.

the two folk remedies, both wrong

SO_LINGER with a zero timeout does make TIME_WAIT go away, by making close() send an RST instead of a FIN. That discards anything still queued in the send buffer — including a response your peer has not read yet — and destroys precisely the guarantee TIME_WAIT exists to provide. In .NET it is new LingerOption(true, 0), and if it is in your codebase without a written justification, it is a bug.

Cranking the timeout down is the other one. On Linux there is no timeout to crank; on platforms where there is, shortening it trades a bounded resource problem for an unbounded correctness one — stray segments delivered into the wrong connection, which will not look like a networking bug when it finally bites.

The fix for port exhaustion is connection reuse: IHttpClientFactory or a long-lived SocketsHttpHandler for HTTP, the ADO.NET connection pool for databases. Stop closing the connection and there is no TIME_WAIT to manage. If you genuinely need a kernel knob after that, net.ipv4.tcp_tw_reuse lets a new outbound connection reuse a TIME_WAIT tuple when TCP timestamps make it safe. Its old sibling tcp_tw_recycle was removed from Linux entirely because it broke every client behind a NAT.

The pooling design that prevents all of this — keep-alive, pool sizing, per-host limits — belongs to the network path. The reason the ports ran out belongs here.

SO_REUSEADDR, SO_REUSEPORT, and ExclusiveAddressUse

These three get conflated constantly, and every claim about them has to name its platform, because they do not mean the same thing on Linux and Windows.

option platform what it actually does what it does not do
SO_REUSEADDR Linux, and BSD-family lets bind() succeed on a local address that still has connections sitting in TIME_WAIT, so a restarted server does not have to wait them out it does not let two live listeners share one address and port
SO_REUSEPORT Linux 3.9 and later lets several sockets bind the exact same address and port, provided every one of them sets the option; the kernel then hashes each incoming connection’s four-tuple to choose which of them gets it it is not available on Windows, and it is not a fix for TIME_WAIT
SO_REUSEADDR Windows historically permitted a second socket to bind an address another socket already held — which is a hijacking primitive, not a convenience it is not the Linux semantics, and reasoning about it as though it were is how the confusion starts
SO_EXCLUSIVEADDRUSE Windows the option added to refuse exactly that hijack; .NET surfaces it as Socket.ExclusiveAddressUse and TcpListener.ExclusiveAddressUse, and it is mutually exclusive with SO_REUSEADDR it has no Linux counterpart, because Linux never had the hole

SO_REUSEPORT is the interesting one, because it is the mechanism behind multi-process accept: run four worker processes, have each bind the same port with the option set, and the kernel distributes connections across them without any of them sharing a listening socket or fighting over one. Every process must set it, and by default they must all run as the same effective user, which is the guard against a hostile process stealing your traffic.

.NET exposes SO_REUSEADDR by name and does not expose SO_REUSEPORT, so on Linux you set the latter raw:

using System.Net;
using System.Net.Sockets;

// Linux only. SOL_SOCKET is 1 and SO_REUSEPORT is 15 on the common architectures —
// these are ABI constants, so confirm them for anything exotic.
const int SOL_SOCKET = 1;
const int SO_REUSEPORT = 15;

var listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);

// The portable one, and the one people almost always mean: bind over TIME_WAIT.
listener.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);

// The one that actually lets a sibling process bind this same endpoint.
listener.SetRawSocketOption(SOL_SOCKET, SO_REUSEPORT, BitConverter.GetBytes(1));

listener.Bind(new IPEndPoint(IPAddress.Any, 8080));
listener.Listen(backlog: 128);

what a socket is, in C#

“Socket” means three different things and this subject uses all three in the same sentence constantly:

  1. The kernel object. On Linux it is a file descriptor in the same table as your open files — accept() returns one, close() releases one, and running out of them is an EMFILE/ENFILE problem, not a networking one. Every read and write on it crosses into the kernel, which is a real cost with a real mechanism behind it: processes, threads and the kernel has that story.
  2. The C# class. System.Net.Sockets.Socket is a managed wrapper around a handle to that object. TcpListener is bind + listen + accept with a friendlier surface; TcpClient is connect plus a NetworkStream over the result.
  3. An address and port written down203.0.113.10:443. This one is really an endpoint, and .NET names it correctly: IPEndPoint. It is not a socket in either of the senses above, and half the confusion in this area comes from the same word covering both a kernel object and a pair of numbers.

Here is the whole shape of the thing in one program: bind to port 0, ask the kernel what it gave you, take one connection, and print both halves of the four-tuple from the server’s side.

using System.Net;
using System.Net.Sockets;

// Port 0 is not a port. It is "kernel, allocate me one", drawn from the same
// ephemeral range a client's connect() draws from — 32768–60999 on the Linux
// box these pages were written on.
var listener = new TcpListener(IPAddress.Loopback, 0);
listener.Start(backlog: 16);

var bound = (IPEndPoint)listener.LocalEndpoint;
Console.WriteLine($"listening on {bound.Address}:{bound.Port}");

// One client, dialling the port we were just handed. It does not bind anything:
// connect() fills in its local half from the ephemeral range too.
using var client = new TcpClient();
await client.ConnectAsync(bound.Address, bound.Port);

// Accept returns a NEW socket. The listener still exists, still on the same port.
using var accepted = await listener.AcceptTcpClientAsync();
var conn = accepted.Client;

Console.WriteLine($"local  {conn.LocalEndPoint}");   // the listening port, reused as-is
Console.WriteLine($"remote {conn.RemoteEndPoint}");  // the client's ephemeral port

listener.Stop();

What this would print is three lines: a listening endpoint on some ephemeral port, a local line carrying that same port, and a remote line carrying a different one. Working out why the first two match and the third does not is one port, many connections.

the mental model

  a port  = 16 unsigned bits in the TCP or UDP header. Nothing else. 0–65535.
            IP has no such field, so ICMP has no ports.

  the key the kernel demultiplexes on:

      (protocol, src addr, src port, dst addr, dst port)

  matched in this order:
      1. a connected socket with all four fields equal      ← data goes here
      2. a listening socket matching the local half only     ← only SYNs get here
      3. nothing → RST → "connection refused"

  two objects, not one:
      listening socket  = local half + wildcard peer. A factory. No data.
      connected socket  = all four fields. Produced by accept() or connect().

  who spends a finite resource:
      the server's port      no  — it is in every key, so it separates nothing
      the client's ports     YES — one ephemeral port per connection,
                                   held past close by TIME_WAIT on the end
                                   that closed FIRST

  bind address is half the key:
      127.0.0.1  only packets to loopback can match   ← the classic outage
      0.0.0.0    any address this host has
port field width = 16 bits, unsigned — 0 to 65535
where it lives = TCP and UDP headers only, first 4 bytes
well-known range = 0–1023, privileged bind on Unix
ephemeral range, this Linux box = `32768–60999`, from `/proc/sys/net/ipv4/ip_local_port_range`
demux key = protocol + four-tuple
`accept()` returns = a new socket; the listener is unchanged
`TIME_WAIT` is held by = the end that sent the first FIN, for 2×MSL
a refusal vs a timeout = RST = something said no · silence = something dropped it

why you should care

The bind-address outage, which you will meet more than once. The symptom is exact: works from inside the container or off the developer’s laptop, connection refused from anywhere else, and nothing wrong in any log. The refusal itself is the diagnosis — a dropped packet gives you a timeout instead, so an immediate refusal proves the host answered and its kernel found no listening socket for that destination address. Check the bind address before you check anything else: ASPNETCORE_URLS, the UseUrls call, or the Kestrel:Endpoints config — which is why it opens the ladder in seeing the network.

Ephemeral port exhaustion, and the HttpClient shape that causes it. A new HttpClient() per request opens a fresh connection per request and drops it, spending one ephemeral port each time and holding it through TIME_WAIT. Under load, connect() eventually has nothing to hand out and you get SocketException / SocketError.AddressNotAvailable — a message about addresses, from a problem about ports, on a call site that looks like HTTP. The trap has a second jaw: the “just make it static then” fix pins DNS results forever, so a failover the DNS layer already published never reaches you. IHttpClientFactory is the answer that closes both, because it pools connections and rotates the underlying handler on a lifetime. Both halves live on DNS and the network path; the port-exhaustion half is this page’s.

A silent service that is actually a full accept queue. Clients report requests that hang and eventually time out, while the service’s own metrics look fine — no errors, no slow queries, request counts simply lower than the clients claim to be sending. That gap is the shape: the connections completed their handshake in the kernel and are sitting in the accept queue, so the client believes it is connected and your application has never seen them. Correlate the client’s “connected but no response” against the server’s request count, and check whether the process is actually calling accept() — which usually means checking whether the thread pool is starved. The queue is a symptom; the blocked threads are the disease.

The code review you can now do. A new HttpClient() inside a method. A LingerOption(true, 0) anywhere. A service that binds IPAddress.Loopback with a config comment saying “for security” — that is not a security control, it is a deployment constraint, and the firewall or network policy is where that belongs. A listen() backlog of 1 or 5 copied from a tutorial. A retry loop that opens a new connection per attempt against a dependency you already have a pool for. And any comment that says “we ran out of ports on the server”, which is a sentence describing something that did not happen.

Two pages follow directly from this one. TCP and UDP picks up what the connected socket does once it exists — the handshake whose completion moved that entry between the two queues, sequence numbers, retransmission, the close sequence that produces TIME_WAIT, and the byte-stream truth that no number of correct four-tuples will save you from. Seeing the network turns the refusal-versus-timeout distinction into a ladder you can walk in order under pressure.

the same idea elsewhere

the idea here where else it shows up what carries over, and what does not
the four-tuple as the identity of a connection any composite key — a dictionary keyed on a tuple, a composite primary key identical: the entity is identified by the whole key, and reasoning about one column of it (the server port) tells you nothing about uniqueness. The difference is that this key is built by two machines that never agreed on it, one field each
a port number naming an endpoint inside a host a process id naming a process inside a kernel — processes and threads both are small integers from a bounded namespace, both are reused after the thing they named is gone, and both produce the same class of bug: you hold the number, the thing dies, the number is reissued, and you are now talking to a stranger. TIME_WAIT is TCP’s answer to exactly that
TIME_WAIT holding a tuple after the work is done an idempotency key or a deduplication window retained past the request it guarded — timeouts, retries and circuit breakers same bargain: you keep state you no longer need so a late duplicate has somewhere to land harmlessly. Same temptation, too — shortening the window to save resources reintroduces exactly the duplicate the window existed to catch
the ephemeral range as a finite pool drawn from per connection a connection pool’s maximum size, or a semaphore bounding concurrency same exhaustion behaviour and the same misleading symptom: the failure surfaces as slowness or a strange error at the acquire point, far from the code that leaked the resource. The difference is that nobody sized the ephemeral range for your workload and you cannot raise it much
the accept queue between the kernel and accept() a bounded Channel<T> between a producer and a consumer that has stalled same backpressure structure, but note which failure the kernel picked: by default it drops the arrival silently rather than rejecting it, so the producer never learns. An application queue that does that instead of throwing is a bug; here it is the documented default

exercises

One listener, three clients, and the printed four-tuples that settle what the kernel is actually keying on.

  1. Open several connections to one listening port and print the four-tuples: predict what the kernel uses to tell them apart before you look.

interview drills

Q. A web server is handling ten thousand concurrent connections on port 443. Is it running out of ports?

  • weak answer — “It has 65535 of them, so it has room for a while.” That answer says the candidate thinks connections consume server ports, and the follow-up will find out.
  • strong answer — No, and it could not. The kernel demultiplexes on the four-tuple, and the server’s port is identical in all ten thousand of those keys, so it distinguishes nothing — the client address and client port are what separate them. What the server is spending is file descriptors and socket memory, one per connected socket. The finite port range belongs to the client side, where each outbound connection needs a distinct ephemeral port per destination.
  • follow-up — “So when does a port range actually run out?” On a machine making many short-lived outbound connections to the same destination: each one holds an ephemeral port through TIME_WAIT after closing, and connect() starts failing with “cannot assign requested address”.

Q. Your service works from your laptop and gets connection refused from the pod. Walk me through it.

  • weak answer — “Check the firewall and the security group.” Possible, but the refusal already ruled them out and the candidate did not notice.
  • strong answer — The word “refused” is the whole clue. A refusal is an RST, which means the packet arrived and the kernel answered — so routing, the security group and the firewall all worked. What failed is the socket lookup: no listening socket matched that destination address and port. The overwhelmingly likely cause is a listener bound to 127.0.0.1 rather than 0.0.0.0, so I would check the bind address first, and only then whether the process is listening on a different port than I think.
  • follow-up — “And if it were a timeout instead?” Then the opposite: nothing answered, the packet was dropped in flight, and now it is a security group, a route, or a wrong address.

Q. What is TIME_WAIT for, which end has it, and how would you get rid of it?

  • weak answer — “It is the OS holding the socket open after close; you lower tcp_fin_timeout.” Two errors: tcp_fin_timeout governs FIN_WAIT_2, and “get rid of it” is the wrong goal.
  • strong answer — It sits on whichever end sent the first FIN, for 2×MSL, and it does two jobs: it keeps the four-tuple reserved so a delayed duplicate from the old connection cannot be delivered into a new one, and it keeps enough state to re-answer a retransmitted FIN instead of replying with an RST. The right way to have less of it is to stop closing connections — pool and reuse them — not to shorten it. Shortening it trades a resource problem for a correctness problem.
  • follow-up — “What about SO_LINGER set to zero?” That replaces the FIN with an RST, which skips TIME_WAIT by discarding unsent data and abandoning the guarantee. It is a way of hiding the symptom that can lose a response the peer has not read.

Q. What is the difference between SO_REUSEADDR and SO_REUSEPORT?

  • weak answer — “Both let you share a port.” That is one of the two, on one platform.
  • strong answer — On Linux they are different mechanisms. SO_REUSEADDR lets bind() succeed when the address still has connections in TIME_WAIT, which is what a restarting server wants; it does not let two live listeners share an endpoint. SO_REUSEPORT, from Linux 3.9, is the one that does: several sockets bind the same address and port, all of them setting the option, and the kernel hashes each connection’s four-tuple to pick which socket receives it. That is how you get multi-process accept without a shared listener.
  • follow-up — “And on Windows?” Different semantics entirely — SO_REUSEADDR there historically allowed binding over another socket, which is why SO_EXCLUSIVEADDRUSE exists; .NET surfaces it as ExclusiveAddressUse. Any claim in this area needs its platform named.

Q. Clients say requests hang; your service shows no errors and a lower request count than they report sending. Where do you look?

  • weak answer — “They must not be reaching us — a network issue.” The count gap says otherwise.
  • strong answer — That gap is the signature of a full accept queue. The handshakes completed in the kernel, so the clients are genuinely ESTABLISHED and waiting, but the connections are queued behind an accept() the process is not getting to, so the application never counts them. Linux drops the completing ACK by default rather than resetting, which is exactly why the client sees silence instead of an error. I would check the queue overflow counters and the listen backlog, and then go looking for why the process is not accepting — usually threads blocked on something synchronous.
  • follow-up — “Would raising the backlog fix it?” It buys headroom for a burst; it does nothing for a sustained mismatch, and a deeper queue just means clients wait longer before failing.

Q. Why does ICMP not have ports?

  • weak answer — “Because ping does not need one.” True and uninteresting.
  • strong answer — Because ports are a field in the TCP and UDP headers, and ICMP is not carried inside either — it rides directly on IP, and the IP header has no port field. Ports are a transport-layer demultiplexing mechanism, and ICMP is not a transport protocol; it is IP’s own error and diagnostic channel. That is also why firewall rules for ICMP are written in types and codes rather than ports, and why a NAT has to improvise, using the echo identifier field as a stand-in so it can map replies back.
  • follow-up — “What breaks if you drop all ICMP?” Path MTU discovery, which depends on receiving type 3 code 4 — the classic black hole where small requests work and large ones hang.

cheat sheet — ports sockets

recognize it

  • "works on my laptop, connection refused from the pod" → a listener bound to 127.0.0.1 instead of 0.0.0.0
  • SocketException with SocketError.AddressNotAvailable ("Cannot assign requested address") → ephemeral ports gone, on the *client* side
  • clients report connections that establish then go silent, while your request count sits below what they sent → a full accept queue, not a network fault
  • a growing pile of TIME-WAIT rows whose local port is ephemeral and whose peer is one dependency → a connection-per-call client
  • hundreds of ESTAB rows sharing one local address:port → normal; that is exactly what a working server looks like

key tricks

  • the demux key is (protocol, src addr, src port, dst addr, dst port) — the server port is in every key, so it distinguishes nothing
  • RST means something answered and said no; silence until a timeout means something dropped it — different diagnoses, never conflate them
  • bind port 0 and read LocalEndPoint back: the only race-free way to get a port the kernel agrees is free
  • the fix for port exhaustion is connection reuse — IHttpClientFactory, a long-lived SocketsHttpHandler, the ADO.NET pool — not a kernel knob
  • SO_REUSEADDR on Linux = bind over TIME_WAIT; SO_REUSEPORT = several sockets on one endpoint; Windows means something else again — name the platform every time

common bugs

  • thinking a busy server runs out of ports — it runs out of file descriptors; the *client* is what spends a finite range
  • SO_LINGER with a zero timeout to "fix" TIME_WAIT — it replaces the FIN with an RST and discards unsent data
  • lowering net.ipv4.tcp_fin_timeout to shorten TIME_WAIT — that knob governs FIN_WAIT_2 and does nothing here
  • reading listen(backlog) as a cap on concurrent connections rather than on completed-but-un-accepted ones
  • assuming accept() returns connections in the order the clients dialled — it is handshake-completion order

// connections