// Seeing the Network

the ground floor

• A name is not an address. The stub resolver in your process turns one into the other, and it can succeed, fail, or succeed with a stale answer — or a split-horizon one, the same name deliberately answered differently depending on which resolver asks — entirely independently of the rest of the path. DNS owns that machinery.
• A connection is a four-tuple — source address, source port, destination address, destination port — and a listening socket is a different kernel object from a connected one; ports and sockets owns both.
• RST is an answer and a drop is not. A reset means a machine received your SYN and refused it; silence means nobody told you anything. TCP and UDP owns the flags; this page owns what to do with the difference.
• A packet only moves if a route exists for it, and ICMP is the channel the network uses to report that it could not deliver one — IP addresses, subnets and routing owns the routing table, the TTL trick behind traceroute, and path MTU discovery.
• Every one of these tools is reading kernel state through a syscall, including the ones that look like they are reading a file: /proc/net/tcp is generated by the kernel on the spot, not stored — processes and threads owns that boundary.

core idea

Every network failure you will debug is a question about how far the bytes got, and there are only five places they stop: the name, the route, the port, the handshake, the payload. Ask them in that order and each answer deletes a region of the search space, so that by the third question you are not guessing any more — you are choosing between two remaining possibilities.

The half of this that people skip is the more valuable half. A rung that passes proves a prefix of the path works. A rung that fails eliminates everything above it, and — this is the part worth internalising — the shape of the failure tells you which direction to look. An answer means something is alive out there and it told you something. Silence means your packet is gone and you have learned nothing except that you are now debugging a filter or a route rather than an application.

how it actually works

the ladder

Two properties of that ordering matter more than the individual rungs.

Each rung is tested against the artifact the previous rung produced, not against the original name. Rung 1 hands you an address; rungs 2 and 3 test that address, never the name again. If you keep typing the hostname into every tool, a stale or split-horizon DNS answer will follow you up the whole ladder and every result will be about a machine you did not mean to talk to.

The rungs do not share a fate. Name resolution goes to your resolver’s address, which is usually nowhere near the service you are debugging — that is why dig cheerfully answers on a box whose route to the database is broken. That independence is a feature: it is exactly what lets one question eliminate the others.

rung 1 — does the name resolve, and does it resolve the way your process resolves it

There are two different questions hiding in “does the name resolve”, and the fastest way to lose an hour is to answer the wrong one.

• dig name asks a DNS server directly. It does not read /etc/hosts, does not consult nsswitch.conf, and does not use your process’s search list unless you make it. It answers: what does the zone, or my resolver’s cache, currently say?
• getent hosts name goes through the name service switch — the same path getaddrinfo takes, which is the same path Dns.GetHostAddressesAsync takes underneath. It answers: what will my process actually get?
• nslookup name asks the first question with friendlier and much less precise output — it hides which section an answer came from and whether it was authoritative. It is what you use when dig is not installed; when its account of what happened differs from dig’s, believe dig.

When dig and getent disagree, you have already found the bug, and it is on this box:

  ANNOTATED SCHEMATIC — illustrative, not a transcript. Placeholder names,
  addresses from the documentation ranges.

  $ dig +short db.internal
  203.0.113.20                ← the zone says this

  $ getent hosts db.internal
  192.0.2.44   db.internal    ← but the process will get THIS
        └─ an /etc/hosts entry, or an nsswitch module, is overriding DNS.
           The zone is innocent. Everything above rung 1 has been testing
           the wrong machine.

The failure modes and what each rules out live on the DNS page — NXDOMAIN (a server said the name does not exist, so the transport is fine and it is a typo, a missing record, or a search suffix), SERVFAIL (your resolver could not produce an answer, so ask a different resolver and ask the authoritative server directly), NOERROR with an empty answer (the name exists but not with the type you asked for), and silence (your query or the answer was dropped). What belongs here is the discipline around them:

• Ask from the box that is failing. A name that resolves on your laptop and not in the pod is not a DNS outage; it is two different resolvers with two different views, and that is the normal state of affairs in a cluster.
• Ask twice and watch the TTL count down. A remaining TTL lower than the published one means you were served from a cache, which is the whole explanation for “it works for me and not for him” during a migration.
• Write the address down. From here on, the address is the subject and the name is a convenience.

rung 2 — is that address routable from here

Three tools, three different questions, and only one of them is about your service.

Is there a route at all? This is answered locally, before anything is sent, by the routing table. It is the cheapest check on the page and it is the only one whose failure is instant — with no route, the connect fails without a single packet leaving the machine. On the Linux box these pages were written on, /proc/net/route holds two rows (the hex is little-endian, which is why they need decoding):

  Iface  Destination  Gateway   Mask
  eth0   00000000     010200C0  00000000     → default via 192.0.2.1
  eth0   000200C0     00000000  00FFFFFF     → 192.0.2.0/24 is on-link

  reading it: anything inside 192.0.2.0/24 is reached directly over eth0;
  everything else goes to 192.0.2.1 and becomes that router's problem.

Longest-prefix match and the decoding are IP and routing’s material. What matters at this rung is the shape of the answer: if the destination has no matching route and there is no default route, you get an immediate error rather than a wait, and in .NET it arrives as a SocketException carrying SocketError.NetworkUnreachable. A NetworkUnreachable that comes back with no wait at all was decided by your own routing table before a byte was sent. That is the distinction to hold: a filter set to drop makes you wait, and a filter set to reject answers immediately — but a reject shows up at rung 3 as a refusal, not here as an unreachable.

Is something at that address alive? That is ping, and it asks precisely one thing: does an ICMP echo request to this address produce an echo reply. A reply proves an IP path exists in both directions and that the host’s stack is up. It proves nothing whatsoever about your port, and on an anycast address (one address announced from several places, so the nearest instance answers) or behind a load balancer it may have been answered by a device in front of the thing you care about.

How far do packets get, and through what? That is traceroute, which is not a special protocol: it sends probes with deliberately small TTLs and collects the ICMP Time Exceeded messages each router sends back when the counter hits zero. The mechanism is on IP and routing. The diagnostic rules are:

• A row of stars is not evidence of a break. Plenty of routers decline to generate ICMP or rate-limit it, and the trace happily continues past them.
• The path a plain traceroute takes is not necessarily the path your connection takes. Firewalls and load balancers treat ICMP, UDP and TCP as three different things. traceroute -T -p 5432 (or tcptraceroute) sends TCP SYNs to the real port, and it is the only variant whose answer is about the traffic you actually care about.
• What it genuinely proves is identity: where the packets are going. Discovering that your database traffic leaves through an internet gateway rather than the peering link is the kind of thing this tool is for.

rung 3 — is something listening, and the distinction the whole page turns on

Here is where a failure stops being ambiguous, because there are exactly two ways to fail a TCP connect and they have opposite causes:

Never let the first two collapse into “it didn’t connect”. They are the difference between “your firewall rule is missing” and “your service is bound to the wrong interface”, and no amount of staring at the application log will separate them.

The tools:

• nc -vz 203.0.113.20 5432 opens a TCP connection and closes it. It answers rung 3 and nothing else, and that focus is why it is the right tool: no TLS, no HTTP, no client library, no ambiguity about which layer failed.
• curl -v will do rung 3 for you as a side effect, but it does rungs 4 and 5 as well, which makes it a bad instrument for isolating this one. When you do not know which rung is broken, climb with the narrow tools.
• ss answers “is it listening” from the authoritative side — as netstat -ltnp and lsof -i do, where they are installed — and it must be run on the server. A client can only ever infer.

  ANNOTATED SCHEMATIC — not a captured session. The shape `ss -ltn` prints for
  listening TCP sockets; addresses are placeholders.

  State    Recv-Q  Send-Q   Local Address:Port
  LISTEN   0       4096     0.0.0.0:5432      ← any interface, including ones added later
  LISTEN   0       128      127.0.0.1:5000    ← loopback only; unreachable from off-box
  LISTEN   19      128      0.0.0.0:8080      ← see below

  on a LISTEN row those two columns are not bytes:
    Send-Q = the accept queue's capacity, from listen(backlog)
    Recv-Q = how many completed connections are sitting in it RIGHT NOW,
             waiting for the process to call accept()

A Recv-Q that is persistently non-zero on a listening socket is one of the most under-read signals in production: the kernel has finished handshakes that your process has not picked up. That is not a network problem at all — it is your accept loop being starved, and the client sees it as latency or, once the queue fills, as connections that hang or get reset. The queues themselves are on ports and sockets.

And when the two ends disagree, packet capture is the arbiter. The point of tcpdump is not its output format; it is that you can run it on both ends and compare, which is the only way to separate “we never sent it” from “they never received it”.

  ANNOTATED SCHEMATIC — not a captured session. Timestamps deliberately omitted:
  this page states ordering, never timing.

  $ tcpdump -n -i any 'host 203.0.113.20 and port 5432'

  IP 192.0.2.15.41102 > 203.0.113.20.5432: Flags [S]     ← our SYN left this box
  IP 203.0.113.20.5432 > 192.0.2.15.41102: Flags [R.]    ← reset: refused
                                                            (the [.] is the ACK flag)

  the four outcomes, and what each one means:

    SYN out, SYN-ACK back ......... rung 3 passed; go up
    SYN out, RST back ............. refused. Reachable, nothing listening / rejecting
    SYN out, nothing back ......... dropped. Now capture on the SERVER:
        server sees the SYN  ......... the drop is on the RETURN path
        server sees nothing  ......... the drop is on the way OUT
    SYN never appears here ........ it never left. Local route, local firewall,
                                     or you are connecting somewhere else entirely

That last row catches an embarrassing number of incidents: the connection string points at a name you have not looked at closely, and the packets are going somewhere perfectly reachable and perfectly wrong.

Wireshark is the same evidence with a dissector on top: capture to a file with tcpdump -w capture.pcap on the box that has no GUI, open it where you have one, and it will decode the handshake, the TLS records and the HTTP headers into a tree instead of a flag list. The filter syntaxes are different — tcpdump uses BPF (host X and port Y), Wireshark uses its own display filters (tcp.port == 5432) — and the common mistake is typing one into the other. Use Wireshark when you need to read the content of a conversation; tcpdump on both ends is enough when the question is only whether packets arrived.

rung 4 — does TLS complete

You only reach this rung with a working TCP connection, which means every failure here has already exonerated the network. openssl s_client is the instrument because it does the handshake and then stops, without an HTTP request muddying the result:

  ANNOTATED SCHEMATIC — illustrative, not a transcript.

  $ openssl s_client -connect 203.0.113.20:443 -servername api.example.com

  CONNECTED(00000003)
      └─ rung 3 passed. Everything after this line is about certificates.

  depth=2 CN = Example Root CA
  depth=1 CN = Example Intermediate CA
  depth=0 CN = api.example.com
      └─ the chain the SERVER sent, plus whatever your trust store filled in.
         A chain that stops at depth=0 means the server is not sending its
         intermediate — see below.

  Verify return code: 0 (ok)
      └─ the one line that matters. Non-zero names the reason: expired,
         self-signed, unable to get local issuer certificate, hostname mismatch.

Three things make this rung worth isolating rather than folding into curl:

• -servername is not optional. SNI is the name the client puts in the handshake so a server hosting several sites knows which certificate to present. Omit it and you may get a default certificate for an unrelated name, and then debug a mismatch you created yourself.
• A missing intermediate certificate fails in some clients and not others. The server is supposed to send the whole chain except the root. When it does not, a client that already has the intermediate cached, or that fetches it from the issuer URL in the certificate, succeeds anyway — and a freshly built container does neither. “It works in my browser and fails in the pod” is very often exactly this, and it is a server misconfiguration regardless of who tolerates it.
• The trust store is per-box, and containers are where it goes missing. A slim base image without the CA bundle installed fails every TLS connection with an issuer error while the same binary works on your laptop. That is a rung-4 failure with a filesystem cause.

In .NET, a handshake failure arrives as an AuthenticationException (usually The remote certificate is invalid according to the validation procedure) wrapped in an HttpRequestException. Read the inner exception; the outer message is generic. What this rung costs — the round trips a cold TLS connection adds before your first byte goes out, and why connection reuse is the fix — belongs to the network path, which is the applied page above this section.

rung 5 — is the payload what you think

The bytes arrived, something understood them, and something answered. Everything from here is application-shaped, and curl -v is the right tool now precisely because it is chatty:

  ANNOTATED SCHEMATIC — illustrative, not a transcript.

  $ curl -v https://api.example.com/orders/42

  *   Trying 203.0.113.20:443...          ← rung 2/3: the address it actually chose
  * Connected to api.example.com          ← rung 3 passed
  * SSL connection using TLSv1.3          ← rung 4 passed
  > GET /orders/42 HTTP/1.1               ← lines starting with ">" are what WE sent
  > Host: api.example.com                 ← the header a proxy routes on
  > Accept: */*
  >
  < HTTP/1.1 502 Bad Gateway              ← lines starting with "<" are the RESPONSE
  < Server: some-proxy
  <

Read the three prefixes and the whole picture falls out: * is curl talking about the connection, > is the request, < is the response. And the status code is the last elimination on the ladder, because a status code is an answer — it proves rungs 1 through 4 all worked, to whichever machine produced it.

Which is why the proxy statuses deserve to be told apart — the protocols on top has the full status anatomy; what matters here is which rung each one has already proved:

• 502 — the proxy reached its upstream and got something it could not use: a refused connection, a reset, or a malformed response. The proxy is fine. Your service, or the hop between them, is not. Start a new ladder from the proxy to the backend.
• 504 — the proxy reached its upstream and gave up waiting. Different diagnosis: the backend accepted the connection and did not answer, which usually means the backend is alive and stuck, not absent.
• 503 — usually the proxy itself, saying it has no healthy backend to try or is shedding load. That is a health-check question, not a connectivity one.

Two payload-rung failures that masquerade as network problems, both worth recognising on sight: a request that reaches the wrong backend because a proxy routes on the Host header and the one you sent does not match the route you meant; and a response that is correct but truncated, which is a framing question — Content-Length versus chunked transfer — and belongs to the protocols on top.

the decision tree

The database case, because it is the one everybody has:

  "the service cannot reach the database"
  run every step FROM THE BOX THAT IS FAILING — a laptop is a different machine
  with a different resolver, a different route table and different firewall rules

  1. does the name resolve?
     |  getent hosts db.internal     (what the process will get: hosts file + NSS)
     |  dig db.internal              (what DNS says: ignores hosts file + NSS)
     |
     +-- no answer / NXDOMAIN ......... STOP. Not a firewall. Wrong name, wrong
     |                                  search suffix, wrong resolver, or a
     |                                  cluster-internal name asked from outside
     +-- the two disagree ............. /etc/hosts or nsswitch is overriding DNS
     +-- answers -> KEEP THE ADDRESS. Everything below tests the address
                                             |
  2. is the address routable from here?      v
     |  route table first (instant answer), then ping, then traceroute -T -p 5432
     |
     +-- no route / network unreachable, and it failed INSTANTLY
     |         ......................... local routing. No packet ever left.
     |                                   Wrong subnet, missing route, dead gateway
     +-- ping fails .................... learn nothing; ICMP is probably filtered.
     |                                   Go to 3 anyway
     +-- ping works or is inconclusive -> 3
                                             |
  3. is something listening on 5432?         v
     |  nc -vz 203.0.113.20 5432
     |
     +-- CONNECTION REFUSED (an RST) ... something answered and said no.
     |     |                             The network is FINE. Two candidates:
     |     +-- on the server: ss -ltn shows 127.0.0.1:5432 -> bound to loopback
     |     +-- ss shows 0.0.0.0:5432 --> a filter is REJECTING, not dropping
     |
     +-- TIMEOUT (silence) ............ dropped. The network is NOT fine.
     |     |                             tcpdump on both ends to place the drop:
     |     +-- server never sees the SYN ....... outbound path: security group,
     |     |                                     network ACL, route, wrong address
     |     +-- server sees SYN, sends SYN-ACK .. RETURN path is blocked.
     |     |                                     Asymmetric rule, or asymmetric route
     |     +-- SYN never leaves our box ........ local firewall or local route
     |
     +-- CONNECTED ---------------------------> 4
                                             |
  4. does TLS complete? (if the link is TLS)  v
     |  openssl s_client -connect 203.0.113.20:5432 -starttls postgres
     |     (a database that UPGRADES a plain connection needs -starttls;
     |      a port that is TLS from the first byte, like 443, needs
     |      -servername NAME instead, to send SNI)
     |
     +-- verify code non-zero ......... certificate, chain, or trust store.
     |                                  The network already worked
     +-- ok ---------------------------------> 5
                                             |
  5. is the payload what you think?           v
     |  the driver's own log, or curl -v for an HTTP dependency
     |
     +-- auth error ................... you are talking to the right thing.
     |                                  Credentials, database name, TLS mode
     +-- 502 from a proxy ............. restart this whole tree from the PROXY
     |                                  to the backend. Yours succeeded
     +-- a correct answer ............. not a connectivity problem at all.
                                        Pool exhaustion, queueing, or the far end

The tree is worth memorising for one structural reason: every branch that says “the network is fine” was produced by a failure, not by a success. Refused is a better outcome than connected- and-hanging, because it eliminates more.

the same ladder from inside the process

The tools above run in a shell. The incident happens in your service, which has its own view of the same five rungs — and it is a better view, because it is the process’s own resolver, route table, and socket. The mapping is exact:

That last row is the trap, and it is worth being blunt about: when HttpClient.Timeout fires you get a cancellation, not a SocketException, and in modern .NET the inner exception is a TimeoutException rather than anything about the network. The diagnostic information the socket would have given you never arrives, because you cancelled before the socket had an answer. Set the client timeout above the connect timeout deliberately, or you will convert every informative failure into an uninformative one — which is the same trap that retries and timeouts covers from the design side.

So: log the SocketError, not the message. The message is localised, the enum is not.

using System.Net.Http;
using System.Net.Sockets;
using System.Security.Authentication;
using Microsoft.Extensions.Logging;

// `client`, `logger` and `ct` come from the surrounding class; this is the
// catch block, not the call site.
try
{
    using var response = await client.GetAsync("https://api.example.com/orders/42", ct);
    response.EnsureSuccessStatusCode();
}
catch (HttpRequestException ex)
{
    // The outer message is generic. The rung lives in the inner exception —
    // and in .NET 8 and later, ex.HttpRequestError names the same thing directly.
    string rung = ex.InnerException switch
    {
        SocketException { SocketError: SocketError.HostNotFound }
            => "rung 1 — the name never resolved",
        SocketException { SocketError: SocketError.NetworkUnreachable or SocketError.HostUnreachable }
            => "rung 2 — no route: nothing was sent",
        SocketException { SocketError: SocketError.ConnectionRefused }
            => "rung 3 — refused: reachable, nothing listening",
        SocketException { SocketError: SocketError.TimedOut }
            => "rung 3 — silence: dropped, suspect a filter",
        AuthenticationException
            => "rung 4 — TCP connected, TLS did not",
        _ => "rung 5 — the connection worked; the answer did not",
    };

    logger.LogError(ex, "call to {Host} failed at {Rung}", "api.example.com", rung);
    throw;
}
catch (TaskCanceledException ex) when (!ct.IsCancellationRequested)
{
    // Our own HttpClient.Timeout fired. This tells us nothing about the network.
    logger.LogError(ex, "we gave up before the socket had an answer");
    throw;
}

And the other direction — the questions ss answers, asked from inside the process that is supposed to be listening. This is the fastest way to settle “am I bound where I think I am” in a container that has no networking tools installed at all:

using System.Net;
using System.Net.NetworkInformation;

IPGlobalProperties props = IPGlobalProperties.GetIPGlobalProperties();

// On Linux this reads /proc/net/tcp; on Windows it is an IP Helper call.
foreach (IPEndPoint listener in props.GetActiveTcpListeners().OrderBy(e => e.Port))
{
    string reach = listener.Address switch
    {
        var a when a.Equals(IPAddress.Any) || a.Equals(IPAddress.IPv6Any)
            => "wildcard: any interface, including ones added later",
        var a when IPAddress.IsLoopback(a)
            => "loopback only: nothing off this host can ever reach it",
        var a => $"only packets addressed to {a}",
    };

    Console.WriteLine($"LISTEN {listener,-24} {reach}");
}

// The other half of the census: who we are connected to, and in what state.
// CloseWait climbing and never falling is a socket your code never disposed.
foreach (TcpConnectionInformation c in props.GetActiveTcpConnections())
{
    Console.WriteLine($"{c.State,-12} {c.LocalEndPoint} -- {c.RemoteEndPoint}");
}

Exposed on an admin endpoint, that is a rung-3 answer you can get during an incident without kubectl exec into an image that does not contain ss.

the four container cases you will actually hit

These are not exotic. They are the failures that account for most of the “the network is broken” tickets that turn out not to be.

Bound to 127.0.0.1 instead of 0.0.0.0. The service starts, logs that it is listening, health checks from inside the container pass, and everything from outside is refused. The refusal is the diagnosis: the packet arrived and a live kernel found no socket matching that destination address. In ASP.NET Core the knob is the URL — http://localhost:5000 is loopback, http://0.0.0.0:8080 or http://+:8080 is the wildcard — and the default is right for a laptop and wrong for every container. Ports and sockets has the matching rules.

A service DNS name that only resolves inside the cluster. db.default.svc.cluster.local is served by the cluster’s own resolver, which your laptop is not configured to use, so the same name gives an address in one place and NXDOMAIN in another. Neither answer is wrong. This is why rung 1 must be climbed from the failing box: the search list, the resolver and the ndots threshold in a pod’s resolv.conf are all different from yours, and the consequences of that are on the DNS page.

A security group that drops rather than rejects. Cloud security groups are default-deny by dropping, so the symptom is a timeout, never a refusal — and a refusal therefore rules a security group out, which is the single most useful inference in the whole section. Two extra things to check before you blame the rule you can see: security groups are usually stateful (the reply to an allowed outbound connection is allowed back automatically) while network ACLs are not, so an asymmetric rule blocks the SYN-ACK and looks exactly like a blocked SYN from the client; and the rule that matters may be on the other account’s side of a peering link.

MTU black-holing. The connection establishes, small requests work perfectly, and anything with a large body hangs. The fingerprint is a size threshold rather than an endpoint or a protocol one, and that fingerprint is diagnostic on its own: nothing else in this section behaves that way. The cause is a link in the path with a smaller MTU than the sender assumed, plus somebody dropping the ICMP Fragmentation Needed messages that would have told the sender to shrink. On the Linux box these pages were written on, eth0 has an MTU of 1400 rather than the Ethernet default of 1500 — a tunnelled interface, exactly the situation that creates the mismatch — while lo is 65536, which is why this never reproduces over loopback. Overlay networks in Kubernetes are the same shape. The mechanism is on the link layer and the discovery protocol is on IP and routing.

the mental model

  ask in this order; each answer deletes a region

    1  name     dig / getent / Dns.GetHostAddressesAsync   -> produces AN ADDRESS
    2  route    route table, ping, traceroute -T -p PORT   -> can packets leave
    3  port     nc -vz, ss -ltn on the server, tcpdump     -> can we connect
    4  TLS      openssl s_client -servername NAME          -> does the chain verify
    5  payload  curl -v, the driver's log, the status code -> is the answer right

  the two failures that are not the same failure:

    RST     -> something answered and said no      -> the network is FINE
               (nothing listening, or a REJECT rule)
    silence -> your packet is gone, nobody told you -> the network is NOT fine
               (a DROP rule, a routing hole, a wrong address, or the RETURN path)

Three lines to carry:

1. Test the address, not the name, from rung 2 up. The name is a rung-1 artifact and it will lie to you at every rung above if you keep re-resolving it.
2. An answer beats silence, even when the answer is a rejection. Refused eliminates the entire network in one move; a timeout eliminates nothing; and a client-side timeout you configured destroys the evidence before it arrives.
3. Run it from the box that is failing, and when the two ends disagree, capture on both. One vantage point cannot tell “never sent” from “never received”.

why you should care

The incident shape: “the network is broken” that is a bind address. Every rung-3 refusal you receive is the network telling you it works. When a deploy goes out and every caller reports connection refused while the pod’s own readiness probe is green, you have already been handed the answer — the probe runs inside the container and reaches loopback, the callers do not. The metric that moves is connection errors at the caller with a flat error rate inside the service, because your service never sees the requests at all. That asymmetry — errors upstream, silence downstream — is worth recognising as its own signature.

The incident shape: retries hiding the rung. A dependency starts timing out; a retry policy turns each failure into three; the connection pool fills with connections stuck in the handshake; ephemeral ports start running short, and now you are also seeing SocketError.AddressNotAvailable. By the time anyone looks, the original signal — was it a refusal or a timeout — is buried under a self-inflicted second failure. This is why the timeout and retry design on reliability is a diagnostic concern and not only a resilience one: a client timeout shorter than the connect timeout permanently destroys the information that would have told you which rung failed.

The code review you can now do. A HttpClient constructed per request (ephemeral port exhaustion) or held forever with no PooledConnectionLifetime (a pinned DNS answer that outlives a failover). A catch block that logs ex.Message from an HttpRequestException and discards the inner SocketException — that is the enum in the table above, thrown away. A health check that binds loopback while the service binds the wildcard, or the reverse. A connection string with a name in it and a firewall ticket with an address in it, which nobody has checked resolve to the same machine. And a catch (Exception) around an outbound call that reports “database unavailable” for all six of the distinct failures on this page.

What to instrument before the next incident. The SocketError value as a log field, so the refusal-versus-timeout question is answerable from a dashboard rather than from a shell. The listener census from the snippet above, on an admin endpoint. And enough of the request path recorded on both sides of every hop that you can tell, at 3am, which end stopped seeing traffic first — the same both-ends principle as running tcpdump on the server, applied to your logs.

the same idea elsewhere

interview drills

Q. The service can reach the database from my laptop but not from the pod. Walk me through what you check.

• weak answer — “It’s probably a firewall rule, I’d open a ticket for the security group.” Might be right, proves nothing, and it skips the two rungs where the answer usually lives.
• strong answer — First, everything from inside the pod, because the laptop is a different box with a different resolver, route table and rules. Resolve the name there and compare the address with the one my laptop gets — a cluster-internal name or a split-horizon zone explains it outright. If the address matches, connect to the port and read which failure I get: refused means the network is fine and I go look at what the far end bound, a timeout means a drop and I capture on both ends to work out whether it is the outbound path or the return path.
• follow-up — “It’s a timeout and the server sees the SYN.” Then the return path is blocked, so I look for an asymmetric rule — a network ACL rather than a stateful security group, or asymmetric routing — not for anything on the outbound side.

Q. Connection refused versus connection timeout. Why do you care which one you got?

• weak answer — “Both mean the service is down.” They mean opposite things and lead to opposite investigations.
• strong answer — Refused means an RST came back: my packet reached a live host, its kernel looked for a socket matching that four-tuple, found none, and answered. The address is right, the route works both ways, and no filter is dropping anything — the problem is a listener that is not there or is bound to the wrong interface. A timeout means silence, so I have learned almost nothing: a drop rule, a routing hole, a wrong address, or a blocked return path all look the same. Refused is the more useful failure, because it eliminates the whole network.
• follow-up — “How do you tell the drop directions apart?” Capture on both ends. Server never sees the SYN: outbound. Server sees it and answers: the return path.

Q. ping to the host fails. What have you learned?

• weak answer — “The host is down.” Nothing supports that.
• strong answer — Essentially nothing. ICMP is a separate protocol with its own filtering, and on the major clouds inbound echo is not permitted by default, so a service that is perfectly healthy routinely fails to answer a ping. I would go straight to a TCP connect on the actual port, which is the traffic I care about. The asymmetry is what makes ping useful at all: a reply proves an IP path exists in both directions, so a success is informative and a failure is not.
• follow-up — “So blocking ICMP is harmless?” No — it breaks path MTU discovery. Drop the Fragmentation Needed messages and senders never learn to send smaller packets, which is the black hole where small requests succeed and large ones hang.

Q. Your gateway returns 502. Where do you look?

• weak answer — “The gateway is broken.” The gateway is the one component that just proved it works.
• strong answer — A 502 is a complete request and response, so every rung — DNS, route, port, TLS, HTTP — passed between me and the proxy. What failed is the proxy’s own connection to its upstream, and it has its own five rungs. From the proxy’s network position I would ask the same questions about the backend: does the upstream name resolve there, does a connect to the backend port succeed, and does it get refused or time out. I would also check whether the proxy is routing on a Host header that matches the route I think it does.
• follow-up — “And if it were a 504 instead?” Then the backend accepted the connection and did not answer in time — it is alive and stuck, not absent, which points at the backend’s own dependencies rather than at connectivity.

Q. Small requests to a service succeed and large ones hang, on a connection that establishes fine. What is happening?

• weak answer — “The service is slow with big payloads.” That would produce a slow response, not a hang, and it would not have a sharp size threshold.
• strong answer — That fingerprint is MTU black-holing. A link somewhere in the path carries a smaller MTU than the sender assumed — a tunnel or an overlay network — and the routers that would have said “too big, here is my MTU” are having their ICMP dropped. Small segments fit and are delivered; anything past the threshold is discarded silently and retransmitted forever. I would check the interface MTUs on both ends, and confirm the size threshold by sending payloads either side of it.
• follow-up — “Why does it never reproduce locally?” Loopback has an enormous MTU — 65536 on the box these pages were written on — so nothing on the local path is ever too big.

Q. How do you prove a packet actually left your machine?

• weak answer — “The application log says it tried to connect.” That says a connect call was made, which is not the same claim.
• strong answer — Capture on the interface and look for the SYN. If it never appears, nothing left: a local route, a local firewall, or a destination that is not what I think it is. If it appears and nothing comes back, capture on the server and see whether it arrived — that is the only way to place the drop on the outbound path versus the return path. A route-table check comes first, though, because a missing route fails immediately and locally, and in .NET that shows up as SocketError.NetworkUnreachable rather than as a timeout.
• follow-up — “And if you cannot install tcpdump in the container?” The process can answer part of it itself: IPGlobalProperties will list the listeners and connections the kernel has, which settles what is bound and what states connections are in without any tooling in the image.