Linux Network Troubleshooting with ss, netstat, tcpdump, and Friends

When a service “is down,” the network is usually blamed first and understood last. I have seen people restart perfectly healthy daemons because they never checked whether the port was listening, whether a firewall was dropping the traffic, or whether the client was even talking to the right IP. Good network troubleshooting is less about memorizing fifty commands and more about asking the next precise question.

This is the toolchain I use on Linux when I need to answer practical questions fast: is the port open, which process owns it, does DNS resolve, can I connect from here, where does the traffic die, and what packets are actually hitting the wire. The old netstat muscle memory still matters, but ss and tcpdump do most of the real work now.

If the problem is really TLS rather than raw connectivity, I usually use certificate chain verification with OpenSSL. But first I confirm the network basics.

Start with the simplest question: is anything listening?

Before I test from another host, I check the server locally. If the process is not listening, upstream network checks do not matter.

ss is the current tool I use first

To show listening TCP and UDP sockets with process names:

sudo ss -tulpn

The flags mean:

• -t TCP
• -u UDP
• -l listening sockets only
• -p show process info
• -n no DNS or service-name resolution

I almost always include -n because waiting for name resolution during troubleshooting is irritating and sometimes misleading.

Example output will show the local address, port, and owning process. That answers a lot quickly:

• Is the service bound to 127.0.0.1 only?
• Is it bound to IPv6 only?
• Is it listening on the expected port?
• Is the wrong process holding the port?

To look for one specific port:

sudo ss -tulpn | grep ':443 '

To filter directly in ss:

sudo ss -ltn '( sport = :443 )'

That form is cleaner when I want less noise.

netstat is legacy, but you still need to recognize it

Many older runbooks still use:

sudo netstat -tulpn

That is fine if net-tools is installed, but I treat netstat as the legacy tool now. On modern systems, ss is the better default: faster, richer, and usually installed already.

Common mistake: the service is listening, but only on localhost

I run into this constantly with databases, web apps, and admin interfaces.

sudo ss -ltnp | grep ':8080 '

If the address is 127.0.0.1:8080, remote clients will never reach it directly. That may be correct behind a reverse proxy, or it may be the root cause.

Get a quick network summary with ss -s

When the server is under load or acting strangely, I like a fast high-level view:

ss -s

This summarizes socket states: established, time-wait, orphaned, closed, and more. I use it to spot patterns like:

• enormous numbers of TIME-WAIT sockets
• lots of established connections during load spikes
• connection churn that suggests abuse or a retry loop

It does not diagnose the problem by itself, but it tells me where to dig next.

Understanding connection states matters more than people think

A port check is useful, but connection state tells a story.

To see established connections to port 443:

sudo ss -tn state established '( sport = :443 )'

To look for SYN backlog symptoms:

sudo ss -tn state syn-recv

To inspect connections stuck in TIME-WAIT:

sudo ss -tn state time-wait | head

I do not panic when I see TIME-WAIT; that is normal after connections close. I do care when I see huge numbers and performance complaints together. That can point to bad keepalive settings, aggressive health checks, or chatty short-lived clients.

Testing connectivity: refused and timed out are not the same failure

People often summarize everything as “the port is blocked,” but the failure mode matters.

Connection refused

Usually means:

• the target host is reachable
• nothing is listening on that port, or
• a firewall is actively rejecting the traffic

Connection timed out

Usually means:

• packets are being dropped somewhere
• a firewall is silently discarding them
• routing is broken
• the remote network path is failing

That distinction changes how I troubleshoot. Refused means I focus on the target host and application binding. Timed out means I investigate firewalling, routing, and path issues.

Using nc and telnet to test whether a TCP port is reachable

I prefer nc (netcat) these days because it is simple and scriptable.

nc -vz example.com 443

Or for an IP:

nc -vz 203.0.113.10 22

The -z flag means zero-I/O mode, just scan for listening daemons. -v gives readable output.

To test multiple ports:

nc -vz example.com 80 443 8443

telnet still works for basic reachability:

telnet example.com 25

But I treat it as a legacy habit. On many minimal systems it is not installed, and nc is usually more useful.

Testing UDP is different

UDP troubleshooting is trickier because no connection handshake exists. You can still use netcat in UDP mode, but lack of an error does not prove success.

nc -vzu 8.8.8.8 53

For UDP-heavy troubleshooting, packet capture becomes much more important.

HTTP debugging: curl -v tells me a lot very quickly

For web problems, I jump to curl -v almost immediately.

curl -v http://example.com/

For HTTPS with full headers only:

curl -vkI https://example.com/

Useful things this shows:

• DNS resolution result
• target IP address
• TCP connect success or failure
• HTTP status line
• redirects
• certificate issues if you omit -k

I use -k only when I intentionally want to ignore certificate trust during testing. It is a troubleshooting shortcut, not a deployment fix. If you need deeper TLS inspection, pair this with OpenSSL certificate commands I actually use.

To force a specific IP while preserving the Host header and SNI name:

curl -v --resolve example.com:443:203.0.113.10 https://example.com/

That is extremely useful behind load balancers and during DNS cutovers.

ping is useful, but its failure does not prove much

I still use ping, but carefully.

ping -c 4 example.com

If it succeeds, great: name resolution and basic IP reachability are working.

If it fails, do not jump to conclusions. ICMP may be blocked by design. Plenty of healthy production systems ignore ping.

What I use ping for in practice:

• checking whether DNS resolves to the expected IP
• quick latency sanity checks
• confirming packet loss trends on a path that allows ICMP

What I do not use it for: proving that a web service itself is healthy.

Path analysis with traceroute and mtr

When traffic disappears somewhere between client and server, I want hop visibility.

traceroute

traceroute example.com

This can show where packets stop advancing, but individual hops may rate-limit or suppress responses. So I never treat one silent hop as definitive proof of failure.

mtr is usually better for live investigation

mtr -rw example.com

The -r report mode and -w wide output make it practical for saved output. mtr gives repeated probes and better packet loss context than a one-shot traceroute.

I trust mtr more than traceroute when a network issue is intermittent.

DNS checks: dig, host, and nslookup

Bad DNS is a common cause of fake network incidents. I want to know exactly what name resolved, from which server, with what TTL.

My default is dig

Basic lookup:

dig example.com

Short answer only:

dig +short example.com

Check a specific record type:

dig example.com A
dig example.com AAAA
dig example.com MX
dig _acme-challenge.example.com TXT

Query a specific resolver:

dig @1.1.1.1 example.com A

Seeing TTL clearly

dig example.com A +noall +answer

That answer section includes TTL. During cutovers, I check whether stale TTL expectations are causing people to hit old IPs.

host and nslookup

These are also fine for quick checks:

host example.com
nslookup example.com

I use them less because dig is more explicit and easier to script.

If you are debugging DNS for certificate issuance, building your own CA with OpenSSL and CSR generation the right way are related background, but DNS itself usually comes down to authoritative answers and propagation reality.

Firewall checks on the server

A listening service does not help if the host firewall drops the traffic.

ufw

On Ubuntu systems using UFW:

sudo ufw status verbose

If I need the full workflow, I fall back to my practical UFW guide, but during an incident I mainly want to know whether the expected port is allowed for the right interface and family.

iptables

For iptables-based systems:

sudo iptables -L -n -v

For NAT rules:

sudo iptables -t nat -L -n -v

Counters matter. If packet counters do not increment on a rule you expected to match, traffic may be missing the host or hitting another rule first.

nft

On newer systems using nftables:

sudo nft list ruleset

I am mentioning it because plenty of modern distributions use nftables under the hood even if old articles still talk mostly about iptables.

Packet capture with tcpdump: where the truth usually is

When arguments start and certainty ends, I open tcpdump.

Capture on an interface

sudo tcpdump -i eth0

On busy hosts, that is too noisy. I filter quickly.

Capture one port

sudo tcpdump -i eth0 port 443

Capture one host and port

sudo tcpdump -i eth0 host 203.0.113.50 and port 443

Capture only TCP SYN packets

sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-syn != 0 and port 443'

That is useful when I want to see incoming connection attempts without all the data packets.

Write to a capture file for later analysis

sudo tcpdump -i eth0 -w tls-incident.pcap port 443

Then open it in Wireshark on a workstation:

wireshark tls-incident.pcap

I prefer saving a filtered capture rather than scrolling endless terminal output during a messy incident.

Reading tcpdump output without overcomplicating it

A normal TCP three-way handshake looks like this conceptually:

• client sends SYN
• server replies SYN,ACK
• client replies ACK

If I see SYN from the client and no reply from the server, I ask:

• did the packet reach the right host?
• is the firewall dropping it?
• is the service bound to the wrong IP?

If I see SYN and immediate RST from the server, that usually points to no listener or active rejection.

If I see the handshake complete but the application still fails, the problem is probably above TCP: TLS, HTTP, app logic, or backend dependency.

Useful display tweaks

Do not resolve names:

sudo tcpdump -n -i eth0 port 443

Show ASCII payload when appropriate:

sudo tcpdump -A -i eth0 port 80

Show packet contents in hex and ASCII:

sudo tcpdump -X -i eth0 port 80

I use -A and -X carefully. They can expose sensitive data. Never collect more than you need.

Common problems and the pattern I look for

Connection refused

I check in this order:

1. Is the process running?
2. Is it listening on the expected port?
3. Is it bound to the correct address?
4. Is a reverse proxy pointing to the right backend?

Commands:

sudo ss -ltnp | grep ':8080 '
systemctl status myservice
curl -v http://127.0.0.1:8080/

Connection timed out

I check:

1. local firewall rules
2. cloud firewall or security group rules
3. routing path with mtr
4. whether packets reach the host with tcpdump

Commands:

sudo ufw status verbose
sudo iptables -L -n -v
mtr -rw example.com
sudo tcpdump -n -i eth0 host 198.51.100.20 and port 443

DNS resolves to the wrong IP

I check the answer and TTL from multiple resolvers:

dig +short example.com
dig @1.1.1.1 example.com +noall +answer
dig @8.8.8.8 example.com +noall +answer

Reverse proxy works locally but not externally

That often means the backend is fine and the issue is firewalling, NAT, or an upstream load balancer. I test from the inside and outside separately.

Security considerations while troubleshooting

Troubleshooting commands can expose more than you expect.

• ss -p shows process details that may be sensitive on shared systems
• tcpdump can capture credentials, cookies, tokens, and application data
• curl -v may print authorization headers if you are not careful
• packet capture files should be handled like sensitive logs

I keep captures short, filtered, and deleted when I am done. If I need long-term retention, I treat them as restricted incident artifacts.

A few small commands that solve real incidents

These are not glamorous, but I use them constantly.

Show the kernel routing decision for a target:

ip route get 1.1.1.1

List interface addresses cleanly:

ip addr show

Check local name service behavior through NSS instead of raw DNS tools:

getent hosts example.com

That last one matters on systems using /etc/hosts, LDAP, or unusual resolver rules. I have seen dig return one answer while the application effectively used another because NSS resolution was the real path.

My practical troubleshooting sequence

When I want to move fast without skipping steps, this is the flow I use:

1. Confirm DNS
2. Confirm listening socket on server
3. Test local connection on server
4. Test remote connection with nc or curl
5. Check host firewall
6. Check path with mtr if needed
7. Use tcpdump to confirm what packets actually arrive

A sample session:

dig +short app.example.com
sudo ss -ltnp | grep ':443 '
curl -vk https://127.0.0.1/
nc -vz app.example.com 443
sudo ufw status verbose
mtr -rw app.example.com
sudo tcpdump -n -i eth0 host 198.51.100.20 and port 443

That sequence is boring, and boring is good. It prevents guesswork.

Conclusion

Good network troubleshooting is mostly disciplined elimination. ss tells me what is listening now. netstat helps me read older procedures without getting lost. nc, curl, and dig tell me whether the service is reachable, speaking, and resolving correctly. mtr shows me where the path gets ugly. tcpdump tells me what really happened when everyone else’s assumptions start to conflict.

If you build the habit of separating refused from timed out, local bind issues from firewall issues, and DNS mistakes from transport problems, you solve incidents faster and with less random restarting. That has been my experience over and over: the right small command at the right stage beats the giant “let’s reboot it” instinct every time.