Lesson 9.11 · TCP, TLS & HTTP · 22 min read

Reading tcpdump

Q: How would you capture the traffic between a server and its database on port 5432, and what would you look for?

- -nn no name or port lookups, -i any every interface (and it shows which NIC and direction), -c stop after N packets, then a filter. Add -w db.pcap to save it and read it later with -r (no root needed to read). Then read the flags: - Healthy: [S], [S.], [.], then [P.] lines with data. - Refused: [S] answered by [R.]. - Dropped: the same [S] with the same seq again and again, 1, 2, 4 s apart. - Reset mid-flight: data, then an [R] from the other side. The strongest evidence is a capture on both ends: "the SYN left here and never arrived there" localises a drop to the path in between.

In plain words

Imagine standing next to a mailbox with a notebook, writing down every envelope that goes in or comes out: who sent it, who it is for, what kind of envelope it is. You don't open the letters, you just record them. Later, you can prove "the letter left at 10:40 and no reply ever came".

tcpdump is that notebook for network packets. You tell it which door to watch (-i enp0s1), which envelopes you care about (a filter like host 10.0.3.12 and port 5432), and it prints one line per packet with its flags: [S] for "can we talk", [R] for "go away". -w saves the notebook to a file you can read later with -r.

Why this matters

"The network drops our calls" - "our firewall is fine". Two teams, two opinions. A packet capture ends the argument: it is a record of exactly which packets left and arrived, with timestamps. tcpdump makes one, and reading it is a skill you will use in every chapter from here on.

What you need to know already: the flags and the handshake (9.1), the three failure shapes (9.1), interfaces like enp0s1 and lo (1.13, Chapter 8), and running a command in the background with & and waiting for it with wait (3.10).

The minimum you need

sudo tcpdump -i enp0s1 -nn host 10.0.3.20 and port 443
             |         |   |
             |         |   filter (BPF): what to capture
             |         no name or port lookups (always use this)
             interface: enp0s1, lo, any

-i picks the interface (network card) to listen on; any = all of them.
-nn: the first n stops turning IPs into names, the second stops turning port numbers into service names. You want to see 10.0.3.20.443, not guesses.
The rest is a capture filter, written in BPF (Berkeley Packet Filter), a tiny language the kernel runs on every packet to decide whether to copy it.

Capturing needs root. Reading a saved capture does not:

$ tcpdump -nn -i any port 53
tcpdump: any: You don't have permission to perform this capture on that device
(socket: Operation not permitted)
$ sudo tcpdump -nn -i any -c 4 -w /tmp/dns.pcap port 53 & sleep 1; dig +short example.com; wait
$ tcpdump -nn -r /tmp/dns.pcap
reading from file /tmp/dns.pcap, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144
...

The second line starts the capture in the background (&), waits a second for it to start, makes a DNS query to capture, then waits for tcpdump to finish.

-c N stops after N packets - always use it, or a busy interface floods your terminal. -w FILE saves the raw packets to a pcap file (the standard capture file format; Wireshark, a desktop program for browsing captures, opens it too); -r FILE reads one back.

Filters

host 10.0.3.12                    either direction
src host 10.64.0.2             from us
dst port 443                      to port 443
port 53                           DNS, both directions
tcp / udp / icmp / arp            protocol
net 10.0.0.0/8                    a range
host 10.0.3.12 and port 5432      combine with and / or / not
not port 22                       exclude your own SSH session
'tcp[tcpflags] & (tcp-syn|tcp-rst) != 0'   only SYNs and RSTs

not port 22 matters when you capture over SSH without a filter: every line of output is itself SSH traffic, which is captured, which prints more output. The last filter reads the flag bits of the TCP header directly - copy it as a recipe for "only SYNs and RSTs"; you do not need to write your own.

Reading a line

10:40:01.101200 IP 10.64.0.2.41822 > 10.0.3.12.5432: Flags [S], seq 771820, win 64240, options [mss 1460,sackOK,TS val 1301 ecr 0,nop,wscale 7], length 0
|               |  |                    |               |           |           |          |                                                     |
time            IP source.port          dest.port       TCP flags   seq         window     TCP options                                           payload bytes

The flags:

[S]    SYN          open
[S.]   SYN-ACK      accept
[.]    ACK          acknowledgement only
[P.]   PUSH+ACK     data
[F.]   FIN+ACK      close
[R]    RST          reset
[R.]   RST+ACK      reset (typical "port closed" answer)

-i any adds two columns - the interface and the direction:

10:40:01.101200 enp0s1 Out IP 10.64.0.2.41822 > 10.0.3.12.5432: Flags [S], ...

which is exactly what you want when the question is "which NIC did it use".

The four shapes

A healthy connection with a request:

IP 10.64.0.2.40117 > 93.184.216.34.80: Flags [S], seq 3812201, ...
IP 93.184.216.34.80 > 10.64.0.2.40117: Flags [S.], seq 190112, ack 3812202, ...
IP 10.64.0.2.40117 > 93.184.216.34.80: Flags [.], ack 1, win 502, ...
IP 10.64.0.2.40117 > 93.184.216.34.80: Flags [P.], seq 1:79, ack 1, win 502, length 78: HTTP: GET / HTTP/1.1
IP 93.184.216.34.80 > 10.64.0.2.40117: Flags [P.], seq 1:1257, ack 79, win 509, length 1256: HTTP: HTTP/1.1 200 OK
IP 10.64.0.2.40117 > 93.184.216.34.80: Flags [F.], seq 79, ack 1257, ...
IP 93.184.216.34.80 > 10.64.0.2.40117: Flags [F.], seq 1257, ack 80, ...
IP 10.64.0.2.40117 > 93.184.216.34.80: Flags [.], ack 1258, ...

[P.] lines carry data (tcpdump even decodes the first line of HTTP for you). seq 1:79 ... length 78 - bytes 1 to 78 of our stream. The server's ack 79 - "I have everything up to 78". We sent the first FIN, so the TIME-WAIT is ours.

Refused - SYN, RST:

IP 127.0.0.1.51712 > 127.0.1.1.9999: Flags [S], seq 3318821, ...
IP 127.0.1.1.9999 > 127.0.0.1.51712: Flags [R.], seq 0, ack 3318822, win 0, length 0

Dropped - the same SYN, again and again:

10:40:01.101200 IP 10.64.0.2.41822 > 10.0.3.12.5432: Flags [S], seq 771820, ...
10:40:02.123900 IP 10.64.0.2.41822 > 10.0.3.12.5432: Flags [S], seq 771820, ...
10:40:04.171400 IP 10.64.0.2.41822 > 10.0.3.12.5432: Flags [S], seq 771820, ...
10:40:08.267800 IP 10.64.0.2.41822 > 10.0.3.12.5432: Flags [S], seq 771820, ...

Same port and seq; gaps of 1, 2, 4 seconds. If you capture on the server too and see nothing arrive, the drop is between you.

Reset mid-flight - data, then an RST from the other side:

IP 10.64.0.2.52210 > 10.0.3.77.8080: Flags [P.], seq 1:120, ack 1, length 119
IP 10.0.3.77.8080 > 10.64.0.2.52210: Flags [R], seq 1, win 0, length 0

DNS in tcpdump

IP 127.0.0.1.40112 > 127.0.0.53.53: 3321+ A? api.lab. (25)
IP 127.0.0.53.53 > 127.0.0.1.40112: 3321 1/0/0 A 10.0.3.20 (41)
IP 127.0.0.1.40113 > 127.0.0.53.53: 8812+ A? nothing.lab. (29)
IP 127.0.0.53.53 > 127.0.0.1.40113: 8812 NXDomain 0/1/0 (104)

DNS goes over UDP (a simpler protocol than TCP: single packets, no handshake, no connection), so there are no flags here. 3321 the query id (matches question to answer), + recursion desired, A? the question, 1/0/0 answer/authority/additional counts, and NXDomain, ServFail or Refused when it fails - the same codes as dig in 8.18. A query with no answer line is a timeout.

ICMP and ARP

ICMP (the protocol ping uses, and the one routers use to report errors) and ARP (8.14) show up like this:

IP 10.64.0.2 > 140.82.121.3: ICMP echo request, id 1203, seq 1, length 64
IP 140.82.121.3 > 10.64.0.2: ICMP echo reply, id 1203, seq 1, length 64
IP 10.8.0.1 > 10.8.0.2: ICMP 10.20.1.10 unreachable - need to frag (mtu 1400), length 556
ARP, Request who-has 10.64.0.10 tell 10.64.0.2, length 28
ARP, Reply 10.64.0.10 is-at 52:54:00:1f:aa:10, length 28

The "need to frag" line is the key actor in the MTU lesson: a router telling you your packets are too big. When it never arrives, you get a black hole.

Capture on both ends

The strongest statement tcpdump can make is a pair: "the SYN left here at 10:40:01.1012 (client capture) and never arrived there (server capture)". That localises a drop to the path between two machines, which is what you hand to whoever owns the firewalls in between.

Practising without root

You will often be given a capture file from someone else's box. Everything above works on -r files, including filters:

tcpdump -nn -r capture.pcap 'tcp[tcpflags] & (tcp-rst) != 0'     every reset
tcpdump -nn -r capture.pcap host 10.0.3.12 and port 5432         one flow
tcpdump -nn -A -r capture.pcap port 80                           payloads as text

(A flow is all the packets of one connection. -A prints each packet's data as text - useful for plain HTTP, useless for encrypted TLS.)

What you can now do

Capture one conversation with a filter and a packet count, or save it with -w and read it with -r.
Read a line: time, source, destination, flags, seq/ack, length.
Recognise the four shapes: healthy, refused, dropped, reset mid-flight.

Why it helps

tcpdump turns an argument into a fact. When the application team says "the network is dropping our calls" and the network team says "our firewall is fine", a capture on both ends settles it: the SYN left the orders VM at 10:40:01 and never arrived at the database. That is exactly the evidence the firewall owners need.

You will also be handed pcap files from boxes you cannot log in to, and tcpdump -r with a filter for RSTs finds the problem in seconds. Recognising the four shapes (healthy, refused, dropped, reset mid-flight) and DNS lines like NXDomain means you can read a capture in an incident instead of guessing, and it is a common hands-on question in SRE interviews.

Commands in this lesson

tcpdump

FAQ

Why do I need sudo for tcpdump?

Capturing means opening a raw packet socket on an interface, which needs the CAP_NET_RAW and CAP_NET_ADMIN capabilities, so normally root. Without it you get "You don't have permission to perform this capture on that device". Reading a saved capture with tcpdump -r file.pcap needs no privileges at all, which is how you practise and how you analyse files others send you.

Why should I always use -nn?

Without it, tcpdump tries to turn every IP into a hostname and every port into a service name. That makes output harder to match against ss and logs, and the reverse DNS lookups are themselves network traffic that can slow the capture or even appear in it. -n disables name lookups, and the second n disables port-name lookups, so you see 5432 instead of postgresql.

My terminal floods with output when I capture over SSH. Why?

Because every line tcpdump prints is sent to you over SSH, which is traffic on the same interface, which gets captured, which prints another line. Exclude your own session with not port 22, always use a specific filter, and use -c 20 so the capture stops after a fixed number of packets. For anything longer, write to a file with -w and read it afterwards.

Is tcpdump the same as Wireshark?

Same capture format, different front end. tcpdump runs on the server with no GUI and uses the same BPF filter syntax for capturing. Wireshark is a desktop tool with protocol decoders and a friendlier view. The common workflow is to capture on the server with sudo tcpdump -w /tmp/x.pcap, copy the file to your Mac with scp, and open it in Wireshark. Wireshark's display filters are a different syntax from capture filters.

Can tcpdump show me the contents of HTTPS requests?

No. You see the TCP handshake, the TLS ClientHello (including the SNI hostname, which is still sent in clear by default) and then encrypted records. For plain HTTP, -A prints payloads as text. For HTTPS content you need to look where the TLS ends: at the proxy's access log, with curl -v on the client, or application logs. That limitation is the point of TLS.

In an interview Junior

How would you capture the traffic between a server and its database on port 5432, and what would you look for?

sudo tcpdump -nn -i any -c 50 host 10.0.3.12 and port 5432

-nn no name or port lookups, -i any every interface (and it shows which NIC and direction), -c stop after N packets, then a filter. Add -w db.pcap to save it and read it later with -r (no root needed to read).

Then read the flags:

Healthy: [S], [S.], [.], then [P.] lines with data.
Refused: [S] answered by [R.].
Dropped: the same [S] with the same seq again and again, 1, 2, 4 s apart.
Reset mid-flight: data, then an [R] from the other side.

The strongest evidence is a capture on both ends: "the SYN left here and never arrived there" localises a drop to the path in between.

Also asked: How do you prove that a firewall is dropping traffic between two hosts? · Why do you use -nn with tcpdump? · How do you avoid capturing your own SSH session when you run tcpdump over SSH?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.