OnCallReadyStart learning

Linux Lesson 8.11 · Linux Performance & eBPF · 4 min read

Network: throughput, connections and retransmits

In plain words

A network link is a road. Throughput is how many cars pass per second. Errors and drops are cars that crashed or were turned away. Retransmits are letters that got lost and had to be sent again: each one makes that particular delivery late, even when the road is almost empty. A few lost letters out of a hundred is enough to make many customers angry.

Is it the network?

Three questions, three tools:

1. How much traffic? sar -n DEV 1: packets and kB per second per interface.

$ sar -n DEV 1 1
20:00:12        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
20:00:13           lo      0.66      0.66      0.05      0.05      0.00      0.00      0.00      0.00
20:00:13       enp0s1    397.18    496.72    321.03    374.47      0.00      0.00      1.59      0.00

%ifutil needs the link speed; virtio interfaces report none, so it stays 0 - compare rxkB/s and txkB/s with what you know the link and the instance type allow.

2. Errors and drops? sar -n EDEV 1: rxerr/s, rxdrop/s (the kernel had no room for the packet), txdrop/s. Non-zero drops under load mean ring buffers or backlog queues are too small, or the box cannot keep up.

3. Is TCP healthy? sar -n TCP,ETCP 1:

$ sar -n TCP,ETCP 1 1
...
20:00:07     active/s passive/s    iseg/s    oseg/s
20:00:08         4.15      0.20    425.59    436.88

20:00:07     atmptf/s  estres/s retrseg/s isegerr/s   orsts/s
20:00:08         0.00      0.00     35.92      0.00      0.43
ColumnMeaning
active/sconnections this box opened (connect)
passive/sconnections it accepted (accept)
iseg/s, oseg/ssegments received and sent
atmptf/sconnection attempts that failed
estres/sestablished connections reset
retrseg/ssegments retransmitted
orsts/sRSTs sent

Here 36 segments retransmitted for 437 sent: 8%. On a healthy LAN that is well under 1%. Retransmits add a timeout to the request that hit them (200 ms or more for the first one), so p99 goes up while averages barely move. The cause is lost packets: a congested link, a bad NIC or cable, a firewall, or a remote that is overloaded and dropping.

Which connection? The counters are box-wide. tcpretrans-bpfcc (eBPF, later) prints each retransmit with its addresses and ports; ss -ti shows per-connection retrans: counts and the RTT; nstat -az TcpRetransSegs the raw kernel counter.

In an interview: "I check retransmits with sar -n ETCP or nstat. A few percent of segments retransmitted explains tail latency even when bandwidth is nowhere near the limit; then I find which peer with tcpretrans or ss -ti."

Why it helps

Network problems often show up as latency, not as a full link: a lossy path retransmits a few percent of segments and every affected request waits hundreds of milliseconds. Reading interface throughput, error counters and TCP retransmits takes three sar commands, and naming the peer takes one more tool. It is how you decide between "our server", "their server" and "the network in between".

Commands in this lesson

sar

FAQ

How do I see interface throughput?

sar -n DEV 1 prints rxkB/s and txkB/s per interface every second, plus packets per second. Compare it with the link speed. A 1 Gbit/s interface carries at most about 120000 kB/s; far below that, bandwidth is not the problem.

What do the EDEV counters show?

sar -n EDEV 1 shows receive and transmit errors, drops and FIFO overruns per interface. Drops on receive often mean the box could not keep up (a full ring buffer or backlog); errors point at the link or the driver. On a healthy virtual NIC they are zero.

How much retransmission is too much?

Compare retrseg/s from sar -n ETCP with oseg/s from sar -n TCP. Well under 1% is normal on a LAN. A few percent is already enough to raise p99 latency noticeably, because each retransmit waits for a timeout of 200 ms or more before resending.

How do I find which connection retransmits?

sudo tcpretrans-bpfcc prints each retransmit with the local and remote address and port, and its TCP state. ss -ti shows per-connection counters (retrans, rtt) for live sockets. Either way you get the peer, which tells you whose network or server to look at.

What is nstat for?

nstat prints the kernel's SNMP counters, such as TcpRetransSegs and TcpExtListenDrops, as deltas since the last run. It is the quick way to see which TCP counters are moving right now without remembering which sar report holds them.

In an interview Mid

Latency is bad but the network link is nowhere near full. Could it still be the network?

Yes: retransmits. I check retrseg/s with sar -n TCP,ETCP 1 or nstat and compare it with the segments sent; a few percent retransmitted adds a timeout to every affected request, so p99 rises while bandwidth is idle. sar -n EDEV shows drops and errors on the interface, and tcpretrans-bpfcc or ss -ti name the connection and the peer.

Also asked: How do you measure the retransmit rate on a box? · What does a receive drop on an interface usually mean? · How do you find which peer is losing packets?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.