OnCallReady

Lesson 8.14 · Addressing & DNS · 15 min read

Layer 2: ARP and the neighbour table

In plain words

In a block of flats, letters are addressed by person name, but the postman needs a flat number to put the letter in the right box. So he shouts in the hallway: "Who is Maria?" Maria shouts back: "Flat 12!" He writes that on a notepad so he does not have to shout next time.

On a network, IP addresses are the names and MAC addresses are the flat numbers. Ethernet frames are delivered by MAC. ARP is the shout ("who has 10.64.0.1?") and the reply ("it is at 9e:3d:..."), and ip neigh is the notepad. For a far-away host, you only shout for the gateway's flat, because the gateway carries the letter onward.

Why this matters

ping prints "Destination Host Unreachable" - but from your own address. Or a connection works, breaks, and works again every few minutes. Both are problems on your local wire, below IP addresses. This lesson shows that layer and the one table that explains it.

What you need to know already: 8.11 - routes, on-link vs via a gateway, ping.

Layers, briefly

Networking is described in layers, each built on the one below. You need two of them now:

(The full list, the OSI model, has seven layers; people say "layer 2" and "layer 3" all the time, rarely the others.)

IP addresses do not travel on the wire

On a local segment, machines use Ethernet, and Ethernet addresses its frames (the layer 2 envelope around a packet) to a MAC address: a hardware address burned into each network card, written as six pairs of hex digits like 9e:3d:7a:12:6b:64. So before the kernel can send anything, it needs the MAC of the next hop:

Finding a MAC is ARP (Address Resolution Protocol). Your box shouts to everyone on the segment - a broadcast - "who has 10.64.0.1? tell 10.64.0.2", and the owner replies "10.64.0.1 is at 9e:3d:7a:12:6b:64".

You can watch it with tcpdump, a tool that prints every packet passing an interface (it needs sudo). Flags used here: -nn show numbers, not names; -i enp0s1 capture on that interface; -c 2 stop after two packets; arp is a filter: only ARP.

$ sudo ip neigh flush dev enp0s1; sudo tcpdump -nn -i enp0s1 -c 2 arp & sleep 1; ping -c1 10.64.0.1 >/dev/null; wait
listening on enp0s1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:00:04.000037 ARP, Request who-has 10.64.0.1 tell 10.64.0.2, length 28
20:00:04.000474 ARP, Reply 10.64.0.1 is-at 9e:3d:7a:12:6b:64, length 28

The one-liner: forget the cached MACs (ip neigh flush) so the kernel has to ask again, start the capture in the background (&, from Ch 3), wait a second, give the box a reason to talk to the gateway (ping), then wait for the capture to finish. Each capture line is: time, protocol, what was said.

The neighbour table

The answers are cached in the neighbour table, shown by ip neigh:

$ ip neigh
10.64.0.1 dev enp0s1 lladdr 9e:3d:7a:12:6b:64 REACHABLE
fe80::1 dev enp0s1 lladdr 9e:3d:7a:12:6b:64 router STALE

Columns: the IP, the interface, lladdr (link-layer address = the MAC), and a state. (The fe80::1 line is the same gateway's IPv6 address - IPv6 is the newer, 128-bit version of IP; you can ignore it in this chapter.) The states you will see:

REACHABLE    confirmed recently (seconds). Traffic is flowing.
STALE        valid but not confirmed lately. Still used; re-checked on next use.
DELAY/PROBE  being re-confirmed right now.
INCOMPLETE   a request went out, no answer yet.
FAILED       no answer after the retries. Nothing with that IP is on the segment.
PERMANENT    added by hand (ip neigh add ... nud permanent).

STALE is normal and harmless. You may see guides use arp -n for the same table; it comes from an old package (net-tools) that Ubuntu Server no longer installs:

$ arp -n
Command 'arp' not found, but can be installed with:
sudo apt install net-tools

Learn ip neigh; it is always there.

Destination Host Unreachable, from yourself

Ping a free address on your own subnet (-c 3 = three tries):

$ ping -c 3 10.64.0.50
PING 10.64.0.50 (10.64.0.50) 56(84) bytes of data.
From 10.64.0.2 icmp_seq=1 Destination Host Unreachable
From 10.64.0.2 icmp_seq=2 Destination Host Unreachable
From 10.64.0.2 icmp_seq=3 Destination Host Unreachable

--- 10.64.0.50 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2048ms
pipe 3

$ ip neigh | grep 64.50
10.64.0.50 dev enp0s1 FAILED

(icmp_seq numbers each try. The statistics block counts sent, received and lost.) Look at who says "unreachable": From 10.64.0.2 - your own address. No packet reached anything; your own kernel gave up because nobody answered ARP. The neighbour table confirms it: FAILED.

Contrast a far-away host that does not answer:

$ ping -c 3 10.0.3.12
PING 10.0.3.12 (10.0.3.12) 56(84) bytes of data.

--- 10.0.3.12 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2040ms

Silence. The gateway took the packets (its ARP entry is fine) and somewhere beyond it they were dropped - or the host is up and simply does not answer ping. No reply to ping proves nothing about a remote host: many hosts and firewalls drop ICMP on purpose. Test the service you actually care about instead - with nc, which you will use in the mission after this lesson.

So there are three shapes, and each points somewhere different:

"From <your IP> ... Destination Host Unreachable"   on-link host not answering
                                                    ARP: down, on another
                                                    segment, wrong subnet mask
"From <router IP> ... Destination Host Unreachable" a router further along has
                                                    no way to deliver it
silence                                              dropped somewhere, or ping
                                                    is filtered

(A VLAN - virtual LAN - splits one physical network into separate segments; a machine "on the wrong VLAN" is plugged in but on a different segment.)

The wrong-mask bug

A host configured 10.0.3.40/16 instead of /24 believes everything in 10.0.x.x is on its own wire. It ARPs for 10.0.9.5 instead of sending it to the gateway, nobody answers, and exactly the addresses outside the real /24 are unreachable - with "Destination Host Unreachable" from itself. ip route shows it at once: the kernel route says 10.0.0.0/16 dev ... where you expected 10.0.3.0/24.

Duplicate IPs

Two machines claiming one address answer ARP in turn, and the neighbour entry flips between two MACs. Connections to that IP work, then break, then work. If ip neigh shows a different MAC every few minutes for the same IP, stop debugging the application. arping -D (package iputils-arping) checks for a duplicate before you configure an address.

Later (Ch 23): cloud networks have no real layer 2 - the provider answers every ARP itself - so tricks built on ARP do not work there.

What you can now do

Why it helps

This is the layer where a whole class of confusing incidents lives. "Destination Host Unreachable" from your own IP means nobody answered ARP on your segment: the host is down, on the wrong VLAN, or your subnet mask is wrong. A connection that works, breaks and works again every few minutes is often a duplicate IP, visible as the MAC flipping in ip neigh. Recognising these saves you from debugging the application.

It also teaches you how little a failed ping means. A host that answers ARP but not ping is up and filtering; a host that does not answer ARP is not there at all. Knowing which one you are looking at decides whether you call the host's owner or keep looking at your own box.

Commands in this lesson

ip arp ping

FAQ

Is ARP used for addresses outside my subnet?

Not for the destination itself. If the route has a via, the kernel ARPs for the gateway's MAC and sends the frame there, while the IP header still carries the far destination. If the route is on-link (no via), it ARPs for the destination directly. That is why a wrong subnet mask breaks things: the host thinks a remote address is on-link and ARPs for it.

Is STALE in ip neigh a problem?

No. STALE means the entry is still valid but has not been confirmed recently. It is still used, and the kernel re-confirms it on the next use (DELAY, PROBE, then REACHABLE). The states to worry about are INCOMPLETE (a request went out with no answer yet) and FAILED (no answer after retries: nothing with that IP is on the segment).

Why is arp not found on Ubuntu Server?

arp, ifconfig, route and netstat belong to net-tools, which Ubuntu Server no longer installs by default. You can install it, but the modern equivalents are already there: ip neigh for the neighbour table, ip a for addresses, ip route for routes and ss for sockets. Learn those, because they are what you will find on minimal systems too.

If ping gets no reply, is the host down?

Not necessarily. Many hosts and firewalls drop ICMP, ping's protocol, on purpose. Silence only means the echo reply did not come back. Test the port you actually care about with nc -zv host port. The exception is "Destination Host Unreachable" from your own IP, which does mean nothing answered ARP on your segment.

Does IPv6 use ARP?

No, IPv6 uses Neighbour Discovery (NDP), which runs over ICMPv6 instead of broadcast. The kernel keeps both in the same neighbour table, which is why ip neigh shows fe80::1 ... router next to the IPv4 entries. The states (REACHABLE, STALE, FAILED) and the troubleshooting logic are the same.

In an interview Junior

What is ARP and when is it used?

On the local segment, frames are addressed to MAC addresses, not IPs. ARP (Address Resolution Protocol) finds the MAC for an IP: the box broadcasts "who has 10.64.0.1? tell 10.64.0.2" and the owner replies with its MAC.

Answers are cached in the neighbour table: ip neigh shows each IP, its lladdr (MAC) and a state - REACHABLE, STALE (normal), INCOMPLETE, FAILED.

Why it matters: ping saying "Destination Host Unreachable" from your own address means nobody answered ARP (host down, wrong segment, or a wrong subnet mask). A MAC that keeps flipping for one IP in ip neigh means a duplicate IP.

Also asked: What is the difference between layer 2 and layer 3? · What does a FAILED entry in ip neigh tell you? · Why does a ping that gets no reply not prove a remote host is down?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.