Addressing & DNS: interview questions

What happens on the box, network-wise, when you run curl http://api.lab/? Junior

Four steps, each with its own check:

1. Name - curl calls getaddrinfo(). glibc follows /etc/nsswitch.conf (hosts: files dns): /etc/hosts first, then the server in /etc/resolv.conf, which on Ubuntu is systemd-resolved's stub at 127.0.0.53. resolved answers from its cache or forwards upstream. Check: getent hosts api.lab.
2. Route - the kernel picks the route by longest prefix match: interface, next hop, source address. Check: ip route get IP.
3. Neighbour - it needs the MAC of the next hop (the gateway, or the host itself if on-link), so it asks with ARP and caches the answer. Check: ip neigh.
4. Connection - TCP sends a SYN to port 80, the handshake completes, curl sends the request. Check: nc -zv IP 80, curl -v.

Every step caches something, so "I changed it and nothing happened" is usually a cache.

Also asked: What is the difference between a private and a public IP address? · How does DNS resolution work, from the program to the authoritative server? · What is a default gateway, and what happens if a box has none?

What are the network address, broadcast address and usable range of 10.0.3.200/26? Junior

/26 means the first 26 bits are the network part; the remaining 6 are host bits.

• The split falls in the last octet: the mask is 255.255.255.192 (192 = 11000000).
• Network address = address AND mask: 200 is 11001000, keep the top two bits -> 11000000 = 192. So 10.0.3.192.
• Broadcast address = all host bits set to 1: 10.0.3.255.
• Usable hosts: .193 to .254, 2^6 - 2 = 62 of them (the network and broadcast addresses never go on a machine).

On a real box I would confirm with ipcalc 10.0.3.200/26, which prints the same Network, Broadcast, HostMin/HostMax and Hosts/Net lines.

Also asked: What is the difference between a netmask and a prefix length? · What does 127.0.0.1 mean, and what does 0.0.0.0 mean as a listen address? · How many bits is an IPv4 address, and what is an octet?

Learn it: 8.1 An address is a 32-bit number

How many usable hosts are in a /27, and how do you work it out without a calculator? Junior

addresses = 2^(32 - prefix), usable = addresses - 2 (the network and broadcast addresses are not usable).

/27: 2^5 = 32 addresses, 30 usable.

The shortcut is that each bit off the prefix doubles the block: /24 = 256, /25 = 128, /26 = 64, /27 = 32, /28 = 16. To find the range an address is in, use the block-size method: block = 256 minus the mask octet (224 for /27, block 32), the network is the largest multiple of 32 at or below the octet, the broadcast is network + 31. So 192.168.10.77/27 is .64 to .95, hosts .65 to .94.

Also asked: Which IPv4 ranges are private (RFC 1918)? · Is 172.31.9.9 a private address? How do you know? · What is a /32 used for?

Learn it: 8.3 CIDR without a calculator

Two networks you need to connect use overlapping address ranges. Why is that a problem, and what can you do? Junior

Routing needs every destination to be unique. If both sides use 10.0.1.5, a packet to 10.0.1.5 has two valid destinations, so the two networks cannot be joined while they overlap (cloud providers refuse the connection outright).

Options:

• NAT in the middle: a box rewrites addresses as packets pass, so each side sees unique ones. Works for a handful of connections, miserable to run.
• Renumbering one side: the real fix - new ranges, move every machine, update every firewall rule, route, allow-list and DNS name.

That is why the address space is planned once, for the whole organisation, before anything is built, with each site or environment getting one aligned, power-of-two block (so it can be summarised in one route).

Also asked: How many /26 subnets fit in a /22? · Is 10.0.6.0/22 a valid network? Why or why not? · How would you split 10.0.0.0/16 into four equal subnets?

Learn it: 8.6 Carving an address space: alignment, overlap, and plans you cannot undo

Someone asks for a subnet for a cluster of 50 machines, each running up to 30 apps that get their own address. What size do you give them? Junior

First, state the assumptions: flat design (app addresses come from the same subnet), surge of 1 node during upgrades, no growth.

• Formula: addresses = (nodes + surge) + (nodes + surge) x apps_per_node
• (50 + 1) + 51 x 30 = 51 + 1530 = 1581 addresses
• The cloud reserves five addresses per subnet, so usable = 2^(32 - prefix) - 5.
• /22 = 1024 - 5 = 1019: too small. /21 = 2048 - 5 = 2043: fits. Answer: /21.

Then mention the traps: a subnet in use cannot be resized, so size for growth; load balancer front-ends take addresses too; and "how many apps per node, at most?" changes the answer the most. With an overlay design the subnet only needs the nodes, so a /26 or /25 would do.

Also asked: How many usable addresses does a /24 have on a cloud subnet, and why not 254? · Why can you not just make a subnet bigger later? · What is the difference between a flat and an overlay design for app addresses?

Learn it: 8.9 Cloud subnets and sizing a cluster of machines

Explain this line: default via 10.64.0.1 dev enp0s1 proto dhcp src 10.64.0.2 metric 100 Junior

It is a route from ip route:

• default = 0.0.0.0/0, matches every address: the route of last resort.
• via 10.64.0.1 - hand the packet to this gateway, the next hop.
• dev enp0s1 - leave through this interface.
• proto dhcp - DHCP added it (kernel = automatic from an address, static = from config).
• src 10.64.0.2 - the source address put on the packets.
• metric 100 - tie-breaker between equal routes; lower wins.

When several routes match, longest prefix match decides: the most specific route wins, and default loses to everything. I would not guess: ip route get IP prints the route the kernel actually picks. A route added with ip route add is gone after a reboot; the permanent one goes in netplan.

Also asked: What is the difference between "Network is unreachable" and a timeout? · How do you add a static route, and how do you make it survive a reboot? · What does longest prefix match mean?

Learn it: 8.11 The routing table: which way does a packet go

What is ARP and when is it used? Junior

On the local segment, frames are addressed to MAC addresses, not IPs. ARP (Address Resolution Protocol) finds the MAC for an IP: the box broadcasts "who has 10.64.0.1? tell 10.64.0.2" and the owner replies with its MAC.

• On-link destination (no via): it ARPs for the destination itself.
• Via a gateway: it ARPs for the gateway; the IP destination stays the far machine.

Answers are cached in the neighbour table: ip neigh shows each IP, its lladdr (MAC) and a state - REACHABLE, STALE (normal), INCOMPLETE, FAILED.

Why it matters: ping saying "Destination Host Unreachable" from your own address means nobody answered ARP (host down, wrong segment, or a wrong subnet mask). A MAC that keeps flipping for one IP in ip neigh means a duplicate IP.

Also asked: What is the difference between layer 2 and layer 3? · What does a FAILED entry in ip neigh tell you? · Why does a ping that gets no reply not prove a remote host is down?

Learn it: 8.14 Layer 2: ARP and the neighbour table

A service cannot resolve a hostname, but dig resolves it fine. What do you check? Junior

dig asks DNS only. Programs go through glibc's getaddrinfo(), which follows /etc/nsswitch.conf (hosts: files dns): /etc/hosts first, then the server in /etc/resolv.conf. So they can disagree.

1. getent hosts NAME - resolves exactly like the program does. Run it first.
2. /etc/hosts - a stale or wrong entry there wins over DNS for every program.
3. /etc/nsswitch.conf - the order of files and dns.
4. ls -l /etc/resolv.conf - on Ubuntu it should be a symlink to systemd-resolved's stub file (nameserver 127.0.0.53); resolvectl status says resolv.conf mode: stub, foreign if someone replaced it.
5. Caches: resolved's (sudo resolvectl flush-caches), and the program's own (nginx resolves its config names once at startup; Java keeps answers for 30 s).

When getent and dig @the-real-server disagree, the problem is on this box, not in DNS.

Also asked: What is /etc/hosts, and does it win over DNS? · What is 127.0.0.53 on an Ubuntu server? · What is the difference between a recursive resolver and an authoritative server?

Learn it: 8.16 How a name becomes an address on Ubuntu

What is the difference between NXDOMAIN, SERVFAIL and REFUSED? Junior

They are the status: in the header of dig output - the first thing to read:

• NOERROR - the server answered. The ANSWER section can still be empty: NODATA, the name exists but has no record of that type (dig AAAA api.lab).
• NXDOMAIN - the name does not exist at all, for any type. The SOA in the AUTHORITY section says how long that negative answer is cached.
• SERVFAIL - the resolver tried and failed: the zone's servers are down or the chain is broken. Next: dig +trace, or ask the zone's server with dig @server.
• REFUSED - the server will not answer you, typically a zone's own server asked about a zone it does not hold.
• A timeout (communications error ... timed out) means nothing answered on port 53.

Scripting trap: dig exits 0 for NXDOMAIN and SERVFAIL, so check the answer ([ -n "$(dig +short name)" ]), not the exit code.

Also asked: How can you tell from dig whether an answer came from a cache? · What does dig +trace show, and when do you use it? · What does the aa flag mean in dig output?

Learn it: 8.18 Reading dig properly

How would you move a service to a new IP with DNS, without users hitting the old one for an hour? Junior

Plan around the TTL: every resolver keeps the old answer until its TTL runs out, and you cannot flush someone else's cache.

1. Check the current TTL on the authoritative server: dig +noall +answer @ns1.lab orders.lab.
2. Lower it (to 60 or 300) at least one full old TTL before the change - the lowering itself only spreads as the old TTL expires.
3. Change the record (and bump the SOA serial, or secondaries never pick it up).
4. Keep the old server running until the old TTL plus some margin has passed: programs that cache names themselves (nginx at startup, Java) hold them longer.
5. Raise the TTL again afterwards.

Also watch negative caching: if clients looked the new name up before it existed, the NXDOMAIN is cached for the SOA minimum.

Also asked: What is the difference between an A record and a CNAME? · Why can you not put a CNAME at the zone apex? · What is split-horizon DNS, and how can it break a connection?

Learn it: 8.22 Records, TTLs, negative caching and split horizon

A resolv.conf has "search default.svc.cluster.local svc.cluster.local cluster.local" and "options ndots:5". What happens when a program looks up api.github.com? Junior

The ndots rule: a name with fewer than ndots dots is tried with every search domain appended first, and only then as written.

api.github.com has 2 dots, 2 < 5, so the resolver tries:

1. api.github.com.default.svc.cluster.local - NXDOMAIN
2. api.github.com.svc.cluster.local - NXDOMAIN
3. api.github.com.cluster.local - NXDOMAIN
4. api.github.com - the answer

glibc asks for A and AAAA for each, so that is eight queries for one outside name, six certain to fail. At high request rates that is load on the internal DNS server and latency on every call.

Fixes, cheapest first: a trailing dot (api.github.com., fully qualified, no search), a lower ndots for that program, a cache close by, and reusing connections. You can count the queries with tcpdump -i lo port 53.

Also asked: What is a search domain in resolv.conf? · What does a trailing dot at the end of a hostname mean? · Why does host find a short name that dig says is NXDOMAIN?

Learn it: 8.27 Search domains and the ndots trap

What is the difference between "connection refused" and "connection timed out"? Junior

Both come from trying to open a TCP connection, for example with nc -zv -w 3 IP PORT:

• Connection refused - the host answered the SYN with a RST: it is reachable, but nothing listens on that port. Look at the service (is it running, which port and address does it listen on).
• Timed out - no reply at all: something dropped the SYN (usually a firewall) or the route loses it. Look at the path.
• No route to host - ARP failed or a router sent back "host unreachable": the neighbour step.

They fit into the order for any "cannot reach X": name (getent hosts), route (ip route get), neighbour (ip neigh), port (nc -zv), packets (tcpdump). With tcpdump a timeout shows as the same Flags [S] retransmitted with nothing back.

Also asked: An application cannot connect to a database. How do you troubleshoot it? · How do you check whether a port is open on a remote host? · How would you use tcpdump to see whether packets leave the box?

Learn it: 8.29 A method for "I cannot reach X"