OnCallReady

TCP, TLS & HTTP: interview questions

The question you are most likely to get for each topic, a model answer, and what else comes up. From chapter 9 of the course.

Walk me through what happens between typing https://shop.lab/cart and the first byte of the response arriving. Junior

  1. Resolve shop.lab: getaddrinfo -> nsswitch -> /etc/hosts -> the stub resolver -> the recursive chain. (time_namelookup)
  2. Route: the kernel picks the interface and next hop and ARPs for the gateway.
  3. TCP handshake: SYN, SYN-ACK, ACK - one RTT. Fails as refused (RST), timeout (dropped) or port exhaustion. (time_connect)
  4. TLS handshake: ClientHello with SNI and ALPN, the server sends its leaf + intermediates, the client verifies the chain against its trust store and the name against the SAN. One RTT on TLS 1.3. (time_appconnect)
  5. Request: GET /cart with a Host header, over HTTP/2 if ALPN agreed.
  6. Far side: a load balancer and reverse proxy (TLS termination, X-Forwarded-For), then the app, which does its own lookups, connections and database calls.
  7. First byte arrives. (time_starttransfer)

When it is slow, curl -w times each phase so you know whose it is.

Also asked: What is the difference between TCP and UDP? · What is the difference between HTTP and HTTPS? · How do you check which process is listening on a port?

Explain the TCP three-way handshake, and what "connection refused" and "connection timed out" tell you. Junior

Before data flows, the kernels set up the connection: the client sends SYN ("let's talk", its starting seq), the server answers SYN-ACK (its own seq, ack = client seq + 1), the client sends ACK. It costs one RTT.

The failures, from the wire:

The timing is the diagnosis: instant means something answered.

Also asked: What does it mean when a service listens on 127.0.0.1 instead of 0.0.0.0? · What is the difference between a firewall that rejects and one that drops? · How do you test whether a TCP port is reachable from a box?

Learn it: 9.1 The handshake, and the three ways a connection fails

How do you find which process is listening on a port, and what do the socket states in ss tell you? Junior

sudo ss -tlnp - TCP, listening, numeric, with the process: users:(("java",pid=1210,fd=41)). Without sudo the Process column is empty for other users' sockets. ss -tan shows every state:

Count one state with ss -Htan state close-wait | wc -l (-H drops the header). ss -s is the first command in "out of connections".

Also asked: You see thousands of sockets in CLOSE-WAIT. What does that mean? · Is a large number of TIME-WAIT sockets a problem? · What do Recv-Q and Send-Q mean on a LISTEN socket?

Learn it: 9.5 Socket states, and reading ss properly

A client suddenly fails with "Cannot assign requested address" to one server, while everything else works. What is going on? Junior

Ephemeral port exhaustion. A connection is a 4-tuple (source IP, source port, destination IP, destination port); towards one destination only the source port varies, from net.ipv4.ip_local_port_range (32768-60999, about 28,000 ports).

A client that opens a new connection per request and closes it first leaves each one in TIME-WAIT for 60 seconds, holding its port: about 470 new connections a second to one destination and connect() fails with EADDRNOTAVAIL. Other destinations still work - the second clue.

Prove it: ss -Htan state time-wait '( dport = :443 )' | wc -l close to the range size.

Fix: connection reuse - HTTP keep-alive and a connection pool. A wider port range or tcp_tw_reuse only buys headroom. Behind a NAT gateway the same arithmetic hits its SNAT ports.

Also asked: Why does the first request after a quiet period sometimes fail with "connection reset"? · What is the listen backlog, and what happens when the accept queue is full? · What is the difference between TCP keepalive and HTTP keep-alive?

Learn it: 9.8 Ports, queues and exhaustion

How would you capture the traffic between a server and its database on port 5432, and what would you look for? Junior

sudo tcpdump -nn -i any -c 50 host 10.0.3.12 and port 5432

Then read the flags:

The strongest evidence is a capture on both ends: "the SYN left here and never arrived there" localises a drop to the path in between.

Also asked: How do you prove that a firewall is dropping traffic between two hosts? · Why do you use -nn with tcpdump? · How do you avoid capturing your own SSH session when you run tcpdump over SSH?

Learn it: 9.11 Reading tcpdump

What is MTU, and why would small requests work while big downloads hang? Junior

MTU is the largest IP packet an interface sends - 1500 on Ethernet (ip link show). MSS is the largest TCP payload, MTU - 40 = 1460, announced in the SYN.

If the path is smaller than the ends - a VPN tunnel that carries only 1400 - full-size packets with DF (Don't Fragment) set are dropped by the router, which should send back ICMP "fragmentation needed" so the sender shrinks them (Path MTU Discovery). If a firewall blocks that ICMP, the big packets just vanish: an MTU black hole. Handshakes and small responses fit; big responses and TLS certificates hang.

Prove it with ping -M do -s 1472 -c 1 HOST (1500 bytes) vs -s 1372 (1400): big vanishes, small works. Fix: lower the MTU on that interface (ip link set dev X mtu 1400, then netplan), MSS clamping on the gateway, and let ICMP type 3 through.

Also asked: What is the typical MTU on Ethernet, and how do you check it? · What is the difference between MTU and MSS? · Why is blocking all ICMP a bad idea?

Learn it: 9.13 MTU, MSS and black holes

What is a certificate chain, and how does a client verify it? Junior

The server's leaf certificate says "this public key belongs to shop.lab", signed by an intermediate CA, which is signed by a root CA. The client:

  1. Gets the leaf and the intermediates from the server in the TLS handshake (the server should not send the root).
  2. Builds the chain up to a root in its trust store (/etc/ssl/certs/ca-certificates.crt on Ubuntu).
  3. Checks each signature, the validity dates, and that the hostname is in the SAN list (the CN is ignored).

Common failures: unable to get local issuer certificate = the server left out the intermediate (browsers hide it, curl and Java fail; fix the server, serve a fullchain); self-signed in chain = root not trusted (add it with update-ca-certificates); expired; name mismatch. openssl s_client -connect host:443 -servername host shows what the server sent. Never fix it with -k.

Also asked: What is SNI, and why does it matter when you connect by IP? · Why does a site work in the browser but fail with curl or Java? · What is the difference between TLS 1.2 and TLS 1.3 for latency?

Learn it: 9.15 TLS: the handshake, the chain, and trust

How do you check when a server's certificate expires, and how would you catch expiry before users do? Junior

From the server, without saving anything:

echo | openssl s_client -connect vault.lab:443 2>/dev/null | openssl x509 -noout -subject -enddate

From a file: openssl x509 -in shop.lab.crt -noout -subject -issuer -dates.

For monitoring, -checkend N exits 1 if the certificate expires within N seconds:

openssl x509 -in shop.lab.crt -noout -checkend 604800   # one week

Run it from a systemd timer or cron and alert on a non-zero exit. Every certificate should have an owner and alerts at 30, 14 and 7 days - an expired internal certificate is one of the most common and most preventable outages.

Also asked: A Java service fails with "PKIX path building failed" but curl from the same host works. Why? · How do you check that a private key matches a certificate? · What is mTLS, and when is it used?

Learn it: 9.17 Certificates with openssl, and the JVM truststore

How do you debug an HTTP endpoint from the command line? Junior

With curl -v URL, which is the server-side network tab:

Useful variants:

Read the status by family: 2xx success, 3xx go elsewhere, 4xx the client's fault (401 not authenticated, 403 not allowed), 5xx the server's.

Also asked: What is the difference between 401 and 403? · What does idempotent mean, and which HTTP methods are idempotent? · What is the difference between a 301 and a 302 redirect?

Learn it: 9.21 HTTP on the wire, with curl -v

Users get 502s from nginx in front of your application. How do you troubleshoot? Junior

A 502 means nginx could not get a valid response on its connection to the upstream, so look behind it:

  1. sudo tail /var/log/nginx/error.log - the reason is in the line: connect() failed (111: Connection refused) = the app is down or on another port; upstream prematurely closed connection = it crashed or closed mid-request; no live upstreams = every backend marked failed.
  2. Check the app: systemctl status, sudo ss -tlnp for the port it listens on, its logs.
  3. curl the upstream directly (curl -v http://127.0.0.1:8080/...) to skip nginx.

Compare: 504 = upstream reached but slow (proxy_read_timeout, 60 s default); 503 = no backend available; 499 in access.log = the client gave up first.

Also asked: What is the difference between a forward proxy and a reverse proxy? · What is X-Forwarded-For, and why should you not trust it blindly? · What is the difference between an L4 and an L7 load balancer?

Learn it: 9.23 Reverse proxies, load balancers and forward proxies

An API is slow. How do you figure out whether it is the network or the application? Junior

Time each phase with curl -w (the values are cumulative, so read the gaps):

curl -s -o /dev/null -w 'dns %{time_namelookup} tcp %{time_connect} tls %{time_appconnect} ttfb %{time_starttransfer} total %{time_total}\n' https://api.lab/

Measure several times and look at the spread, not the average. Then time each hop separately - through the proxy, the app directly, the app's dependency - to find which one owns the latency.

Also asked: How do you measure how long each part of an HTTP request takes? · Why do you look at percentiles rather than averages for latency? · Why is a new connection per request slower than reusing one?

Learn it: 9.27 Where latency lives: curl -w and the road to the first byte

Practise these answers with flashcards and labs Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.