Lesson 9.21 · TCP, TLS & HTTP · 19 min read

HTTP on the wire, with curl -v

Q: Why does curl -v output break my grep?

Because curl writes the , and < lines to stderr, and only the body to stdout. A pipe only carries stdout, so curl -v URL grep sees just the body. Redirect first: curl -v URL 2&1 grep '^<' gives you the response headers. For headers on stdout without the rest of the trace use -i, and for just the status code -sS -o /dev/null -w '%{httpcode}\n'.

Q: What's the difference between -H 'Host:' and --resolve?

Both let you hit a specific IP while pretending to be a hostname, for example to test a vhost before DNS is updated. -H 'Host: shop.lab' only changes the HTTP header. For HTTPS that is not enough: curl still sends the IP (or no name) as SNI and checks the certificate against it. --resolve shop.lab:443:10.64.0.2 overrides DNS for curl, so SNI, the Host header and certificate checks all use the right name.

Q: Why doesn't curl follow redirects?

By design: curl shows you exactly what the server answered, including a 301 with a Location header. Add -L to follow. -w '%{numredirects} %{urleffective}' shows where you ended up. If you get "Maximum (50) redirects followed" behind a TLS-terminating proxy, look at X-Forwarded-Proto: the app thinks the request was plain HTTP and keeps redirecting to HTTPS.

Q: Which HTTP methods are safe to retry?

GET, HEAD and OPTIONS are safe: they must not change anything. PUT and DELETE are idempotent: doing them twice has the same effect as once. POST and PATCH are neither, so a retry can create a second order or refund. That is why proxies and client libraries retry idempotent requests by default and not POST. Non-idempotent operations that need retries use an idempotency key the server deduplicates on.

Q: Is HTTP/2 always faster?

Often, not always. It multiplexes many streams over one connection with compressed headers, so it avoids opening several connections and removes HTTP-level head-of-line blocking. But all streams share one TCP connection, so one lost packet stalls every stream until it is retransmitted, which HTTP/3 over QUIC addresses. And an L4 load balancer sends that one long-lived connection to a single backend, so balancing HTTP/2 traffic well needs an L7 proxy.

Q: How do you debug an HTTP endpoint from the command line?

With curl -v URL, which is the server-side network tab: - lines - what curl did: resolve, connect, TLS. - lines - the request exactly as sent: method, path, headers (Host!), empty line. - < lines - the response: status line, headers, then the body. Useful variants: - curl -sS -o /dev/null -w '%{httpcode}\n' URL - just the status code. - curl -I URL (headers only) and curl -L to follow redirects. - -f makes curl exit 22 on a 4xx/5xx, which scripts want; exit codes 6 (DNS), 7 (connect), 28 (timeout), 35 (TLS), 60 (certificate) say which phase failed. - --resolve shop.lab:443:IP tests a site on a specific IP while keeping SNI and certificate checks right. Read the status by family: 2xx success, 3xx go elsewhere, 4xx the client's fault (401 not authenticated, 403 not allowed), 5xx the server's.

In plain words

HTTP is like ordering at a counter with a fixed script. You say one line, "GET the menu, please, using version 1.1", then a few notes on separate lines ("I'm at table shop.lab", "I accept anything"), then a blank line to say you're done. The waiter answers with a status ("200, here you go" or "404, we don't have that"), their own notes, a blank line, and then the food.

curl -v lets you watch the whole conversation: lines starting with > are what you said, lines with < are what the server said, and * lines are curl describing what it's doing. Some orders are safe to repeat (asking for the menu twice); others are not (paying twice).

Why this matters

As a frontend developer you have seen HTTP from the browser's side, in the network tab. On a server there is no browser: curl is your network tab. On call, the first question is usually "what exactly does this URL return?", and reading curl -v answers it without guessing.

What you need to know already: TCP connections and curl exit codes (9.1), TLS and ALPN (9.15), stdout vs stderr and 2>&1 (1.7).

The words first

HTTP is a text protocol on top of TCP (or TLS): the client sends a request, the server sends back a response.
A request has a method (the verb: GET = fetch, POST = submit, ...), a path (/api/orders), headers (Name: value lines of extra information) and sometimes a body (the data).
A response has a status code (200 OK, 404 Not Found, ...), headers, and a body.
A URL like https://shop.lab:443/cart is scheme (https), host (shop.lab), port (443, the default for https; 80 for http) and path.

One request, line by line

$ curl -v http://oncall-lab/
* Host oncall-lab:80 was resolved.
* IPv6: (none)
* IPv4: 127.0.1.1
*   Trying 127.0.1.1:80...
* Connected to oncall-lab (127.0.1.1) port 80
> GET / HTTP/1.1
> Host: oncall-lab
> User-Agent: curl/8.14.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< Server: nginx/1.26.3 (Ubuntu)
< Date: Wed, 23 Sep 2026 10:12:01 GMT
< Content-Type: text/html
< Content-Length: 162
< Connection: keep-alive
<
<!DOCTYPE html>
...
* Connection #0 to host oncall-lab left intact

* lines are curl telling you what it did: resolve, connect, TLS.
> lines are the request exactly as sent: request line (method, path, version), headers, and an empty line that ends the headers.
< lines are the response: status line, headers, empty line, then the body.
left intact - the connection stayed open for reuse.

curl -v writes the * > < lines to stderr, so curl -v URL 2>&1 | grep '^<' gives you just the response headers.

curl -I URL          HEAD request: headers only (some apps treat HEAD differently!)
curl -i URL          headers + body, on stdout
curl -sS -o /dev/null -w '%{http_code}\n' URL    the status code, nothing else

In the last one: -s silent (no progress bar), -S but still show errors, -o /dev/null throw the body away, -w FORMAT ("write out") print chosen values after the transfer - %{http_code} is one of many variables (9.27).

The headers that matter operationally

Host                   which site on this IP (virtual hosting) - required in 1.1
Content-Length         body size, or...
Transfer-Encoding: chunked   ...body sent in pieces when the size is not known
Connection: keep-alive / close   reuse the TCP connection or not
Location               where a 3xx redirect points
X-Forwarded-For/-Proto the client's real IP and scheme, added by proxies (9.23)
Retry-After            on 429/503: when to try again

-H 'Host: shop.lab' (-H adds or replaces a header) lets you test a virtual host (9.15) by IP before DNS exists; --resolve shop.lab:443:10.64.0.2 ("for this name and port, use this IP") does the same for HTTPS, and keeps SNI and certificate checks correct, which -H Host does not.

Methods and idempotency

Idempotent means "doing it twice has the same effect as doing it once".

GET, HEAD, OPTIONS   safe: must not change anything
PUT, DELETE          idempotent: doing it twice = doing it once
POST, PATCH          NOT idempotent: twice may mean two orders, two refunds

This is what decides which requests a proxy or a client library may retry (send again) automatically. Retrying a GET after a reset is fine; retrying a POST /payments/refund because the response timed out can refund twice - the server may well have done the work. Non-idempotent calls that must be retried need an idempotency key the server deduplicates on.

Status codes, operationally

The first digit is the family: 2xx success, 3xx go elsewhere, 4xx the client's fault, 5xx the server's fault.

200 OK, 201 Created, 204 No Content
301 / 308   permanent redirect (308 keeps the method and body)
302 / 307   temporary redirect (307 keeps the method)
304         not modified (conditional GET)
400         your request is malformed
401         not authenticated (no or bad credentials)
403         authenticated but not allowed - or a WAF (web firewall) or proxy said no
404         no such path - or no such vhost on a shared proxy
405         wrong method
408         the client was too slow sending the request
413         body too large - usually a proxy limit (nginx client_max_body_size, 1m)
429         rate limited; read Retry-After
499         (nginx only) the CLIENT closed before we answered
500         the application failed
502 / 503 / 504   the proxy's view of the upstream - next lesson

-f (fail) turns HTTP errors into a curl failure, which is what scripts want:

$ curl -sf http://oncall-lab/nope; echo "exit $?"
exit 22
$ curl -sS -f http://oncall-lab/nope
curl: (22) The requested URL returned error: 404

The curl exit codes worth knowing: 6 DNS, 7 connect failed, 28 timeout, 35 TLS handshake, 47 too many redirects, 52 empty reply, 56 receive failure (reset, proxy tunnel failed), 60 certificate.

Redirects

$ curl -sI http://ubuntu.com/ | grep -iE '^(HTTP|location)'
HTTP/1.1 301 Moved Permanently
Location: https://ubuntu.com/
$ curl -sL -o /dev/null -w '%{http_code} %{num_redirects} %{url_effective}\n' http://ubuntu.com/
200 1 https://ubuntu.com/

A redirect is a 3xx answer with a Location header: "ask there instead". curl does not follow redirects unless you pass -L (%{num_redirects} and %{url_effective} then say how many it followed and where it ended). A loop ends with:

curl: (47) Maximum (50) redirects followed

which, behind a TLS-terminating proxy, is almost always the X-Forwarded-Proto problem in the next lesson.

Keep-alive and connection reuse

Every new connection costs a TCP handshake (1 RTT) and a TLS handshake (1 RTT on 1.3, 2 on 1.2) before the request is even sent. For a small JSON response that setup is most of the time. curl reuses a connection across URLs in one command, and -w shows it (time_connect = when TCP was connected, time_appconnect = when TLS was done, time_total = the end; 9.27 covers these):

$ curl -s -o /dev/null -o /dev/null -w '%{time_connect} %{time_appconnect} %{time_total}\n' https://example.com/ https://example.com/
0.097012 0.193104 0.290530
0.000033 0.000037 0.096870
* (second line: no connect, no TLS - one round trip for the request itself)

The second request is three times faster. This is what a connection pool (9.8) buys a service, and why "new HTTP client per request" is a performance bug and (at scale) the port-exhaustion incident.

HTTP/2

HTTP/1.1 sends one request at a time per connection. HTTP/2 sends many at once over one TCP connection, each in its own stream, in a binary format with compressed headers. It is negotiated by ALPN inside the TLS handshake:

* ALPN: curl offers h2,http/1.1
* ALPN: server accepted h2
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://github.com/
> GET / HTTP/2
< HTTP/2 200
< content-type: text/html; charset=utf-8

Header names are lowercase in HTTP/2. It removes HTTP-level head-of-line blocking (one slow response no longer blocks the others on the connection), but moves it to TCP: one lost packet stalls every stream until it is retransmitted. That is the motivation for HTTP/3 over QUIC (UDP).

Operationally: a load balancer that only looks at connections, not requests (an "L4" balancer, 9.23), sees one long-lived HTTP/2 connection and sends everything on it to one backend (one of the servers behind it).

What you can now do

Read curl -v: * what curl did, > what it sent, < what came back.
Get just the status code in a script, and make curl fail on 4xx/5xx with -f.
Say which methods are safe to retry, and why connection reuse makes requests faster.

Why it helps

On-call, the first question is often "what exactly does the server return?", and curl -v or -sS -o /dev/null -w '%{http_code}' answers it without guessing. It lets you test a new site on a proxy by IP before DNS exists with --resolve, spot that a 404 comes from the shared proxy and not your app, and tell a 413 from nginx's 1 MB body limit apart from an application error.

Idempotency matters when you review a proxy's retry settings or a client library config: retrying a GET is harmless, retrying POST /api/payments/refund after a timeout can refund a customer twice. And curl's exit codes (6, 7, 28, 35, 60) make health-check scripts say what failed instead of just "failed".

Commands in this lesson

curl

FAQ

Why does curl -v output break my grep?

Because curl writes the *, > and < lines to stderr, and only the body to stdout. A pipe only carries stdout, so curl -v URL | grep sees just the body. Redirect first: curl -v URL 2>&1 | grep '^<' gives you the response headers. For headers on stdout without the rest of the trace use -i, and for just the status code -sS -o /dev/null -w '%{http_code}\n'.

What's the difference between -H 'Host:' and --resolve?

Both let you hit a specific IP while pretending to be a hostname, for example to test a vhost before DNS is updated. -H 'Host: shop.lab' only changes the HTTP header. For HTTPS that is not enough: curl still sends the IP (or no name) as SNI and checks the certificate against it. --resolve shop.lab:443:10.64.0.2 overrides DNS for curl, so SNI, the Host header and certificate checks all use the right name.

Why doesn't curl follow redirects?

By design: curl shows you exactly what the server answered, including a 301 with a Location header. Add -L to follow. -w '%{num_redirects} %{url_effective}' shows where you ended up. If you get "Maximum (50) redirects followed" behind a TLS-terminating proxy, look at X-Forwarded-Proto: the app thinks the request was plain HTTP and keeps redirecting to HTTPS.

Which HTTP methods are safe to retry?

GET, HEAD and OPTIONS are safe: they must not change anything. PUT and DELETE are idempotent: doing them twice has the same effect as once. POST and PATCH are neither, so a retry can create a second order or refund. That is why proxies and client libraries retry idempotent requests by default and not POST. Non-idempotent operations that need retries use an idempotency key the server deduplicates on.

Is HTTP/2 always faster?

Often, not always. It multiplexes many streams over one connection with compressed headers, so it avoids opening several connections and removes HTTP-level head-of-line blocking. But all streams share one TCP connection, so one lost packet stalls every stream until it is retransmitted, which HTTP/3 over QUIC addresses. And an L4 load balancer sends that one long-lived connection to a single backend, so balancing HTTP/2 traffic well needs an L7 proxy.

In an interview Junior

How do you debug an HTTP endpoint from the command line?

With curl -v URL, which is the server-side network tab:

* lines - what curl did: resolve, connect, TLS.
> lines - the request exactly as sent: method, path, headers (Host!), empty line.
< lines - the response: status line, headers, then the body.

Useful variants:

curl -sS -o /dev/null -w '%{http_code}\n' URL - just the status code.
curl -I URL (headers only) and curl -L to follow redirects.
-f makes curl exit 22 on a 4xx/5xx, which scripts want; exit codes 6 (DNS), 7 (connect), 28 (timeout), 35 (TLS), 60 (certificate) say which phase failed.
--resolve shop.lab:443:IP tests a site on a specific IP while keeping SNI and certificate checks right.

Read the status by family: 2xx success, 3xx go elsewhere, 4xx the client's fault (401 not authenticated, 403 not allowed), 5xx the server's.

Also asked: What is the difference between 401 and 403? · What does idempotent mean, and which HTTP methods are idempotent? · What is the difference between a 301 and a 302 redirect?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.