Lesson 9.15 · TCP, TLS & HTTP · 25 min read

TLS: the handshake, the chain, and trust

Q: It works in the browser, so why does curl say "unable to get local issuer certificate"?

Because the server only sends its leaf certificate and not the intermediate. Browsers compensate: they cache intermediates from other sites and can download a missing one from the URL in the certificate (AIA). curl, Python, Go and the JVM do not. So the chain from the leaf to the trusted root is broken for them. The fix is on the server: serve a fullchain file (leaf, then intermediates). Never patch clients around it.

Q: Should the server send the root certificate too?

No. The client must already have the root in its trust store, otherwise it would not trust it anyway, so sending it only wastes bytes in every handshake. The server sends the leaf plus every intermediate up to, but not including, the root. The client builds the chain from what it received plus its own store, and verifies each signature along the way.

Q: What's SNI and why does it matter when I connect by IP?

Server Name Indication is the hostname the client puts in its ClientHello, so a server hosting many sites on one IP can pick the right certificate. openssl sclient sends it automatically from the -connect host, but sends none when you connect by IP or through a tunnel to localhost, and you get the default certificate. Always add -servername orders.lab in those cases.

Q: Does the certificate's CN still matter?

Not for hostname checks. Modern clients (browsers since 2017, Go since 1.15, Python, curl) only look at the Subject Alternative Name list. A certificate with CN=vault.lab and a SAN of only vault.internal.lab is invalid for vault.lab. Wildcards cover exactly one label: .lab matches api.lab, not x.api.lab and not lab itself.

Q: Is curl -k OK for internal services?

Only as a one-off diagnostic, to prove the problem is trust and not the service itself. -k, verify=False, InsecureSkipVerify and "trust all" Java TrustManagers disable the one thing TLS does against an attacker: proving who you are talking to. Encryption without authentication still lets anyone in the middle impersonate the server. Install the right CA with update-ca-certificates or pass --cacert instead.

Q: What is a certificate chain, and how does a client verify it?

The server's leaf certificate says "this public key belongs to shop.lab", signed by an intermediate CA, which is signed by a root CA. The client: 1. Gets the leaf and the intermediates from the server in the TLS handshake (the server should not send the root). 2. Builds the chain up to a root in its trust store (/etc/ssl/certs/ca-certificates.crt on Ubuntu). 3. Checks each signature, the validity dates, and that the hostname is in the SAN list (the CN is ignored). Common failures: unable to get local issuer certificate = the server left out the intermediate (browsers hide it, curl and Java fail; fix the server, serve a fullchain); self-signed in chain = root not trusted (add it with update-ca-certificates); expired; name mismatch. openssl sclient -connect host:443 -servername host shows what the server sent. Never fix it with -k.

In plain words

Imagine a stranger at your door who says "I'm from the gas company". You ask for a badge. The badge was signed by their regional office, and the regional office's stamp was signed by the national office, whose stamp you already keep in a drawer at home. You check each signature up the chain until you reach one you know. If the regional office's page is missing, you can't connect the badge to the stamp in your drawer, so you don't open the door.

In TLS, the badge is the leaf certificate, the regional office is the intermediate, the stamp in your drawer is a root in the trust store (/etc/ssl/certs/ca-certificates.crt). openssl s_client shows you the whole chain the server sent.

Why this matters

Every https:// URL, and most database connections, run inside TLS. When TLS fails you get a scary one-line error ("unable to get local issuer certificate") and a developer saying "but it works in my browser". Certificate problems are among the most common outages in companies, and each has a precise cause you can prove in one command.

What you need to know already: the TCP handshake and RTT (9.1), curl -v (9.1), DNS names (Chapter 8).

The words first

TLS (Transport Layer Security; its old name is SSL) does two jobs on top of a TCP connection: it encrypts the bytes so nobody in the middle can read them, and it proves who the server is. HTTPS is simply HTTP inside TLS, on port 443.

Key pair: two linked keys. The private key stays secret on the server; the public key can be given to anyone. Something signed with the private key can be checked with the public key.
Certificate (cert): a small file that says "this public key belongs to shop.lab, valid from date A to date B", signed by someone.
CA (Certificate Authority): an organisation whose signature clients believe. A root CA certificate signs itself; an intermediate CA is signed by the root and does the day-to-day signing.
Leaf: the server's own certificate, at the bottom of the chain.
Chain: leaf -> intermediate(s) -> root, each signed by the next.
Trust store: the list of root CAs a machine believes, a file on disk.

The handshake, in the parts that break

After the TCP handshake, the client and server exchange TLS messages:

ClientHello   ->  TLS versions, ciphers, ALPN (h2, http/1.1), and SNI: the NAME
                  of the site you want
ServerHello   <-  chosen version and cipher
Certificate   <-  the leaf certificate PLUS the intermediates
CertVerify, Finished

A cipher is the encryption method; the two sides pick one both support.
ALPN lets them agree which HTTP version to speak inside (h2 = HTTP/2, 9.21).
SNI (Server Name Indication) is the hostname the client wants, sent in the clear so a server hosting many sites can pick the right certificate.
CertVerify proves the server holds the private key for its certificate; Finished closes the handshake.

TLS 1.3 does this in one round trip after TCP; TLS 1.2 needs two. That is why the TLS part of a curl timing is about one RTT on a modern server and two on an old one - and why connection reuse (9.8) matters so much for latency.

curl -v narrates it ((OUT) = we sent, (IN) = we received):

* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519MLKEM768 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=github.com
*  start date: Jul 25 20:00:00 2026 GMT
*  expire date: May  1 20:00:00 2027 GMT
*  subjectAltName: host "github.com" matched cert's "github.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo ECC Domain Validation Secure Server CA
*  SSL certificate verify ok.

CAfile is the trust store curl used. Under Server certificate: subject (who the cert is for; CN = Common Name), the validity dates, subjectAltName (the names it covers - the one that is checked), and issuer (who signed it). verify ok is the verdict.

(X25519MLKEM768 is the hybrid post-quantum key exchange that OpenSSL 3.5 uses by default. You will see it more and more.)

The chain

A certificate is signed by an issuer. The chain goes up to a root that your machine trusts:

leaf           CN=registry.lab            signed by ->
intermediate   CN=LabCorp Issuing CA G2   signed by ->
root           CN=LabCorp Root CA         (self-signed, in the trust store)

The server must send the leaf and the intermediates. It should not send the root (the client has it or it would not trust it anyway). The client builds the chain from what it received plus its trust store.

openssl is the Swiss-army knife for TLS. openssl s_client -connect host:port opens a TLS connection like a client would and prints everything (</dev/null gives it empty input so it exits straight away). It shows exactly what was sent:

$ openssl s_client -connect registry.lab:443 </dev/null
Connecting to 10.0.3.40
depth=2 C=RO, O=LabCorp, CN=LabCorp Root CA
verify return:1
depth=1 C=RO, O=LabCorp, CN=LabCorp Issuing CA G2
verify return:1
depth=0 CN=registry.lab
verify return:1
CONNECTED(00000003)
---
Certificate chain
 0 s:CN=registry.lab
   i:C=RO, O=LabCorp, CN=LabCorp Issuing CA G2
   a:PKEY: RSA, 2048 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Jul 20 20:00:00 2026 GMT; NotAfter: Jul 18 20:00:00 2027 GMT
 1 s:C=RO, O=LabCorp, CN=LabCorp Issuing CA G2
   i:C=RO, O=LabCorp, CN=LabCorp Root CA
...
Verify return code: 0 (ok)

Read it: depth= lines are the chain the client built, root first (these go to stderr). Certificate chain is what the server sent: s: subject, i: issuer - each i: should be the next entry's s:. And the last line is the verdict. s_client exits 0 even when verification fails; read the Verify return code.

Trust stores

/etc/ssl/certs/ca-certificates.crt   the bundle curl, OpenSSL, git, apt use
/usr/local/share/ca-certificates/    where YOU put extra CAs (.crt files!)
update-ca-certificates               rebuilds the bundle from both
$JAVA_HOME/lib/security/cacerts      the JVM's own store (next lesson)

(A bundle is many certificates concatenated in one file.) Big companies run their own CA for internal sites - LabCorp's here - and every machine must be told to trust its root. Adding a company CA:

# with the CA file in your current directory (on this box LabCorp's is already there:
# ls /usr/local/share/ca-certificates/)
sudo cp labcorp-root.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

(The rehash warning is normal.) The file must end in .crt - a .pem in that directory is silently ignored, which has wasted many afternoons.

The failures, with their real numbers

OpenSSL's verify errors are numbered; curl prints the text, s_client both.

20  unable to get local issuer certificate    the chain stops: an issuer is
                                              neither sent nor trusted
21  unable to verify the first certificate    (s_client, alongside 20) the
                                              leaf alone could not be verified
19  self-signed certificate in certificate chain   the chain ends in a root
                                              that is not in the trust store
18  self-signed certificate                   the leaf IS a self-signed cert
10  certificate has expired
 9  certificate is not yet valid              clock skew, or a cert issued with
                                              a future start date
62  hostname mismatch                         (only when asked to check names)

The missing intermediate - the defining TLS bug

$ curl https://api.lab/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the webpage mentioned above.

$ openssl s_client -connect api.lab:443 </dev/null 2>&1 | grep -E 'depth|verify error|^ [0-9] s:|Verify return'
depth=0 CN=api.lab
verify error:num=20:unable to get local issuer certificate
verify error:num=21:unable to verify the first certificate
 0 s:CN=api.lab
Verify return code: 21 (unable to verify the first certificate)

One certificate in the chain. The server sends the leaf only. The root (LabCorp Root CA) is trusted on this box - but nothing connects the leaf to it.

Browsers still show a green padlock: they cache intermediates from other sites and can fetch a missing one from a URL written in the certificate (the AIA, Authority Information Access, field). curl, Python, Go and the JVM do not. Hence "it works in the browser, so the certificate is fine". The fix is on the server: serve the leaf followed by the intermediate (a "fullchain" file). No client should be changed to work around it.

Untrusted root

curl: (60) SSL certificate problem: self-signed certificate in certificate chain

The server sent a complete chain up to a root your store does not have - a company CA on a machine that was never given it, or a TLS-inspecting proxy (a company proxy that decrypts traffic with its own CA). Fix: install the right CA (update-ca-certificates), or --cacert FILE (trust this CA file for one command) for a one-off. Not -k (skip verification - see the end).

Expired

curl: (60) SSL certificate problem: certificate has expired

No client setting fixes this and none should.

Name mismatch

curl: (60) SSL: no alternative certificate subject name matches target host name 'vault.lab'

Modern clients check only the Subject Alternative Name (SAN) list; the CN is ignored entirely (browsers since 2017, Go since 1.15, Python, curl). A cert with CN=vault.lab and SAN: vault.internal.lab is invalid for vault.lab. A wildcard certificate (*.lab) matches exactly one label (one dot-separated part of the name): *.lab covers api.lab, not x.api.lab, and not lab.

SNI: one IP, many certificates

A server with several sites on one IP (virtual hosts) picks the certificate from the SNI name in the ClientHello. Modern openssl s_client sends SNI automatically, taken from the host in -connect. It sends none when you connect by IP address - or through a tunnel to localhost - and then you get the server's default certificate:

$ openssl s_client -connect 10.0.3.20:443 </dev/null 2>/dev/null | grep subject=
subject=CN=api.lab                          <- the default vhost, maybe not yours
$ openssl s_client -connect 10.0.3.20:443 -servername orders.lab </dev/null 2>/dev/null | grep subject=

Rule: whenever you connect by IP (or through any tunnel to localhost), pass -servername NAME. Otherwise you debug the wrong certificate for twenty minutes.

Note also: s_client does not check the hostname unless you add -verify_hostname name. "Verify return code: 0 (ok)" with the wrong SAN is entirely possible. curl always checks.

Never ship -k

curl -k (--insecure: skip certificate checks) and its equivalents in program code (verify=False in Python, InsecureSkipVerify in Go, a Java "trust all" TrustManager) turn off the only thing TLS does against an attacker: proving who you are talking to. Use them for one diagnostic command, to prove the problem is trust and not the service; never in code, never in an automated script.

What you can now do

Explain leaf, intermediate, root and trust store, and why the server must send the intermediate.
Read openssl s_client output: the chain the server sent and the verdict.
Map the common errors to their cause: missing intermediate, untrusted root, expired, name mismatch - and pass -servername when connecting by IP.

Why it helps

Certificate problems are among the most common outages in enterprise platforms, and each has a distinct message. "unable to get local issuer certificate" after someone renewed the proxy's certificate and forgot the intermediate: the browser still works, curl and Java fail. "self-signed certificate in certificate chain" on a freshly installed server that never got the company root CA. "certificate has expired" at midnight on a Saturday. A SAN that doesn't include the new hostname in a change request.

Knowing how the chain and SNI work lets you diagnose each in one command, tell whether the fix belongs on the server or the client, and push back when someone proposes curl -k or InsecureSkipVerify in a script.

Commands in this lesson

openssl curl

FAQ

It works in the browser, so why does curl say "unable to get local issuer certificate"?

Because the server only sends its leaf certificate and not the intermediate. Browsers compensate: they cache intermediates from other sites and can download a missing one from the URL in the certificate (AIA). curl, Python, Go and the JVM do not. So the chain from the leaf to the trusted root is broken for them. The fix is on the server: serve a fullchain file (leaf, then intermediates). Never patch clients around it.

Should the server send the root certificate too?

No. The client must already have the root in its trust store, otherwise it would not trust it anyway, so sending it only wastes bytes in every handshake. The server sends the leaf plus every intermediate up to, but not including, the root. The client builds the chain from what it received plus its own store, and verifies each signature along the way.

What's SNI and why does it matter when I connect by IP?

Server Name Indication is the hostname the client puts in its ClientHello, so a server hosting many sites on one IP can pick the right certificate. openssl s_client sends it automatically from the -connect host, but sends none when you connect by IP or through a tunnel to localhost, and you get the default certificate. Always add -servername orders.lab in those cases.

Does the certificate's CN still matter?

Not for hostname checks. Modern clients (browsers since 2017, Go since 1.15, Python, curl) only look at the Subject Alternative Name list. A certificate with CN=vault.lab and a SAN of only vault.internal.lab is invalid for vault.lab. Wildcards cover exactly one label: *.lab matches api.lab, not x.api.lab and not lab itself.

Is curl -k OK for internal services?

Only as a one-off diagnostic, to prove the problem is trust and not the service itself. -k, verify=False, InsecureSkipVerify and "trust all" Java TrustManagers disable the one thing TLS does against an attacker: proving who you are talking to. Encryption without authentication still lets anyone in the middle impersonate the server. Install the right CA with update-ca-certificates or pass --cacert instead.

In an interview Junior

What is a certificate chain, and how does a client verify it?

The server's leaf certificate says "this public key belongs to shop.lab", signed by an intermediate CA, which is signed by a root CA. The client:

Gets the leaf and the intermediates from the server in the TLS handshake (the server should not send the root).
Builds the chain up to a root in its trust store (/etc/ssl/certs/ca-certificates.crt on Ubuntu).
Checks each signature, the validity dates, and that the hostname is in the SAN list (the CN is ignored).

Common failures: unable to get local issuer certificate = the server left out the intermediate (browsers hide it, curl and Java fail; fix the server, serve a fullchain); self-signed in chain = root not trusted (add it with update-ca-certificates); expired; name mismatch. openssl s_client -connect host:443 -servername host shows what the server sent. Never fix it with -k.

Also asked: What is SNI, and why does it matter when you connect by IP? · Why does a site work in the browser but fail with curl or Java? · What is the difference between TLS 1.2 and TLS 1.3 for latency?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.