The orders service cannot call an internal API. Someone opened the URL in a browser, saw a green padlock and closed the ticket as "a code problem". From the server:
$ curl -sS https://api.lab/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the webpage mentioned above.
$ echo "exit=$?"
exit=60
$ curl -sSk https://api.lab/
{"service":"api","version":"2.14.1"}-k skips the certificate check, and the service answers. So the app is fine and the problem is trust. Keep -k for exactly this one test, never in a script.
What the error means
To trust a server, the client builds a chain: the server's own certificate (the leaf) was signed by an intermediate CA, which was signed by a root CA. The client must end at a root it already has in its trust store; on Ubuntu that is /etc/ssl/certs/ca-certificates.crt, which curl, wget, git and Python's ssl module read.
"Unable to get local issuer certificate" means: for some certificate in the chain, the client has no copy of the certificate that signed it. There are exactly two ways to get there, and they are fixed in different places:
- the server sends its leaf but not the intermediate, or
- the server sends everything, and your machine does not have the root.
Tell them apart: what the server sends
openssl s_client prints the chain as the server sent it. Two servers, the same curl error:
$ openssl s_client -connect api.lab:443 -servername api.lab </dev/null 2>/dev/null | grep -E '^ *[0-9] s:|^ *i:|Verify return'
0 s:CN=api.lab
i:C=RO, O=LabCorp, CN=LabCorp Issuing CA G2
Verify return code: 21 (unable to verify the first certificate)
$ openssl s_client -connect registry.lab:443 -servername registry.lab </dev/null 2>/dev/null | grep -E '^ *[0-9] s:|^ *i:|Verify return'
0 s:CN=registry.lab
i:C=RO, O=LabCorp, CN=LabCorp Issuing CA G2
1 s:C=RO, O=LabCorp, CN=LabCorp Issuing CA G2
i:C=RO, O=LabCorp, CN=LabCorp Root CA
Verify return code: 20 (unable to get local issuer certificate)s: is a certificate's subject, i: its issuer. Read the last i: line:
- api.lab sends one certificate, issued by an intermediate, and stops. Verify code 21. The intermediate is missing: cause 1, fix the server.
- registry.lab sends the leaf and the intermediate; the last issuer is LabCorp Root CA. Verify code 20. The chain is complete, but this box does not trust that root: cause 2, fix the client.
Always pass -servername: a server with several sites picks the certificate by that name (SNI), and without it you may be looking at another site's chain.
Why the browser said the padlock was green: browsers fill in a missing intermediate themselves (they cache intermediates and can download them from the URL inside the leaf), and a company laptop has the company root installed. curl, the JVM and Go do neither.
Cause 1: the server does not send its intermediate
Prove it with the files before you touch the config. The leaf alone does not verify; with the intermediate as the missing link, it does:
$ openssl verify /etc/ssl/lab/shop.lab.crt
CN=shop.lab
error 20 at 0 depth lookup: unable to get local issuer certificate
error /etc/ssl/lab/shop.lab.crt: verification failed
$ openssl verify -untrusted /etc/ssl/lab/labcorp-issuing-g2.crt /etc/ssl/lab/shop.lab.crt
/etc/ssl/lab/shop.lab.crt: OKBuild a fullchain file, leaf first and then the intermediate (not the root), and point nginx at it:
$ cat /etc/ssl/lab/shop.lab.crt /etc/ssl/lab/labcorp-issuing-g2.crt | sudo tee /etc/ssl/lab/shop.lab.fullchain.crt >/dev/null
$ grep ssl_certificate /etc/nginx/sites-available/shop.lab
ssl_certificate /etc/ssl/lab/shop.lab.crt;
ssl_certificate_key /etc/ssl/private/shop.lab.key;
$ sudo sed -i 's#/etc/ssl/lab/shop.lab.crt;#/etc/ssl/lab/shop.lab.fullchain.crt;#' /etc/nginx/sites-available/shop.lab
$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
$ sudo systemctl reload nginx
$ openssl s_client -connect shop.lab:443 -servername shop.lab </dev/null 2>/dev/null | grep -E '^ *[0-9] s:|^ *i:|Verify return'
0 s:CN=shop.lab
i:C=RO, O=LabCorp, CN=LabCorp Issuing CA G2
1 s:C=RO, O=LabCorp, CN=LabCorp Issuing CA G2
i:C=RO, O=LabCorp, CN=LabCorp Root CA
Verify return code: 0 (ok)Two certificates, verify code 0, and curl gets the page. With Let's Encrypt this is the whole difference between cert.pem and fullchain.pem: ssl_certificate always takes the fullchain. nginx -t before the reload matters: a reload with a broken file keeps the old configuration running, so the change silently does not happen (the same habit as in the nginx 502 post).
Do not "fix" this on the clients by adding the intermediate to their trust stores. It works until the server's certificate is renewed under a different intermediate, and then every client breaks at once.
Cause 2: your machine does not trust the root
Internal CAs and TLS-inspecting company proxies sign with a root no public trust store contains. The box needs that root, once. Here it was missing:
$ curl -sS https://registry.lab/v2/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the webpage mentioned above.Check what you were given really is the root (subject = issuer), then add it the Ubuntu way:
$ openssl x509 -in /tmp/labcorp-root.crt -noout -subject -issuer -dates
subject=C=RO, O=LabCorp, CN=LabCorp Root CA
issuer=C=RO, O=LabCorp, CN=LabCorp Root CA
notBefore=Apr 5 17:43:01 2024 GMT
notAfter=Dec 9 17:43:01 2034 GMT
$ sudo cp /tmp/labcorp-root.crt /usr/local/share/ca-certificates/
$ sudo update-ca-certificates
Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
$ curl -sS https://registry.lab/v2/
{}1 added is the line to look for; the rehash warning is normal. Two details that cost people an afternoon: the file must be PEM and must end in .crt, or update-ca-certificates skips it without a word; and a container image has its own trust store, so the same two lines belong in the Dockerfile. Some runtimes ignore the system store altogether: Python's requests uses its own certifi bundle (REQUESTS_CA_BUNDLE), Node uses NODE_EXTRA_CA_CERTS, and Java has its own.
Java says the same thing differently: "PKIX path building failed"
A Java service on the same box, same URL, after the root was added for the OS:
$ journalctl -u ledger-sync --no-pager | grep -m2 -E 'ERROR|Exception'
Sep 22 20:00:03 oncall-lab java[17385]: 2026-09-22T20:00:03.000+00:00 ERROR 17385 --- [main] c.l.ledger.sync.LedgerClient : sync failed: I/O error on GET request for "https://registry.lab/v2/": PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Sep 22 20:00:03 oncall-lab java[17385]: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
$ systemctl cat ledger-sync | grep ExecStart
ExecStart=/opt/java/temurin-21/bin/java -Xmx256m -jar /opt/app/ledger-sync.jar"curl works, Java does not" is two trust stores. This JVM is a vendor JDK in /opt with its own cacerts, which update-ca-certificates never touches (Ubuntu's own openjdk-* packages are kept in sync by ca-certificates-java; a JDK from a tarball is not). Import the root into that JDK and restart, because the JVM reads its trust store once, at startup:
$ keytool -list -cacerts -storepass changeit | head -4
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 141 entries
$ keytool -list -cacerts -storepass changeit | grep -i labcorp
$ sudo /opt/java/temurin-21/bin/keytool -importcert -noprompt -alias labcorp-root -file /usr/local/share/ca-certificates/labcorp-root.crt -cacerts -storepass changeit
Certificate was added to keystore
$ sudo systemctl restart ledger-sync
$ journalctl -u ledger-sync -n 1 --no-pager
Sep 22 20:00:15 oncall-lab java[17392]: 2026-09-22T20:00:15.700+00:00 INFO 17392 --- [main] c.l.ledger.sync.LedgerClient : GET https://registry.lab/v2/ -> 200 in 11ms, 0 pending entriesUse the keytool of the JDK that runs the service, and expect the import to vanish on the next JDK upgrade, which replaces cacerts. The durable fix is to add the root in whatever installs the JDK, or to give the JVM a trust store you own with -Djavax.net.ssl.trustStore=....
What not to do
curl -k,verify=Falsein Python, a trust-allTrustManagerin Java: each one turns an outage into a vulnerability and stays in the code long after the outage.- Copying the server's intermediate onto clients (see above).
- Adding a root you cannot trace to someone who owns it. A root in the trust store can vouch for any name.
The other certificate errors curl prints
The text after SSL certificate problem: names the fault:
| curl says | it means | fix |
|---|---|---|
unable to get local issuer certificate | no chain to a trusted root | this post: the server's chain or your trust store |
certificate has expired | notAfter is in the past | reissue it; on a kubeadm cluster, renew its certificates |
self-signed certificate in certificate chain | the server sent a root you do not trust | cause 2: add that root, if it is really yours |
no alternative certificate subject name matches target host name | the name you used is not in the SAN | reissue with the right name, or call the name it was issued for |
Inside Kubernetes the same chain rules apply to Ingress certificates; when the certificate never gets issued in the first place, that is cert-manager's Order and Challenge.
Practise it
Incident: works in the browser, fails in the app gives you api.lab with its missing intermediate and vault.lab with two more faults to name. Serve the whole chain has you build the fullchain, test nginx and reload. Incident: curl works, Java does not is the vendor JDK with its own truststore, and Drill: what is wrong with this certificate varies the fault every round.