Lesson 9.17 · TCP, TLS & HTTP · 23 min read

Certificates with openssl, and the JVM truststore

Q: Why does openssl x509 show only one certificate from my bundle?

Because openssl x509 reads only the first PEM block in a file. A fullchain or CA bundle has several. To list them all use openssl crl2pkcs7 -nocrl -certfile bundle.crt openssl pkcs7 -printcerts -noout, and check that each issuer= equals the next subject=. grep -c 'BEGIN CERTIFICATE' bundle.crt gives a quick count.

Q: What's the difference between -untrusted and -CAfile in openssl verify?

-untrusted supplies intermediates that openssl may use to build the chain but will not treat as trust anchors. -CAfile replaces the trust store with the roots you give it. By default openssl verify uses the system store. So openssl verify -untrusted issuing-ca.crt leaf.crt answers "does this leaf, with this intermediate, chain to something the system trusts?", which is exactly the pre-deployment check.

Q: How do I check that a key matches a certificate?

Compare the public keys: openssl x509 -in cert.crt -noout -pubkey sha256sum and openssl pkey -in key.key -pubout sha256sum. Same hash, same pair. Older guides compare -modulus, which only works for RSA keys; the public-key comparison works for RSA and EC alike. Check before you reload nginx, which otherwise fails with "key values mismatch".

Q: How do you check when a server's certificate expires, and how would you catch expiry before users do?

From the server, without saving anything: From a file: openssl x509 -in shop.lab.crt -noout -subject -issuer -dates. For monitoring, -checkend N exits 1 if the certificate expires within N seconds: Run it from a systemd timer or cron and alert on a non-zero exit. Every certificate should have an owner and alerts at 30, 14 and 7 days - an expired internal certificate is one of the most common and most preventable outages.

In plain words

A certificate is like a passport. It says who you are (the name), who issued it (the country), and from when to when it is valid. openssl x509 is the border officer reading the passport: -subject, -issuer, -dates, and the list of names it covers.

The private key is like your fingerprint: the passport is useless unless the fingerprint matches, which is why nginx refuses to start when key and certificate don't belong together.

And Java is a traveller who carries their own list of trusted countries (cacerts) instead of using the one the building gave everyone (/etc/ssl/certs). Adding a country to the building's list doesn't change Java's.

Why this matters

The previous lesson was about reading what a server sends. This one is the other side of the job: checking certificate files before you deploy them, building the chain file a server needs, catching expiry before users do, and the classic "curl works, the Java service does not" puzzle.

What you need to know already: leaf, intermediate, root, chain and trust store (9.15), openssl s_client (9.15), pipes and 2>/dev/null (1.7), the JVM (5.13), and restarting a systemd service (Chapter 2).

Reading a certificate file

Certificates are usually stored as PEM files: base64 text between -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- lines, often named .crt or .pem. openssl x509 reads one: -in FILE the input, -noout do not print the certificate itself back, then the fields you want.

$ openssl x509 -in /etc/ssl/lab/shop.lab.crt -noout -subject -issuer -dates
subject=CN=shop.lab
issuer=C=RO, O=LabCorp, CN=LabCorp Issuing CA G2
notBefore=Aug 24 17:43:01 2026 GMT
notAfter=Nov 22 17:43:01 2026 GMT

$ openssl x509 -in /etc/ssl/lab/shop.lab.crt -noout -ext subjectAltName
X509v3 Subject Alternative Name:
    DNS:shop.lab

The four facts that matter: who (subject and SAN), signed by whom (issuer), when (dates). Options print in the order you give them. In a name like C=RO, O=LabCorp, CN=...: C = country, O = organisation, CN = common name. -ext subjectAltName prints one extension (an extra field; X509v3 is the certificate format version).

Straight from a server, without saving anything:

$ echo | openssl s_client -connect vault.lab:443 2>/dev/null | openssl x509 -noout -subject -enddate -ext subjectAltName
subject=CN=vault.lab
notAfter=Sep 11 17:43:01 2026 GMT
X509v3 Subject Alternative Name:
    DNS:vault.internal.lab

echo | closes s_client's stdin so it exits; 2>/dev/null drops the depth lines; openssl x509 reads the first PEM block from the pipe.

The full decode

$ mkdir -p ~/certs && cp /etc/ssl/lab/*.crt ~/certs/ && cd ~/certs
$ openssl x509 -in shop.lab.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            7e:be:de:17:07:0c:b6:d0:d1:eb:ec:36:36:71:06:5e:77:cc:5d:78
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: C=RO, O=LabCorp, CN=LabCorp Issuing CA G2
        Validity
            Not Before: Aug 24 17:43:01 2026 GMT
            Not After : Nov 22 17:43:01 2026 GMT
        Subject: CN=shop.lab
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
        ...
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:shop.lab
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                ...

-text decodes everything. The serial number and signature algorithm identify the certificate and how it was signed; skip them for now. Beyond the four facts, look at Basic Constraints (CA:TRUE for an intermediate or root, CA:FALSE for a leaf), Extended Key Usage (what the key may be used for: a server cert needs TLS Web Server Authentication; client certificates, see mTLS below, need TLS Web Client Authentication), and Authority Information Access, the URL browsers use to fetch a missing intermediate.

Expiry monitoring in one line

$ openssl x509 -in shop.lab.crt -noout -checkend 604800; echo "exit $?"
Certificate will not expire
exit 0
$ echo | openssl s_client -connect vault.lab:443 2>/dev/null | openssl x509 -noout -checkend 0; echo "exit $?"
Certificate will expire
exit 1

-checkend N exits 1 if the certificate expires within N seconds (604800 = one week). That is a complete health check for a systemd timer (2.16) or cron job. An expired internal certificate is a top-five enterprise outage cause and the most preventable one on the list; every certificate you deploy should have an owner and an alert at 30, 14 and 7 days.

Verify a chain offline

Before you deploy files, prove they form a chain:

$ openssl verify shop.lab.crt
CN=shop.lab
error 20 at 0 depth lookup: unable to get local issuer certificate
error shop.lab.crt: verification failed

$ openssl verify -untrusted labcorp-issuing-g2.crt shop.lab.crt
shop.lab.crt: OK

openssl verify FILE checks that a certificate chains to a trusted root. -untrusted supplies intermediates (to use but not trust); -CAfile replaces the trust store with specific roots. By default verify uses the system store, which here contains the LabCorp root.

Building a fullchain file

A fullchain is simply the leaf followed by its intermediates, concatenated:

$ cat shop.lab.crt labcorp-issuing-g2.crt > shop.lab.fullchain.crt

Order matters: leaf first, then the certificate that signed it, then the one that signed that. Not the root. Check the result - but note that openssl x509 reads only the first certificate in a file, so it cannot show you a bundle:

$ openssl crl2pkcs7 -nocrl -certfile shop.lab.fullchain.crt | openssl pkcs7 -print_certs -noout
subject=CN=shop.lab
issuer=C=RO, O=LabCorp, CN=LabCorp Issuing CA G2

subject=C=RO, O=LabCorp, CN=LabCorp Issuing CA G2
issuer=C=RO, O=LabCorp, CN=LabCorp Root CA

(The recipe converts the file to PKCS#7, another certificate file format, whose -print_certs lists every certificate - copy it as is.) Each issuer= equals the next subject=: a correct chain. grep -c 'BEGIN CERTIFICATE' file counts them quickly.

Does this key belong to this certificate?

nginx refuses to start if they do not match:

nginx: [emerg] SSL_CTX_use_PrivateKey("/etc/ssl/private/shop.lab.key") failed (SSL: error:05800074:x509 certificate routines::key values mismatch)

Check before deploying - compare the public keys:

$ openssl x509 -in shop.lab.crt -noout -pubkey | sha256sum
9f2c...  -
$ sudo openssl pkey -in /etc/ssl/private/shop.lab.key -pubout | sha256sum
9f2c...  -

-pubkey prints the public key inside the certificate; openssl pkey -pubout derives the public key from the private key; sha256sum turns each into a short fingerprint (a hash). Same hash, same key pair. (Older guides compare -modulus, which only works for RSA keys; the public-key comparison works for every key type - RSA and EC are the two common kinds.)

The JVM has its own truststore

A JDK (Java Development Kit) is an installed copy of Java: the java program that starts the JVM (5.13), plus tools like keytool. $JAVA_HOME is the directory it lives in. Java does not read /etc/ssl/certs; it has its own trust store (Java calls it a truststore), a file called cacerts:

$JAVA_HOME/lib/security/cacerts      PKCS12, default password "changeit"

(PKCS12 is the file format; "changeit" is the famous default password, which only protects the file against being changed.) This is the most common "curl works, Java fails" cause in a company with its own CA. The nuance you need, because it confuses people:

Ubuntu's own OpenJDK packages (installed with apt) link cacerts to /etc/ssl/certs/java/cacerts, which the ca-certificates-java hook regenerates when you run update-ca-certificates. On those, the system store and Java agree.
A JDK from a download or a vendor build - Temurin (a popular free JDK build) unpacked in /opt, anything installed by the SDKMAN tool - has its own cacerts, untouched by update-ca-certificates. The corporate CA is simply not there.

The Java error is always this one:

javax.net.ssl.SSLHandshakeException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid
certification path to requested target

Look, then import:

$ keytool -list -cacerts -storepass changeit | grep -i labcorp
$ sudo keytool -importcert -noprompt -alias labcorp-root -file /usr/local/share/ca-certificates/labcorp-root.crt -cacerts -storepass changeit
Certificate was added to keystore

keytool manages Java key files: -list shows entries, -importcert adds one, -noprompt skips the "Trust this certificate?" question, -alias is the entry's name, -file the certificate, -storepass the password. -cacerts means "this JDK's cacerts" (use keytool from the same JDK). Two things to remember: the JVM reads the truststore at startup - restart the service - and the JDK's cacerts is replaced on every JDK upgrade, so a hand import is a future outage. Durable options: add the CA wherever the JDK gets installed (the same script or package that installs it), point the JVM at a store you manage with the option -Djavax.net.ssl.trustStore=..., or (newer Temurin builds) enable their "use system CA certificates" option.

mTLS

In mutual TLS (mTLS) the client presents a certificate too, and the server verifies it against its own trusted CA list - so both sides prove who they are. Banks use it for calls between internal services. With curl:

curl --cert client.crt --key client.key --cacert server-ca.crt https://payments.internal/

(--cert our certificate, --key its private key, --cacert the CA to trust for the server's certificate.)

Everything above applies twice: two chains, two sets of expiry dates, two trust stores. The classic mTLS failure is the server's error, not the client's: the client cert expired, or was issued by a CA the server does not list. openssl s_client -connect host:443 -cert client.crt -key client.key shows the "Acceptable client certificate CA names" the server asks for.

What you can now do

Read a certificate file's subject, issuer, dates and SAN with openssl x509.
Prove a chain offline with openssl verify -untrusted, build a fullchain in the right order, and check a key matches its certificate.
Explain "curl works, Java fails" as two trust stores, and fix it with keytool.

Why it helps

This is the day-to-day certificate work on a platform team. A cert rotation ticket: check the new files form a chain with openssl verify, build the fullchain in the right order, confirm the key matches before nginx fails to reload. A monitoring ticket: -checkend in a systemd timer so nobody learns about expiry from users. And the classic enterprise incident: curl works, the Java service throws "PKIX path building failed", because the JDK unpacked in /opt has its own cacerts without the company root.

Knowing where each trust store lives, and that a JDK upgrade silently replaces cacerts, turns a two-day escalation into a ten-minute fix, and lets you suggest the durable version (add the CA wherever the JDK is installed) in review.

Commands in this lesson

openssl echo mkdir cat keytool

FAQ

Why does openssl x509 show only one certificate from my bundle?

Because openssl x509 reads only the first PEM block in a file. A fullchain or CA bundle has several. To list them all use openssl crl2pkcs7 -nocrl -certfile bundle.crt | openssl pkcs7 -print_certs -noout, and check that each issuer= equals the next subject=. grep -c 'BEGIN CERTIFICATE' bundle.crt gives a quick count.

What's the difference between -untrusted and -CAfile in openssl verify?

-untrusted supplies intermediates that openssl may use to build the chain but will not treat as trust anchors. -CAfile replaces the trust store with the roots you give it. By default openssl verify uses the system store. So openssl verify -untrusted issuing-ca.crt leaf.crt answers "does this leaf, with this intermediate, chain to something the system trusts?", which is exactly the pre-deployment check.

I ran update-ca-certificates. Why does Java still fail?

It depends on where the JDK came from. Ubuntu's own OpenJDK packages link cacerts to /etc/ssl/certs/java/cacerts, which is regenerated by that command. A downloaded JDK, a vendor build like Temurin, and SDKMAN installs carry their own cacerts, which the command never touches. Import with keytool -importcert -cacerts using that JDK's keytool, then restart: the JVM reads its truststore only at startup.

How do I check that a key matches a certificate?

Compare the public keys: openssl x509 -in cert.crt -noout -pubkey | sha256sum and openssl pkey -in key.key -pubout | sha256sum. Same hash, same pair. Older guides compare -modulus, which only works for RSA keys; the public-key comparison works for RSA and EC alike. Check before you reload nginx, which otherwise fails with "key values mismatch".

How long are certificates valid these days?

Shorter every year. The CA/Browser Forum reduced the maximum lifetime of public TLS certificates to 200 days from March 2026, 100 days from March 2027 and 47 days by 2029, and ACME issuers like Let's Encrypt already use 90 days or less. Internal CAs set their own rules. Either way, manual renewal does not scale: automate it (ACME is the protocol Let's Encrypt uses to issue and renew automatically, and tools like certbot speak it) and alert on expiry with -checkend.

In an interview Junior

How do you check when a server's certificate expires, and how would you catch expiry before users do?

From the server, without saving anything:

echo | openssl s_client -connect vault.lab:443 2>/dev/null | openssl x509 -noout -subject -enddate

From a file: openssl x509 -in shop.lab.crt -noout -subject -issuer -dates.

For monitoring, -checkend N exits 1 if the certificate expires within N seconds:

openssl x509 -in shop.lab.crt -noout -checkend 604800   # one week

Run it from a systemd timer or cron and alert on a non-zero exit. Every certificate should have an owner and alerts at 30, 14 and 7 days - an expired internal certificate is one of the most common and most preventable outages.

Also asked: A Java service fails with "PKIX path building failed" but curl from the same host works. Why? · How do you check that a private key matches a certificate? · What is mTLS, and when is it used?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.