OnCallReady

Lesson 32.21 · Vault & Secrets Management · 26 min read

PKI and transit: certificates on demand, encryption as a service

In plain words

Think of a passport office. The government (the root CA) does not stamp every passport itself; it authorises regional offices (intermediate CAs), and those issue passports with an expiry date. Each office has rules about who may get what (roles). If passports only last a few days, a stolen one is soon worthless and nobody needs a long list of cancelled passports.

Vault's PKI engine is a passport office for TLS certificates: a root, an intermediate it signs, roles that limit names and lifetimes, and issue that returns a certificate and key in one call. The transit engine is different: a notary who stamps and unstamps documents for you, so you store only the stamped (encrypted) version and never hold the key yourself.

Why this lesson exists

Two more things Vault does that have nothing to do with storing passwords. PKI: it is a certificate authority, so every internal service can get a TLS certificate in one API call - valid for days, not years, so expiry stops being a yearly outage. Transit: it encrypts and decrypts data for applications that never see the key, so a stolen database dump of card numbers is ciphertext. Both come up in bank platform teams constantly, and both have operational gotchas of their own.

What you need to know already: certificates, chains, CAs and openssl x509 (9.15, 9.17); the expiry incident from the TLS chapter; leases and -field (earlier in this chapter); base64 (15.33).

The words you need first

PKI: a two-level CA in Vault

The shape every guide recommends: a root CA that only signs intermediates (and in a real company lives offline or in its own mount, rarely touched), and an intermediate that issues the day-to-day certificates. If the intermediate is ever compromised you revoke and replace it; the root - which every client trusts - stays.

The root

$ vault secrets enable pki
Success! Enabled the pki secrets engine at: pki/
$ vault secrets tune -max-lease-ttl=87600h pki
Success! Tuned the secrets engine at: pki/
$ vault write -field=certificate pki/root/generate/internal common_name="oncall-lab Root CA" issuer_name=root-2026 ttl=87600h > /tmp/root_ca.crt
$ openssl x509 -in /tmp/root_ca.crt -noout -subject -issuer -dates
subject=CN=oncall-lab Root CA
issuer=CN=oncall-lab Root CA
notBefore=Sep 22 19:59:33 2026 GMT
notAfter=Sep 19 20:00:03 2036 GMT

The intermediate

$ vault secrets enable -path=pki_int pki
$ vault secrets tune -max-lease-ttl=43800h pki_int
$ vault write -field=csr pki_int/intermediate/generate/internal common_name="oncall-lab Intermediate CA 2026" > /tmp/int.csr
$ vault write -field=certificate pki/root/sign-intermediate csr=@/tmp/int.csr format=pem_bundle ttl=43800h > /tmp/int.crt
$ vault write pki_int/intermediate/set-signed certificate=@/tmp/int.crt
Key                 Value
---                 -----
existing_issuers    <nil>
existing_keys       <nil>
imported_issuers    [59281aeb-c3f8-e17a-0d4c-e2748adce9ad]
imported_keys       [634fdb3e-513a-073a-ffa6-05fca105930d]
mapping             map[59281aeb-c3f8-e17a-0d4c-e2748adce9ad:634fdb3e-513a-073a-ffa6-05fca105930d]

The standard four steps: the intermediate mount makes a key and a CSR, the root signs it (five years), the signed certificate goes back to the intermediate mount, which now has an issuer paired with its key. Exactly what you would do with openssl by hand (9.17) - except no private key ever touched a disk.

Roles: what may be issued

$ vault write pki_int/roles/internal allowed_domains=oncall-lab.internal allow_subdomains=true max_ttl=72h
Success! Data written to: pki_int/roles/internal

A role is the policy of the CA: *.oncall-lab.internal names only, at most 72 hours. Who may use the role is a Vault policy on pki_int/issue/internal - the orders team gets update on that path and nothing else.

Issuing

$ vault write -format=json pki_int/issue/internal common_name=orders.oncall-lab.internal alt_names=orders-api.oncall-lab.internal ttl=24h > /tmp/orders.json
$ jq -r .data.certificate /tmp/orders.json > /tmp/orders.crt
$ jq -r .data.private_key /tmp/orders.json > /tmp/orders.key
$ openssl x509 -in /tmp/orders.crt -noout -subject -issuer -dates -ext subjectAltName
subject=CN=orders.oncall-lab.internal
issuer=CN=oncall-lab Intermediate CA 2026
notBefore=Sep 22 19:59:34 2026 GMT
notAfter=Sep 23 20:00:04 2026 GMT
X509v3 Subject Alternative Name:
    DNS:orders.oncall-lab.internal, DNS:orders-api.oncall-lab.internal

The response holds everything a server needs: certificate, private_key, issuing_ca, ca_chain, serial_number, expiration. The private key is generated for this request and not stored by Vault - save it now or issue again. (sign/<role> signs a CSR instead, when the key must be generated on the server itself.)

Note the notBefore thirty seconds in the past: Vault backdates certificates slightly so a client whose clock is a little behind does not reject a brand-new certificate as "not yet valid".

The role enforces its rules:

$ vault write pki_int/issue/internal common_name=orders.example.com
Error writing data to pki_int/issue/internal: Error making API request.

URL: PUT https://127.0.0.1:8200/v1/pki_int/issue/internal
Code: 400. Errors:

* 1 error occurred:
	* common name orders.example.com not allowed by this role

A TTL above the role's max is cut down, with a warning:

* TTL "720h" is longer than permitted maxTTL "72h", so maxTTL is being used

And nothing outlives its CA: asking for a certificate that would expire after the issuing CA does fails with cannot satisfy request, as TTL would result in notAfter ... that is beyond the expiration of the CA certificate.

Short certificates, revocation, CRLs

The point of a Vault PKI is short-lived certificates: 24-72 hours, renewed automatically (Vault Agent templates, cert-manager's Vault issuer in Kubernetes). A certificate that expires tomorrow does not need revoking when a server is decommissioned

year.

When you do need to revoke:

$ vault write pki_int/revoke serial_number=13:8a:33:53:18:2b:95:48:57:af:c6:fd:47:25:31:1f:11:cf:d4:36
Key                        Value
---                        -----
revocation_time            1790107204
revocation_time_rfc3339    2026-09-22T20:00:04.602243238Z
state                      revoked

The serial goes on the CRL, which clients fetch - without a token - from /v1/pki_int/crl/pem (and the CA itself from /v1/pki_int/ca/pem). Revocation only works for clients that check the CRL (or OCSP); many do not. Another reason to prefer short lifetimes. vault list pki_int/certs lists serials of issued certificates (unless the role has no_store); vault write pki_int/tidy tidy_cert_store=true tidy_revoked_certs=true cleans out expired ones, which matters on a busy mount.

Transit: encryption as a service

An application must store card numbers (PANs) in its database. Encrypting them in the app means the app holds the key - and so does every backup of its config. With transit the app sends the plaintext to Vault and stores what comes back.

$ vault secrets enable transit
Success! Enabled the transit secrets engine at: transit/
$ vault write -f transit/keys/cards
Key                       Value
---                       -----
allow_plaintext_backup    false
auto_rotate_period        0s
deletion_allowed          false
derived                   false
exportable                false
imported_key              false
keys                      map[1:1790107204]
latest_version            1
min_available_version     0
min_decryption_version    1
min_encryption_version    0
name                      cards
supports_decryption       true
supports_derivation       true
supports_encryption       true
supports_signing          false
type                      aes256-gcm96

aes256-gcm96 is the default key type. exportable false - the key can never be read out of Vault; deletion_allowed false - it cannot be deleted by accident (that would make every ciphertext garbage).

Transit works on base64, so binary data travels safely in JSON:

$ vault write transit/encrypt/cards plaintext=$(echo -n "4111 1111 1111 1111" | base64)
Key            Value
---            -----
ciphertext     vault:v1:27nN3cSz27F++EXmwFzJhtNcxhXHTcrsxFzae8Fcw2vvbfDY8m31kfFt8/5PRc0=
key_version    1

vault:v1: says which key version encrypted it. Decrypting gives base64 back:

$ vault write -field=plaintext transit/decrypt/cards ciphertext=$CT | base64 -d; echo
4111 1111 1111 1111

Forget the base64 step and Vault refuses: * failed to base64-decode plaintext. Forget it on the way back and you store NDExMSAxMTEx... and wonder why the numbers look odd.

That also means you cannot search the database for a card number by its ciphertext. If you need lookups, store an HMAC alongside (transit/hmac/cards), which is deterministic.

Rotating and rewrapping

$ vault write -f transit/keys/cards/rotate
$ vault write transit/rewrap/cards ciphertext=$CT
Key            Value
---            -----
ciphertext     vault:v2:od8qUlocoh5p8PnQRRfcHVIX345CBsFjRRfR1FwX1+BuJuhzbybqBnAm65kWTxg=
key_version    2

Policies for transit

The app needs exactly two paths:

path "transit/encrypt/cards" {
  capabilities = ["update"]
}
path "transit/decrypt/cards" {
  capabilities = ["update"]
}

Not transit/keys/cards (key management), not rotate. The batch job that rewraps gets transit/rewrap/cards - and no decrypt. Separation of duties in four lines.

In an interview: "How would you protect card numbers in a database so a stolen dump is useless?" - encrypt them through Vault's transit engine: the app sends base64 plaintext to transit/encrypt/<key> and stores the vault:v1:... ciphertext; the key never leaves Vault, the app's policy allows only encrypt/decrypt on that key, rotation adds versions and rewrap upgrades old rows without exposing plaintext.

What you can now do

Why it helps

Running an internal CA by hand means long-lived certificates, spreadsheets of expiry dates and outages when one is forgotten. Vault's PKI makes short-lived certificates practical, which is why it sits behind many service meshes and cert-manager setups. Reading a certificate's issuer, chain and dates with openssl is a daily on-call skill anyway.

Transit answers a different security question: how do you protect sensitive columns, like card numbers, so a stolen database dump is useless? Encrypting through Vault keeps the key out of the app and the database, and key rotation and rewrap are standard compliance requirements.

Commands in this lesson

vault openssl jq

FAQ

Why have an intermediate CA instead of issuing from the root?

So the root's key is used rarely and can be kept offline or tightly guarded. If an intermediate is compromised, you revoke it and create a new one signed by the same root, and clients that trust the root keep working. Rebuilding trust in a new root means updating every client.

Why are short-lived certificates better than revocation?

Revocation depends on clients checking CRLs or OCSP, which many do badly or not at all. A certificate valid for 72 hours expires before most revocation would even propagate. Clients renew automatically (Vault Agent, cert-manager), so short lifetimes cost nothing once renewal is in place.

What does a PKI role control?

What may be issued under it: allowed_domains and whether subdomains, bare domains or IP SANs are allowed, the key type and size, max_ttl, and key usages. A client with update on pki_int/issue/web can only get certificates the web role allows.

What does vault:v2: in a transit ciphertext mean?

The ciphertext was produced by version 2 of the transit key. Rotating the key adds a version; new encryptions use the latest one, while older ciphertexts still decrypt as long as their version is at or above min_decryption_version. transit/rewrap re-encrypts old ciphertexts with the newest version.

Why does transit want base64 plaintext?

Because the API is JSON and the plaintext can be any bytes, not only text. You encode before encrypting (base64 <<< "...") and decode after decrypting (base64 -d). Forgetting the decode is the most common transit mistake: the "plaintext" you get back is still base64.

In an interview Mid

How would you protect card numbers in a database so that a stolen dump is useless?

Encrypt them with Vault's transit engine: the app sends the base64 plaintext to transit/encrypt/<key> and stores only the returned vault:v1:... ciphertext; to read, it calls transit/decrypt/<key>. The key never leaves Vault, so the database and its backups hold nothing usable on their own.

The app's policy allows only update on encrypt and decrypt for that one key, and every call is audited. Rotation (transit/keys/<key>/rotate) adds versions; transit/rewrap upgrades old rows to the newest version without the app seeing the plaintext, and min_decryption_version retires old versions.

Also asked: Why would you use an intermediate CA in Vault rather than issuing from the root? · How do you check which CA issued a certificate and when it expires? · What is the difference between encryption in transit and encryption as a service?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.