Chapter 32 Vault & Secrets Management
HashiCorp Vault 2.1 from the operator's and the app team's side: seal and unseal, tokens and TTLs, KV v2 and the data/ path, policies, AppRole for CI, dynamic database credentials and leases, PKI and transit, Vault Agent, Vault on Kubernetes (auth, Agent Injector, External Secrets Operator), audit, backups, and Terraform - and the incidents each of them causes.
In plain words
Imagine a building where every office keeps its own keys in a desk drawer. Copies get made, people leave with them, and when a lock must change nobody knows who has a copy. Now imagine one guarded key room instead: you show your badge, the guard checks a list of what your badge may open, hands you a key that stops working tonight, and writes down that you took it.
Vault is that key room for software. Apps and people log in with an auth method and get a token; policies say which paths the token may read; secrets can be stored (KV), created on demand and expired (dynamic secrets), or never handed out at all (transit encrypts for you). Every request is written to an audit log, and when the server restarts it stays sealed until someone, or a key service, opens it.
Why it matters on call
Every platform team has to answer "where do the passwords live?", and the honest answer is usually "in too many places": environment variables, CI variables, Kubernetes Secrets, a wiki page. Vault is the most common answer in banks and large companies, and the questions about it are the same everywhere: why is this token denied, why did the app stop working after a month, why is Vault sealed after a reboot, who read this secret.
This chapter runs a real-shaped Vault on the lab box: a production server under systemd with TLS and Shamir keys, KV v2 with its data/ path trap, least-privilege policies, AppRole for machines, dynamic database users with leases, a PKI, transit, the Agent, the Kubernetes injector and External Secrets, audit logs and snapshots, and Terraform managing it all. Three incidents and the drills turn the error messages into reflexes.
Lessons
- Why a secrets manager, and your first Vault
- How Vault works: the barrier, seal and unseal, storage, HA
- Tokens and auth methods: TTLs, renewal, the token tree
- KV v2: versions, paths and the put that wipes
- Policies: paths, capabilities, and the rule that wins
- AppRole: logins for CI jobs and services
- Dynamic secrets: database credentials and leases
- PKI and transit: certificates on demand, encryption as a service
- Vault Agent: auto-auth, templates, and apps that never see Vault
- Vault on Kubernetes: the auth method, Agent Injector, ESO and CSI
- Operating Vault: health, audit devices, snapshots, the seal, upgrades, OpenBao
- Terraform and Vault: the vault provider, and secrets in state
23 hands-on labs (missions, incidents and drills) run in the terminal: Open this chapter in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.
Questions people ask
Is this HashiCorp Vault or OpenBao?
The commands and outputs are HashiCorp Vault 2.x (the vault CLI, hvs. tokens). OpenBao, the open-source fork of the last MPL-licensed Vault, has the same API for everything this chapter uses, and its CLI is bao. What you learn transfers to either, and the licence question comes up when a company chooses between them.
Do I need to know Go or HCL to use Vault?
No Go at all. Policies and the server configuration are written in HCL, the same block syntax as Terraform: path "..." { capabilities = [...] } and listener "tcp" { ... } blocks. The Agent's templates use Go template syntax ({{ with secret "..." }}), which the agent lesson explains from scratch.
Why does the lab use a production server and not only dev mode?
Dev mode skips everything that breaks in real life: it is unsealed, in memory, on plain HTTP, with a known root token. The incidents you will meet on call - sealed after a restart, TLS errors, expired tokens, a full audit disk - only exist on a real server, so the chapter starts in dev mode for five minutes and then moves to the systemd-managed one.
What should I be able to do after this chapter?
Read any Vault error and know the next command: 403 means policy or path, 400 missing client token means no token, 503 means sealed. Write least-privilege policies for KV v2, set up AppRole and Kubernetes auth, issue dynamic database credentials, run Vault Agent, debug an injector or ExternalSecret, and operate the server: unseal, audit, snapshot, upgrade.
Which parts are simulated?
The Vault server, the database on pg.lab and the Kubernetes cluster run inside the simulator, with real paths, outputs and error texts. A minimal psql is there only to prove the dynamic users work, and anything sim-only is marked "(simulator)". Commands, flags and file locations match a real Ubuntu box with the HashiCorp package.