Lesson 32.1 · Vault & Secrets Management · 29 min read

Why a secrets manager, and your first Vault

Q: What is wrong with environment variables for secrets?

They are inherited by every child process, readable in /proc/<pid/environ by the same user and root, printed by crash handlers, debug pages and env in a support session, and often logged at start-up. They also never expire. They are convenient, but anything that can see the process environment sees the secret.

Q: What does the VAULT_ADDR variable do?

It tells the vault CLI which server to talk to, scheme included: http://127.0.0.1:8200 for the dev server, https://... for a real one. Without it the CLI assumes https://127.0.0.1:8200. A wrong scheme gives very specific TLS errors, which the chapter shows you on purpose so you recognise them later.

Q: Why look at the HTTP API at all if the CLI works?

Because applications, the Agent, Terraform and curl in an incident all use the API, and every CLI error prints the request it made (URL: GET .../v1/...). Knowing that the CLI is a thin HTTP client lets you translate any command into a request and any error back into a path and a policy.

Q: Why use a secrets manager instead of environment variables or Kubernetes Secrets?

A secrets manager gives one encrypted store with identity-based access: each client logs in, gets a token, and policies decide which paths it may read. Every request lands in an audit trail, and secrets can be short-lived or dynamic, created per client and revoked when the lease ends. Environment variables leak through /proc/<pid/environ, crash dumps, child processes and logs, and never expire. Kubernetes Secrets are base64, readable by anyone with get secrets, and have no expiry or read audit. Neither lets you answer "who read this password last week?" or revoke one client's access without changing the secret for everyone.

In plain words

Imagine everyone in a shared flat writes the Wi-Fi password, the door code and the bank PIN on sticky notes and leaves them around the kitchen. It works until a guest reads them, a flatmate moves out, or the PIN changes and nobody knows which notes are stale. A safe with a combination only some flatmates know, and a notebook next to it where every opening is written down, fixes all three.

A secrets manager is that safe for software. Instead of passwords in environment variables, config files and Kubernetes Secrets, there is one encrypted store; an app proves who it is and gets only what its policy allows; every read is in an audit log; and secrets can expire on their own. In this lesson you start Vault in dev mode, write your first secret and see that every CLI command is just an HTTP call.

Why this chapter exists

Count the places the orders database password lives at a typical company. It is in /etc/orders/app.conf on three servers, in a Kubernetes Secret, in the CI system's variables, in Terraform state, in two people's shell history and in a chat message from last year when someone "just needed it quickly". When that person leaves, nobody rotates it, because nobody knows every place it is used. When it leaks, nobody can tell who read it.

A secrets manager fixes this by being the one place secrets live, and by changing the question from "who knows the password?" to "who is allowed to ask for it, and who did?". HashiCorp Vault is the one most platform teams run - in banks very often on their own servers, next to cloud key vaults. This chapter is Vault from both sides: the operator who keeps it running, and the application team that gets secrets out of it without ever seeing a password.

What you need to know already: systemd units, Environment= and why environment variables are not secret (2.21); file permissions (4.5); curl and HTTP status codes (9.21); TLS certificates and CAs (9.15); JSON and jq (7.11); Kubernetes Secrets and ServiceAccounts (15.33, 17.33); Terraform state (13.1).

The words you need first

Secret - anything that grants access: a password, an API key, a token, a private key, a certificate's key. If someone who has it can pretend to be you or your service, it is a secret.
Static secret - a secret that is created once and stays valid until someone changes it by hand: the database password in a config file.
Dynamic secret - a secret that is created on request, for one client, and expires on its own: a database user that exists for one hour.
Rotation - replacing a secret with a new one so the old one stops working.
Blast radius - how much an attacker gets from one leaked secret.
Audit trail - a record of who accessed what, and when.

Where secrets end up without a secrets manager

On this box, the orders service gets its password from a config file:

$ ls -l /etc/orders/app.conf
-rw-r----- 1 root appuser 256 Sep 14 17:43 /etc/orders/app.conf

That is the good version: mode 640, readable by root and the appuser group only (4.5). Even so: every backup of /etc has the password, every admin with sudo can read it, and nothing records who did. The usual places, worst first:

where	what goes wrong
git history	public forever once pushed; rotating is the only fix
CI variables and logs	a `set -x` or `echo` prints it; every pipeline admin can read it
environment variables	`systemctl show -p Environment`, `/proc/PID/environ`, crash dumps, child processes inherit them (2.21)
Kubernetes Secrets	base64 is an encoding, not encryption; anyone with `get secrets` in the namespace reads them
Terraform state	every attribute, in plain text (13.1)
container images	a `COPY .env` or `ENV PASSWORD=` stays in a layer forever (10.40)
a wiki page or chat	no access control at all

The Kubernetes row deserves one more look, because it surprises people:

# an illustration (no ▶): what anyone allowed to "get secrets" sees
$ kubectl get secret orders-db -o jsonpath='{.data.password}' | base64 -d
orders-static-2023

Kubernetes stores Secrets in etcd, and unless the cluster has encryption at rest configured, in plain text there too. Secrets are still the right delivery mechanism to a pod - the question this chapter answers is where the value comes from and who controls it.

What a secrets manager adds

A secrets manager such as Vault gives you five things a config file never can:

One encrypted store. Secrets are encrypted before they reach disk. A copy of the disk, or a backup, is useless without the key.
Identity before access. Every client first proves who it is (authentication): a person with a password or SSO, a CI job with a role, a pod with its Kubernetes ServiceAccount token. Then a policy decides what that identity may read or write (authorization).
An audit trail. Every request - who, which path, when, allowed or denied - goes to an audit log. "Who read the payments API key last Tuesday?" has an answer.
Short lifetimes. Everything Vault hands out has a time to live (TTL). Access tokens expire. Dynamic database passwords are deleted from the database when their time is up. A leaked credential that expires in an hour is a much smaller problem than one that never does.
Revocation. One command takes back a credential - or every credential a token ever received.

On top of storing secrets, Vault can generate them (database users, cloud credentials, TLS certificates from its own certificate authority) and use them without handing them out (encrypting data for an application that never sees the key). Those are later lessons of this chapter.

Vault in one picture

Every interaction with Vault is the same shape:

A client (the vault CLI, an application, curl, Terraform) sends an HTTP request to the server, to a path like secret/data/orders/db.
The request carries a token - Vault's word for "the proof of who you are", a long random string that starts with hvs.. You get a token by logging in through an auth method (userpass, AppRole, Kubernetes, OIDC...).
The server looks at the policies attached to the token and checks that they allow this operation on this path.
The path decides which secrets engine handles it: secret/ is a key/value store, database/ creates database users, pki/ issues certificates. Engines are mounted at paths, like file systems (4.1).

That is all of Vault: paths, tokens, policies, engines. The CLI is a thin wrapper around the HTTP API - which you can see for yourself, later in this lesson.

Which tool, and which licence

HashiCorp Vault - the original, written in Go. Since August 2023 it is under the Business Source License (BSL 1.1): free to use, including in production, but you may not offer a competing hosted product. IBM completed its purchase of HashiCorp in February 2025. The current release is Vault 2.1.1 (September 2026); 2.0 (April 2026) was the first major version after the 1.x line ended at 1.21.
OpenBao - the community fork of Vault 1.14, the last open-source (MPL 2.0) version, run under the Linux Foundation. Its CLI is bao instead of vault, its API and most commands are the same. The current release is OpenBao 2.7. Some banks choose it for the licence; the skills in this chapter apply to both.
Cloud secret stores - AWS Secrets Manager, GCP Secret Manager and each cloud's own key vault service. Simpler, managed for you, tied to one cloud. Vault earns its place when you run on-prem or across clouds, need dynamic credentials or your own PKI, or want one access model for everything.
HCP Vault Dedicated - Vault run for you by HashiCorp.

(simulator) The lab's vault binary is version 2.1.1 and comes from HashiCorp's apt repository. On a real Ubuntu server you install it the way HashiCorp documents:

# an illustration (no ▶): the package repository HashiCorp publishes
$ wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
$ echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(grep -oP '(?<=UBUNTU_CODENAME=).*' /etc/os-release || lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
$ sudo apt update && sudo apt install vault

The same binary is the CLI, the server and the agent:

$ vault version
Vault v2.1.1 (c41f2e8b0d7a9356e1b4f0c2d8a6e3b9f7c5a1d4), built 2026-09-16T10:41:32Z

Your first server: dev mode

Before any production setup, Vault has a dev mode: one command, an in-memory server, already unsealed (the next lesson explains sealing), with a root token printed for you. It exists for exactly two things - learning and local development.

Asking a server that is not running gives the first error you will see a hundred times:

Quick check

$ vault status

No Vault server runs on this box yet. What does vault status print?

Sealed: true, because a stopped server counts as sealed
connection refused on https://127.0.0.1:8200
a usage message, because no -address was given

Answer

connection refused on https://127.0.0.1:8200 - With no VAULT_ADDR the CLI talks to https://127.0.0.1:8200 and nothing listens there.

Note the address in the error: https://127.0.0.1:8200 is the CLI's default. Start a dev server in the background (the & runs it as a background job, so the terminal stays yours):

$ vault server -dev -dev-root-token-id=root &
[1] 17385
==> Vault server configuration:

Administrative Namespace:
             Api Address: http://127.0.0.1:8200
                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Go Version: go1.25.1
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", ..., tls: "disabled")
                   Mlock: supported: true, enabled: false
                 Storage: inmem
                 Version: Vault v2.1.1, built 2026-09-16T10:41:32Z

==> Vault server started! Log data will stream in below:

2026-09-22T20:00:03.700Z [INFO]  core: security barrier initialized: stored=1 shares=1 threshold=1
2026-09-22T20:00:03.700Z [INFO]  core: vault is unsealed
2026-09-22T20:00:03.700Z [INFO]  core: successful mount: namespace="" path=secret/ type=kv version=""
WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.

You may need to set the following environment variables:

    $ export VAULT_ADDR='http://127.0.0.1:8200'

Unseal Key: aN5XpvH4ggL1SvfMTjse1ngkYHRHOQ9f2bf7Oddmveg=
Root Token: root

Development mode should NOT be used in production installations!

Read it line by line:

Storage: inmem - everything lives in the process's memory. Stop the server and every secret is gone.
tls: "disabled" and Api Address: http://... - dev mode speaks plain HTTP. The CLI defaults to HTTPS, hence the hint to export VAULT_ADDR='http://127.0.0.1:8200'. Skip it and you get http: server gave HTTP response to HTTPS client.
-dev-root-token-id=root chose the root token's value. A root token can do anything, and in production you almost never use one (the tokens lesson).
"already authenticated to the CLI": dev mode wrote the root token to ~/.vault-token, the file the CLI reads when VAULT_TOKEN is not set. That file is the CLI's token helper.
A KV secrets engine is already mounted at secret/.

$ export VAULT_ADDR=http://127.0.0.1:8200
$ cat ~/.vault-token; echo
root

The first secret

vault kv put writes a key/value secret. KV stands for key/value: a secret is a small set of named values.

$ vault kv put secret/hello target=world
== Secret Path ==
secret/data/hello

======= Metadata =======
Key                Value
---                -----
created_time       2026-09-22T20:00:04.904634776Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Two surprises, both explained in the KV lesson: the secret's real path is secret/data/hello (with data/ in the middle), and it has a version - write it again and you get version 2, with version 1 still readable.

$ vault kv get -field=target secret/hello
world

-field prints one value and nothing else - with no newline when the output goes to a pipe or $(...), so PW=$(vault kv get -field=password secret/app/db) works in scripts.

Every command is an HTTP call

The CLI can show you the request it would send instead of sending it:

$ vault kv get -output-curl-string secret/hello
curl -H "X-Vault-Request: true" -H "X-Vault-Token: $(vault print token)" http://127.0.0.1:8200/v1/secret/data/hello

The API is /v1/ + the path, the token travels in the X-Vault-Token header, and the answer is JSON:

$ curl -s -H "X-Vault-Token: root" $VAULT_ADDR/v1/secret/data/hello | jq .data
{
  "data": {
    "target": "world"
  },
  "metadata": {
    "created_time": "2026-09-22T20:00:04.904634776Z",
    "custom_metadata": null,
    "deletion_time": "",
    "destroyed": false,
    "version": 1
  }
}

Applications that do not use a Vault library do exactly this: one HTTP GET with a token.

Dev mode is a toy

Stop the server (kill %1, the job number from &) and start it again: secret/hello is gone, the unseal key and root token are new. Dev mode has no persistence, no TLS, a root token in a file, and an unseal key printed to the terminal. Never put it on a network, never put real secrets in it. The next lesson builds a real server.

Dev mode still earns its place, on a laptop and in tests:

flag	what it does
`-dev-root-token-id=root`	a root token you choose instead of a random one (scripts and CI tests use this)
`-dev-listen-address=0.0.0.0:8200`	listen on more than 127.0.0.1 - exactly what you should not do outside a container
`-dev-tls`	dev mode with TLS: Vault writes a throwaway CA and certificate to a temporary directory and prints the `VAULT_CACERT` line to use
`-dev-no-store-token`	do not write the root token to `~/.vault-token`

The pattern you will see in pipelines is a dev server started in a container for the length of an integration test: the test writes the secrets it needs, the app under test reads them, the container is thrown away. Nothing in it outlives the job, which is the one situation where "in memory, no TLS, known root token" is a feature.

Quick check

$ vault kv get -field=target secret/hello

You never started a server in this lesson's box (or you stopped it). What happens?

it prints world: the CLI caches secrets in ~/.vault-token
an error: nothing listens on the address
it prints an empty line

Answer

an error: nothing listens on the address - The CLI keeps nothing but the token; every read is a request to the server.

In an interview: "Why use a secrets manager instead of environment variables or Kubernetes Secrets?" - one encrypted store, identity-based access with policies, an audit trail, short-lived and dynamic credentials, revocation. Environment variables leak through /proc, crash dumps and child processes; Kubernetes Secrets are base64, readable by anyone with get secrets; neither has an audit trail or expiry.

What you can now do

List where secrets leak without a secrets manager, and what each place costs.
Name what Vault adds: encrypted storage, authentication + policies, audit, TTLs, revocation, generated secrets.
Describe a Vault request: client, path, token, policy, secrets engine.
Start a dev server, set VAULT_ADDR, write and read a KV secret, and see the HTTP call behind a CLI command.

Why it helps

Leaked credentials are one of the most common starting points of real breaches, and they usually leak from exactly the places this lesson lists: environment variables visible in /proc, CI logs, git history, Kubernetes Secrets that anyone with get secrets can read. Knowing why each of those is weak, and what a secrets manager adds (identity, policy, audit, expiry, revocation), is the first thing a platform interview asks about Vault.

Seeing that vault kv get is a GET /v1/secret/data/... with an X-Vault-Token header also pays off for the rest of the chapter: every error message names the URL and status code, and reading those is most of debugging Vault.

Commands in this lesson

ls kubectl wget echo apt vault export cat curl

FAQ

Are Kubernetes Secrets not already secret?

They are base64-encoded, which is an encoding, not encryption: base64 -d reverses it. Anyone with get secrets in the namespace can read them, etcd holds them unless encryption at rest is configured, and there is no expiry or record of who read what. They are fine as a delivery format if the value comes from a real secrets manager.

What is wrong with environment variables for secrets?

They are inherited by every child process, readable in /proc/<pid>/environ by the same user and root, printed by crash handlers, debug pages and env in a support session, and often logged at start-up. They also never expire. They are convenient, but anything that can see the process environment sees the secret.

Why does dev mode print a root token?

Dev mode exists for trying things on a laptop: it initialises and unseals itself, keeps everything in memory, listens on plain HTTP and uses a root token you can type (root in the lab). None of that is acceptable on a server, which is why the next lesson builds a real one with storage, TLS and unseal keys.

What does the VAULT_ADDR variable do?

It tells the vault CLI which server to talk to, scheme included: http://127.0.0.1:8200 for the dev server, https://... for a real one. Without it the CLI assumes https://127.0.0.1:8200. A wrong scheme gives very specific TLS errors, which the chapter shows you on purpose so you recognise them later.

Why look at the HTTP API at all if the CLI works?

Because applications, the Agent, Terraform and curl in an incident all use the API, and every CLI error prints the request it made (URL: GET .../v1/...). Knowing that the CLI is a thin HTTP client lets you translate any command into a request and any error back into a path and a policy.

In an interview Mid

Why use a secrets manager instead of environment variables or Kubernetes Secrets?

A secrets manager gives one encrypted store with identity-based access: each client logs in, gets a token, and policies decide which paths it may read. Every request lands in an audit trail, and secrets can be short-lived or dynamic, created per client and revoked when the lease ends.

Environment variables leak through /proc/<pid>/environ, crash dumps, child processes and logs, and never expire. Kubernetes Secrets are base64, readable by anyone with get secrets, and have no expiry or read audit. Neither lets you answer "who read this password last week?" or revoke one client's access without changing the secret for everyone.

Also asked: What is the difference between encoding and encryption? · Where have you seen secrets leak in a real project? · What are the downsides of running your own secrets manager?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.