OnCallReady

Lesson 32.18 · Vault & Secrets Management · 24 min read

Dynamic secrets: database credentials and leases

In plain words

Imagine a hotel that, instead of giving every guest a copy of one master key, cuts a brand-new key for each guest at check-in, valid until their checkout. If a guest loses theirs, only that one key matters, and it stops working at checkout anyway. Nobody ever needs to change the locks for everyone.

Dynamic secrets are that hotel. Vault holds one privileged database connection; when an app reads database/creds/<role>, Vault creates a new database user with a random password and hands it out with a lease. The app renews the lease while it needs it; when the lease expires or is revoked, Vault drops the user. Even the bootstrap password Vault was given is rotated away with rotate-root, so in the end nobody knows a database password at all.

Why this lesson exists

A static database password has three problems no storage can fix: everyone who ever saw it still knows it, every service shares it so you cannot tell who did what, and rotating it means a coordinated change in every app at once. Dynamic secrets remove all three: Vault creates a new database user for each client, with a password nobody else has, that expires on its own. The price is that you have to understand leases - because when a lease ends, the database user is dropped, and the app had better have a new one.

What you need to know already: tokens, TTLs and the token tree (the tokens lesson); policies (the policies lesson); AppRole logins (the previous lesson); SQL roles and GRANT at the level of "a role can be granted another role's privileges"; jq (7.11).

The words you need first

Setting it up (the platform team's part)

Mount the engine, then tell it how to reach PostgreSQL on pg.lab (simulator: a lab database with an orders schema):

$ vault secrets enable database
Success! Enabled the database secrets engine at: database/
$ vault write database/config/orders plugin_name=postgresql-database-plugin connection_url="postgresql://{{username}}:{{password}}@pg.lab:5432/orders?sslmode=require" username=vault password=vault-bootstrap-9c1e allowed_roles=orders-ro
Success! Data written to: database/config/orders
* error creating database object: error verifying connection: failed to connect to `user=vault database=orders`: 10.0.3.44:5432 (pg.lab): failed SASL auth: FATAL: password authentication failed for user "vault" (SQLSTATE 28P01)

Then a role - the SQL that runs for every credential:

$ vault write database/roles/orders-ro db_name=orders creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT orders_ro TO \"{{name}}\";" default_ttl=1h max_ttl=4h
Success! Data written to: database/roles/orders-ro

Getting a credential (the app's part)

$ vault read database/creds/orders-ro
Key                Value
---                -----
lease_id           database/creds/orders-ro/6eQKPsOQKtLL1oRd40PQXfCE
lease_duration     1h
lease_renewable    true
password           2AtQFf-76KWOwkP3iDQk
username           v-token-la-orders-r-I0l6cSmXZoorU0fDWnoT-1790107203

The username is built from a template: v-, the token's display name (8 characters: token-la from token-lab-admin), the role name (8: orders-r), 20 random characters, and the Unix time. You can tell from a database log line which Vault role and roughly which client created a user. It works like any user:

$ PGPASSWORD=$(jq -r .data.password /tmp/creds.json) psql -h pg.lab -U $(jq -r .data.username /tmp/creds.json) -d orders -c 'SELECT count(*) FROM orders;'
 count
-------
 18342
(1 row)

And the database shows what Vault did:

$ PGPASSWORD=vault-bootstrap-9c1e psql -h pg.lab -U vault -d orders -c "\du"
                                   List of roles
                      Role name                      |                 Attributes
-----------------------------------------------------+---------------------------------------------
 orders_app                                          |
 orders_ro                                           | Cannot login
 postgres                                            | Superuser, Create role
 v-token-la-orders-r-I0l6cSmXZoorU0fDWnoT-1790107203 | Password valid until 2026-09-22 21:00:03+00
 v-token-la-orders-r-vArdzEIrP3ZkFvUnfnig-1790107203 | Password valid until 2026-09-22 21:00:03+00
 vault                                               | Create role

Two reads, two users. Each instance of an app gets its own; if one leaks, you revoke that one without touching the others, and the database's own logs say which instance did what.

Leases

Every dynamic credential comes with a lease. Leases are listed by path (a sudo path):

$ vault list sys/leases/lookup/database/creds/orders-ro
Keys
----
2rqS7GFpiIr1jLmaLcNEwL0n
6eQKPsOQKtLL1oRd40PQXfCE
$ vault lease lookup database/creds/orders-ro/2rqS7GFpiIr1jLmaLcNEwL0n
Key             Value
---             -----
expire_time     2026-09-22T21:00:03.902099938Z
id              database/creds/orders-ro/2rqS7GFpiIr1jLmaLcNEwL0n
issue_time      2026-09-22T20:00:03.902699938Z
last_renewal    <nil>
renewable       true
ttl             59m59s

Renewing works exactly like a token's: from now, capped by the role's max_ttl:

$ vault lease renew -increment=8h database/creds/orders-ro/2rqS7GFpiIr1jLmaLcNEwL0n
WARNING! The following warnings were returned from Vault:

  * TTL of "8h" exceeded the effective max_ttl of "3h59m59s"; TTL value is
  capped accordingly

Key                Value
---                -----
lease_id           database/creds/orders-ro/2rqS7GFpiIr1jLmaLcNEwL0n
lease_duration     3h59m59s
lease_renewable    true

After 4 hours no renewal helps: the client must read database/creds/orders-ro again, get a new username and password, and reconnect. A database connection pool that cannot swap credentials at runtime will fail at that moment - design for it (Vault Agent re-rendering the credentials file, or the app re-reading them).

Revoking

$ vault lease revoke database/creds/orders-ro/2rqS7GFpiIr1jLmaLcNEwL0n
All revocation operations queued successfully!
$ PGPASSWORD=... psql -h pg.lab -U v-token-la-orders-r-vArdzEIrP3ZkFvUnfnig-1790107203 -d orders -c 'SELECT 1;'
psql: error: connection to server at "pg.lab" (10.0.3.44), port 5432 failed: FATAL:  password authentication failed for user "v-token-la-orders-r-vArdzEIrP3ZkFvUnfnig-1790107203"

The user is gone from the database. Note the error: PostgreSQL says "password authentication failed" for a user that does not exist (or whose VALID UNTIL has passed), on purpose - it does not reveal which usernames exist. When an app with Vault credentials suddenly gets this error, the first question is "has its lease ended?".

vault lease revoke -prefix database/creds/orders-ro revokes every credential of the role at once (it needs sudo on sys/leases/revoke-prefix/...) - the move when a role's credentials are suspected compromised, or before changing what the role grants.

Leases belong to tokens

A lease is owned by the token that created it. Revoke the token - or let it hit its max TTL - and every lease it owns is revoked too:

$ T=$(vault token create -policy=lab-admin -ttl=5m -field=token)
$ VAULT_TOKEN=$T vault read -field=username database/creds/orders-ro
v-token-orders-r-7TKpbT1LVFTYZQb4bZDm-1790107205
$ vault token revoke $T
Success! Revoked token (if it existed)

The lease (and the database user) went with the token. That is the design: one revoke cleans up everything a compromised client had. It is also the mechanism behind a classic outage - an app whose token expires loses its database access at the same moment as its Vault access.

Rotating the root credentials

The vault database user's password was typed into Vault by a person, so it is a static secret again. Fix that:

$ vault write -f database/rotate-root/orders
Success! Data written to: database/rotate-root/orders
$ PGPASSWORD=vault-bootstrap-9c1e psql -h pg.lab -U vault -d orders -c "SELECT 1"
psql: error: connection to server at "pg.lab" (10.0.3.44), port 5432 failed: FATAL:  password authentication failed for user "vault"

Vault set a new password for its own admin user and now is the only one who knows it. New credentials keep working. Do this right after configuring a connection - and know that from then on, if Vault's storage is lost, so is that password (you reset it in the database by hand).

Static roles, and other engines

In an interview: "What is a dynamic secret and why is it better than a rotated static password?" - Vault creates a unique credential per client on request (vault read database/creds/<role>), with a lease; when the lease expires or is revoked, Vault deletes the credential (the database user is dropped). No shared password, every client is traceable, a leak is limited to one credential for a short time, and "rotation" is just the next read.

What you can now do

Why it helps

Static database passwords are the secrets that leak most and get rotated least, because rotating one means changing every app at once. Dynamic credentials remove the problem: every client has its own user (visible in the database's own logs), a leak is limited to one short-lived credential, and rotation is simply the next read.

They also explain a class of incidents you will see: credentials that stop working at the same time every night because a lease or the token that owns it hit its max TTL. Knowing that leases belong to tokens, and how to look them up and revoke them by prefix, is what makes those quick to solve.

Commands in this lesson

vault

FAQ

What is a lease?

Vault's record of a secret it handed out with a lifetime: an ID, a TTL, whether it can be renewed. Dynamic secrets always have one. vault lease renew <id> extends it up to the role's max_ttl; vault lease revoke <id> (or -prefix) ends it early, which makes Vault run the revocation statements.

Why did my credentials die when my token expired?

Every lease belongs to the token that created it. When that token expires or is revoked, Vault revokes all of its leases, and the database users are dropped. A client that renews its leases but lets its token reach its max TTL loses both; the fix is to log in again and request new credentials.

What does rotate-root do, and is it dangerous?

It changes the password of the database user Vault connects with, to a value only Vault knows. That is the point: the bootstrap password in a README or ticket becomes useless. The danger is that nobody can log in as that user afterwards, so do not use a shared admin account as Vault's connection user.

What are static roles?

For accounts that must keep their name, such as an app user an older system expects, Vault can manage the password of an existing database user and rotate it on a schedule (rotation_period). Clients read the current password from database/static-creds/<role>. It is rotation, not a user per client.

Does this only work for PostgreSQL?

No. The database engine has plugins for MySQL, MSSQL, Oracle, MongoDB, Redis and others, and the same lease model drives other dynamic engines: cloud IAM credentials (AWS, GCP and others), SSH certificates, RabbitMQ users. The setup differs per backend; reading, renewing and revoking are the same.

In an interview Mid

What is a dynamic secret, and why is it better than a regularly rotated static password?

A dynamic secret is created for one client at the moment it asks: vault read database/creds/<role> makes Vault run the role's creation statements and return a new database user and password with a lease. The client renews the lease while it runs; when it expires or is revoked - also when the token that owns it is revoked - Vault drops the user.

Compared with a rotated shared password: no credential is shared, every client is traceable in the database's own logs, a leak is limited to one credential for hours, and rotation is just the next read. After configuring the connection, rotate-root removes the last human-known password.

Also asked: What happens to the database users when Vault is unavailable for an hour? · How would you revoke every credential issued for one role? · When would you choose a static role over dynamic credentials?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.