Vault "permission denied" with the right policy: KV v2 and the data/ path

The nightly report job failed at 02:00. Nobody touched its AppRole, nobody touched its policy, and this is what its script prints:

$ ./fetch.sh
Error reading kv/data/reports/smtp: Error making API request.

URL: GET https://127.0.0.1:8200/v1/kv/data/reports/smtp
Code: 403. Errors:

* 1 error occurred:
	* permission denied

The policy looks right:

$ vault policy read reports-read
# nightly-report job: read its SMTP secret
path "kv/reports/*" {
  capabilities = ["read", "list"]
}

The script asks for kv/reports/smtp. The policy allows kv/reports/*. And yet: 403. Look at the URL in the error again. Vault was not asked for kv/reports/smtp.

What is happening: the CLI rewrites your path

Vault policies are matched against the API path of the request, not against what you typed. With the KV version 1 engine those are the same thing. With KV version 2 they are not: the engine keeps versions and metadata, so every operation lives under its own prefix:

vault kv get hides this: it detects a v2 mount and inserts data/ for you. So a policy written as kv/reports/* silently stops matching the moment the mount becomes v2. Nothing errors when you write the policy; Vault accepts any path. The first sign is a 403 at 02:00.

That is exactly what happened here. The platform channel said "kv/ maintenance last evening". Someone ran vault kv enable-versioning kv/, which upgrades a v1 mount to v2 in place: every secret survives, every path moves.

Diagnosis

1. Read the URL in the error

URL: GET .../v1/kv/data/reports/smtp. The data/ in it already tells you the mount is KV v2. Whenever a "permission denied" looks impossible, compare the path in the URL with the paths in the policy, character by character.

2. Ask Vault what this token can do on that path

Test with the client's own token, not your admin one. Your token can read everything, which proves nothing. Log in as the job and keep the token:

$ export JT=$(vault write -field=token auth/approle/login role_id=$(cat role-id) secret_id=$(cat secret-id))
$ vault token capabilities $JT kv/reports/smtp
list, read
$ vault token capabilities $JT kv/data/reports/smtp
deny

There is the whole incident in two lines. The token may read the path nobody asks for, and is denied on the path every read actually uses. While you are at it, vault token lookup shows which policies the token carries. If the policy you expect is missing from that list, it is a different bug: the role, not the paths.

$ VAULT_TOKEN=$JT vault token lookup
Key                 Value
---                 -----
...
meta                map[role_name:reports]
policies            [default reports-read]
renewable           true
ttl                 9m59s
type                service

3. Confirm what changed on the mount

$ vault read sys/mounts/kv
Key                        Value
---                        -----
accessor                   kv_da914880
config                     map[default_lease_ttl:0 force_no_cache:false max_lease_ttl:0]
deprecation_status         supported
description                team secrets
external_entropy_access    false
local                      false
options                    map[version:2]
...

options map[version:2]. (vault secrets list -detailed shows the same in its Options column.) The data is all still there. With your own token, vault kv metadata get kv/reports/smtp shows the secret with its first version, created when the mount was upgraded.

The fix: the same access, at the v2 paths

Change the policy, not the job. Read goes to data/, list goes to metadata/:

$ vault policy write reports-read - <<'EOF'
# nightly-report job: read its SMTP secret (kv/ is KV v2 since the upgrade)
path "kv/data/reports/*" {
  capabilities = ["read"]
}

path "kv/metadata/reports/*" {
  capabilities = ["list"]
}
EOF
Success! Uploaded policy: reports-read

Policies apply to existing tokens at once, so there is no need to log in again. Check with the same token, including a path the job must not reach:

$ vault token capabilities $JT kv/data/reports/smtp
read
$ VAULT_TOKEN=$JT vault kv get kv/payments/stripe
Error reading kv/data/payments/stripe: Error making API request.

URL: GET https://127.0.0.1:8200/v1/kv/data/payments/stripe
Code: 403. Errors:

* 1 error occurred:
	* permission denied

$ ./fetch.sh
connected to smtp.lab:587 as reports@oncall-lab (password: 21 chars)
report for 2026-09-22 sent to finance@oncall-lab
Success! Revoked token (if it existed)

Two things not to do under pressure. Do not "fix" it with path "kv/*", which quietly gives the report job the payments keys too. And do not copy the old ["read", "list"] onto both new paths: delete and update on kv/metadata/ remove every version of a secret, and the habit of copying capabilities around is how they get there. Keep it to the two paths and the two capabilities the job uses.

Other reasons for a 403 that looks impossible

The same three steps (read the URL, vault token capabilities with the client's token, read the mount) catch the other usual suspects:

• A trailing glob mismatch. kv/data/reports matches only that exact path; kv/data/reports/* matches everything under it. + matches one path segment: kv/data/+/smtp.
• The wrong namespace or mount. The URL shows which mount the request hit.
• An explicit deny somewhere. deny wins over every other capability, in any policy the token carries.
• A token that has expired. That error says permission denied too. If vault token lookup fails with bad token, it is a lifetime problem: see the token that expired at 3 a.m.
• Vault is not answering at all. A 503 that says Vault is sealed is a different incident: unsealing after a restart.

Keeping it from coming back

• Before you upgrade a mount, search every policy for its prefix: vault policy list, then vault policy read each one and grep for kv/. Write the v2 policies first. Policies for both path shapes can exist side by side during the change.
• Test policies with vault token capabilities in CI, with a token created from the policy. A policy is code, so test it like code.
• Keep secret paths per consumer (kv/data/reports/*), so least privilege stays a short policy and nobody is tempted by kv/*.
• And keep the secrets themselves out of logs on the way to the app: that is how most of them leak, whether from Terraform or from an Ansible run at -vvv.

Practise it

The Vault & Secrets Management chapter has this as an incident: Incident: permission denied with the right policy. You get the failing job, the policy and the upgraded mount, and you are done when the job's own script sends the report and its token still cannot read anything else. The drills Write the minimal policy for this request and What can this token do here, and which rule decides? practise the path matching until it is automatic.