Lesson 22.23 · Azure I: CLI, Identity & Data Planes · 21 min read

Key Vault: two permission models, one data plane

Q: Does every secret set create a new version?

Yes. Each az keyvault secret set creates a new version with its own ID, the last part of the secret URL, and readers asking by name get the latest. You can pin a version by using its full ID. secret list-versions shows the history. That makes rotation straightforward, but remember that applications caching the old value, or pinned to a version, won't see the new one until they re-read.

Q: An application gets 403 when reading a Key Vault secret. How do you troubleshoot it?

Read the error - it names everything: - ForbiddenByRbac - the vault uses Azure RBAC and no assignment grants the action. Caller oid = which identity (is it the one you think?), Action = the missing dataAction (getSecret), Resource = the scope checked, Assignment: (not found). - AccessDenied / "does not have secrets get permission" - a legacy access-policy vault; fix the policy (az keyvault show --query properties.enableRbacAuthorization tells you the model). - ForbiddenByFirewall - network, not identity: your IP is not allowed or public access is disabled. No role fixes it. The usual fix: Key Vault Secrets User for that identity's object id, scoped to the vault (or one secret). Traps: Owner and Key Vault Contributor have no data access, and Key Vault Reader reads metadata only, never values - read the dataActions, not the name. A brand-new assignment can take minutes to propagate. If the identity never gets a token at all, it is an AADSTS error, not a 403.

In plain words

Imagine a bank safe-deposit room. The bank manager can build the room, change the alarm system, and decide who may enter, but can't open the boxes: each box has its own list of who may open it. If the manager wants to look inside, he has to put his own name on the list first, and that's written in a logbook. Throwing a box away doesn't destroy it immediately; it sits in a locked bin for 90 days.

Key Vault is that room. Creating and configuring kv-shop-prod is the control plane; reading secrets is the data plane, with its own check. In RBAC mode that check needs a data role like Key Vault Secrets User, so an Owner gets ForbiddenByRbac until he grants himself one. Soft delete is the locked bin, and purge protection means not even the manager can empty it early.

The orders app needs a database password. Put it in a config file and it ends up in git; put it in an environment variable and it shows up in /proc and in systemctl show (2.21). You want one guarded place that stores it, logs every read, and hands it only to the identities you name. In Azure that place is Key Vault.

What you need to know already: 22.1 (control plane vs data plane), 22.13 (roles, dataActions, scopes), 22.10 (granting a role to an identity), 9.15 (keys and certificates), 13.1 (Terraform state holds values in plain text).

Key Vault (KV) is Azure's service for keeping secrets. A vault is one instance of it, a resource with its own address (https://<name>.vault.azure.net). It stores three kinds of thing:

secrets - strings: passwords, API keys, connection strings (one string with a server address plus the credentials to reach it).
keys - asymmetric keys (a private/public pair, Ch 9) that never leave the vault; you send it data and ask it to sign or decrypt.
certificates - a key plus its X.509 certificate (Ch 9), with renewal.

Platform work is mostly secrets, so this lesson is mostly secrets.

Two planes, two checks

az keyvault create / update / delete     ARM   management.azure.com         control plane
az keyvault secret set / show / list     KV    kv-shop-prod.vault.azure.net data plane

The control plane decides who can manage the vault. The data plane decides who can read and write what is inside it. They are authorised separately, and the data plane has two possible models.

az keyvault commands manage vaults (create -n <name> -g <group>, show, update, delete); az keyvault secret commands work on what is inside (--vault-name which vault, -n which secret).

Access policies vs Azure RBAC

--query properties.enableRbacAuthorization asks which model a vault uses:

$ az keyvault show -n kv-shop-prod --query properties.enableRbacAuthorization
true

Access policies (legacy): a list on the vault itself, "object X may get, list, set secrets". Anyone with Microsoft.KeyVault/vaults/write on the vault (Contributor!) can add themselves to it - so Contributor on a policy-model vault is effectively full secret access. Policies cannot be scoped below the vault.
Azure RBAC (the default for new vaults since CLI 2.61, and the default in the 2026-02-01 API): data-plane roles assigned like any other role, scoped to the vault or even to one secret, visible in the same tooling, and Contributor cannot grant itself data access.

Use RBAC. Migrating a policy vault is az keyvault update --enable-rbac-authorization true after creating equivalent role assignments - flip it first and every app loses access at once.

The data-plane roles

role	can	use for
Key Vault Secrets User	read secret values	applications, pods
Key Vault Secrets Officer	create, update, delete secrets	people and pipelines that manage secrets
Key Vault Reader	read vault and secret metadata	audit and inventory - cannot read values
Key Vault Administrator	everything on the data plane	break-glass
Key Vault Contributor	manage the vault (control plane)	no data access at all

Two names that lie: Key Vault Reader sounds like it can read the vault's secrets and cannot; Key Vault Contributor sounds like it can work with the vault's contents and cannot. Read the dataActions, not the name:

az role definition list --name "Key Vault Reader" --query "[0].permissions[0].dataActions"

The error you will see

You are Owner of the subscription. You create a vault. You try to store a secret:

# kv-oncall-lab = the vault you create in the next mission
az keyvault secret set --vault-name kv-oncall-lab -n db-password --value 's3cr3t'
ERROR: (Forbidden) Caller is not authorized to perform action on resource.
If role assignments, deny assignments or role definitions were changed recently, please observe propagation time.
Caller: appid=04b07795-8ddb-461a-bbee-02f9e1bf7b46;oid=99999999-8888-7777-6666-555555555555;iss=https://sts.windows.net/11111111-.../
Action: 'Microsoft.KeyVault/vaults/secrets/setSecret/action'
Resource: '/subscriptions/.../resourcegroups/rg-oncall-lab/providers/microsoft.keyvault/vaults/kv-oncall-lab/secrets/db-password'
Assignment: (not found)
DenyAssignmentId: null
DecisionReason: null
Vault: kv-oncall-lab;location=westeurope

Inner error: {
    "code": "ForbiddenByRbac"
}

Read it line by line - it tells you everything:

Caller: appid=04b07795-... is the Azure CLI's own app id; oid (object id) is you; iss is who issued your token (Entra ID). For a pod it would be the workload identity's appid and oid.
Action: the exact dataAction that was missing (setSecret, getSecret, readMetadata for list).
Resource: the scope that was checked - down to the secret.
Assignment: (not found): no role at that scope or above grants it.
ForbiddenByRbac: the vault uses RBAC. (A policy-model vault says AccessDenied and "does not have secrets get permission" instead.)

Owner has no dataActions, so the fix is a data role for yourself (az ad signed-in-user show --query id -o tsv prints your own object id):

KV=$(az keyvault show -n kv-oncall-lab --query id -o tsv)
az role assignment create --role "Key Vault Secrets Officer" --assignee "$(az ad signed-in-user show --query id -o tsv)" --scope "$KV"

Secrets and versions

# once you hold Key Vault Secrets Officer on it
az keyvault secret set --vault-name kv-oncall-lab -n db-password --value 's3cr3t' --query "{id:id, created:attributes.created}"
{
  "created": "2026-09-24T09:12:41+00:00",
  "id": "https://kv-oncall-lab.vault.azure.net/secrets/db-password/76810bbb17c89ea9e063a9fbf83a648f"
}

secret set stores a value; --query keeps two fields of the reply. Every set creates a new version; the id ends with it (the long hex string). Readers ask for the secret by name (latest) or pin a version:

az keyvault secret show --vault-name kv-oncall-lab -n db-password --query value -o tsv
az keyvault secret list-versions --vault-name kv-oncall-lab -n db-password -o table
az keyvault secret show --id https://kv-oncall-lab.vault.azure.net/secrets/db-password/<version>

show --query value -o tsv prints the current value bare; list-versions lists every version; show --id <url> reads one specific version.

secret list returns metadata only - names, ids, attributes - never values. Printing a value to a terminal puts it in your scrollback and possibly a session recording; in scripts use -o tsv straight into a variable, and never set -x (6.22, it prints every command with its values) around it.

Soft delete and purge protection

Soft delete means a delete is not final: the item goes to a recycle bin first. It is always on now (it can no longer be disabled). Deleting a secret or a whole vault moves it to that bin for the retention period (how long it is kept: 7-90 days, default 90):

az keyvault secret delete --vault-name kv -n db-password
az keyvault secret list-deleted --vault-name kv -o table
az keyvault secret recover --vault-name kv -n db-password
az keyvault secret purge --vault-name kv -n db-password     # gone for good

Two consequences that surprise people:

# right after deleting the secret
az keyvault secret set --vault-name kv -n db-password --value x
ERROR: (Conflict) Secret db-password is currently in a deleted but recoverable state, and its
name cannot be reused; in this state, the secret can only be recovered or purged.

# after deleting a vault of that name
az keyvault create -n kv-sysop-scratch -g rg-oncall-lab
ERROR: (ConflictError) A vault with the same name already exists in deleted state. You need to
either recover or purge existing key vault.

Vault names are globally unique and stay reserved while soft-deleted. A Terraform destroy-and-apply of a vault fails on exactly this (the azurerm provider has purge_soft_delete_on_destroy / recover_soft_deleted_key_vaults feature flags for it).

Purge = delete from the recycle bin, for good. Purge protection makes purge impossible until retention ends - not even an Owner can remove it early - and once enabled it can never be disabled. Turn it on for production vaults: it is what stops a compromised pipeline, or a tired engineer, from destroying every secret irrecoverably.

Network access

A vault has a public endpoint by default: its address is reachable from the internet (still behind sign-in). For production: --public-network-access Disabled plus a private endpoint - an address for the vault inside your own private network (Ch 23 builds one) - or at least the vault's firewall, a list of allowed IP ranges. A 403 can also mean "your IP is not allowed" - the error then says ForbiddenByFirewall instead of ForbiddenByRbac, and no role assignment will fix it.

Terraform shape

The same vault in Terraform (Ch 12-14):

resource "azurerm_key_vault" "this" {
  name                       = "kv-shop-prod"
  resource_group_name        = azurerm_resource_group.this.name
  location                   = "westeurope"
  tenant_id                  = data.azurerm_client_config.current.tenant_id
  sku_name                   = "standard"
  rbac_authorization_enabled = true
  purge_protection_enabled   = true
  soft_delete_retention_days = 90
}

resource "azurerm_role_assignment" "orders_reads_secrets" {
  scope                = azurerm_key_vault.this.id
  role_definition_name = "Key Vault Secrets User"
  principal_id         = azurerm_user_assigned_identity.orders.principal_id
  principal_type       = "ServicePrincipal"
}

(In azurerm 4.x the vault argument is rbac_authorization_enabled; older code says enable_rbac_authorization.) Note what is not there: the secret value. Put secrets into the vault from the system that owns them, not from Terraform, or the value ends up in state.

What you can now do

Read a ForbiddenByRbac error and name the missing dataAction, the caller and the scope.
Pick the right Key Vault role: Secrets User for apps, Officer for people, never "Reader" for values.
Recover a deleted secret or vault, and explain what purge protection prevents.

Why it helps

Every application on your platform gets its secrets from Key Vault, so "the app can't read its secret" will be one of your most frequent tickets. The error is precise if you read it: ForbiddenByRbac means a role is missing, ForbiddenByFirewall means the network blocks it and no role will help, AccessDenied with "secrets get permission" means the vault still uses access policies.

You'll also run into soft delete in Terraform: a destroy and re-apply of a vault fails because the name is reserved in the deleted state. And in design reviews you'll push for RBAC mode, since on an access-policy vault any Contributor can add themselves to the policy, and for purge protection on production vaults, which protects against a compromised pipeline wiping every secret.

Commands in this lesson

az

FAQ

Access policies or RBAC: which should I use?

RBAC. It's the default for new vaults from recent CLI versions and API versions, it uses the same role assignments and tooling as the rest of Azure, it can be scoped down to one secret, and Contributor on the vault cannot grant itself data access. With access policies, anyone with vaults/write, which includes Contributor, can edit the policy and give themselves full secret access. To migrate, create equivalent role assignments first, then flip enable-rbac-authorization.

What can Key Vault Reader do?

Read the vault's properties and the metadata of secrets, keys and certificates: names, IDs, versions, enabled state, expiry dates. It cannot read secret values. That's useful for inventory and audit, for example finding secrets that are about to expire. An application that needs values needs Key Vault Secrets User. Similarly misleading, Key Vault Contributor manages the vault itself on the control plane and has no data access at all.

Why can't I create a secret or vault with a name I just deleted?

Soft delete is always on. A deleted secret or vault stays in a recoverable state for the retention period, 7 to 90 days with 90 as the default, and its name stays reserved. Recover it, or purge it if purge protection isn't enabled. Vault names are globally unique, so a Terraform destroy-and-recreate hits this; the azurerm provider has feature flags to purge or recover on the way.

Does every secret set create a new version?

Yes. Each az keyvault secret set creates a new version with its own ID, the last part of the secret URL, and readers asking by name get the latest. You can pin a version by using its full ID. secret list-versions shows the history. That makes rotation straightforward, but remember that applications caching the old value, or pinned to a version, won't see the new one until they re-read.

Should Terraform create secret values?

Generally no, because anything Terraform manages ends up in its state file, in plaintext, and in the plan output. Let Terraform create the vault, its network rules and the role assignments, and have the system that owns the secret, the database provisioning, a rotation job or a person through a controlled process, put the value in. If Terraform must generate a value, protect the state accordingly and mark outputs sensitive.

In an interview Mid

An application gets 403 when reading a Key Vault secret. How do you troubleshoot it?

Read the error - it names everything:

ForbiddenByRbac - the vault uses Azure RBAC and no assignment grants the action. Caller oid = which identity (is it the one you think?), Action = the missing dataAction (getSecret), Resource = the scope checked, Assignment: (not found).
AccessDenied / "does not have secrets get permission" - a legacy access-policy vault; fix the policy (az keyvault show --query properties.enableRbacAuthorization tells you the model).
ForbiddenByFirewall - network, not identity: your IP is not allowed or public access is disabled. No role fixes it.

The usual fix: Key Vault Secrets User for that identity's object id, scoped to the vault (or one secret). Traps: Owner and Key Vault Contributor have no data access, and Key Vault Reader reads metadata only, never values - read the dataActions, not the name. A brand-new assignment can take minutes to propagate.

If the identity never gets a token at all, it is an AADSTS error, not a 403.

Also asked: What is Azure Key Vault used for, and what can it store? · What do soft delete and purge protection protect you from? · Compare Key Vault access policies with the Azure RBAC permission model.

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.