OnCallReadyStart learning

Infrastructure as code Lesson 16.20 · Terraform in Real Life & the Associate Exam · 19 min read

Secrets: Key Vault references, not tfvars

In plain words

Imagine you need to give the babysitter the key to the medicine cabinet. You could write the key's shape on the fridge note (anyone reading the note can copy it), or you could leave a note saying "the key is in the lockbox by the door", and give only the babysitter the lockbox code. Even better: a cabinet that opens when it recognises the babysitter's face, so there is no key at all.

In Terraform, putting a password in terraform.tfvars is writing it on the fridge note, forever in git history. A Key Vault reference like @Microsoft.KeyVault(SecretUri=...) is the note pointing at the lockbox: the app reads the secret at runtime with its managed identity. And managed identities with roles are the face-recognising cabinet: no password exists. sensitive = true only hides the key from the screen.

The problem: a password in five places

An app needs a database password. The quick way: put it in terraform.tfvars, pass it into the app's settings, done. Six months later that password is in git history, in two CI logs, in the state file, in a saved plan attached to a ticket, and in the app's configuration where anyone with read access to the app can see it. Changing it means finding every one of those places.

This lesson is how to design secrets so they pass through as few places as possible - ideally none.

What you need to know already: sensitive = true and variable precedence incl. TF_VAR_ (15.5), what state stores (15.1: every attribute, in plain text), random_password (16.1), -replace (15.26), managed identities (14.3, 16.2), role assignments and Key Vault (16.2), plan artifacts (16.15), secrets in Docker images (12.40).

A secret is any value that grants access: a password, an API key (a long random string an app uses to log in to another service), a certificate's private key, a token. Rotation means replacing a secret with a new one, so the old one stops working.

Where secrets leak in a Terraform setup

terraform.tfvars committed with db_password = "..."      git history, forever
-var db_password=... in a pipeline                       CI logs, shell history
output "connection_string" without sensitive = true      plan output, CI logs
state                                                    every attribute, plaintext
saved plan files                                         every value
TF_LOG=TRACE                                             request bodies

Row by row:

  • A committed tfvars file stays in git history even after you delete it.
  • -var db_password=... on a command line is printed by pipelines that echo their commands, and stays in shell history (the history command).
  • An output without sensitive = true is printed at the end of every apply.
  • State stores every attribute of every resource in plain text JSON.
  • A saved plan (16.15) contains every value too.
  • TF_LOG=TRACE (15.24), the most verbose debug log, prints the raw requests Terraform sends to Azure - secrets included.

sensitive = true fixes exactly one line of that table (the CLI output). The rest need design.

Rule 1: never put secrets in tfvars

tfvars files are configuration, and configuration is committed. A secret in terraform.tfvars is in git history forever - rewriting history does not un-leak it, because clones, forks and caches already have it. The secret must be rotated.

*.auto.tfvars "just on my machine" is the same mistake one git add . away. If a secret must be passed in at all, it comes from the pipeline's secret store (a setting in the pipeline service that is masked in logs) as the environment variable TF_VAR_db_password - and even then it lands in state.

Rule 2: let Terraform create secrets where they live

The pattern that keeps secrets out of tfvars and pipelines entirely: Terraform generates the value and writes it straight into Key Vault, Azure's safe for secrets (16.2):

resource "random_password" "db" {
  length  = 32
  special = true
}

resource "azurerm_key_vault_secret" "db_password" {
  name            = "orders-db-password"
  value           = random_password.db.result
  key_vault_id    = azurerm_key_vault.this.id
  content_type    = "password"
  expiration_date = "2027-03-31T00:00:00Z"
}
  • random_password (from the hashicorp/random provider) generates a random 32-character string with special characters, once, and keeps it in state.
  • azurerm_key_vault_secret stores it in the vault under the name orders-db-password, with a label (content_type) and an expiry date - the two settings checkov's CKV_AZURE_114 and CKV_AZURE_41 want (16.10).

The password is generated, stored in Key Vault, and no human ever sees it. It is in state (random_password.db.result and the secret's value) - which is why state lives in a locked-down backend (15.4).

Rule 3: apps read from Key Vault at runtime - references, not values

The examples below use App Service: Azure's service that runs a web app for you (you give it code or a container image, it runs it). Its app settings are the environment variables the app sees when it starts.

The worst pattern: Terraform reads the secret and writes it into the app settings.

# DON'T: the plaintext value ends up in the app's configuration and in state twice
app_settings = {
  DB_PASSWORD = random_password.db.result
}

The right pattern: a Key Vault reference. The setting holds a pointer to the secret; App Service fetches the real value when the app starts, logging in with the app's managed identity:

resource "azurerm_linux_web_app" "orders" {
  # ...
  identity {
    type = "SystemAssigned"
  }

  app_settings = {
    DB_PASSWORD = "@Microsoft.KeyVault(SecretUri=${azurerm_key_vault_secret.db_password.versionless_id})"
  }
}

resource "azurerm_role_assignment" "orders_kv" {
  scope                = azurerm_key_vault.this.id
  role_definition_name = "Key Vault Secrets User"
  principal_id         = azurerm_linux_web_app.orders.identity[0].principal_id
}
  • identity { type = "SystemAssigned" } gives the app its own managed identity (17.2). In the mission you use a user-assigned one instead; the app then also needs key_vault_reference_identity_id to know which identity to use.
  • @Microsoft.KeyVault(SecretUri=...) is App Service's reference syntax. The app reads an ordinary environment variable DB_PASSWORD and never knows Key Vault was involved.
  • versionless_id is the secret's address without a version number. Key Vault keeps every version of a secret; the versionless address always means "the current one", so a rotated secret reaches the app without a Terraform change.
  • The role assignment gives the app's identity Key Vault Secrets User (may read secret values) on this vault only - not Contributor, not the whole subscription.
  • Other platforms have the same idea. A container cluster, for example, can use a driver that mounts Key Vault secrets as files into containers using the workload's identity. Terraform's only job is to grant access.

Now a rotation is a Key Vault operation, and Terraform never hands the app the value.

Rule 4: identities instead of passwords

Many "secrets" exist only because one thing needs to log in to another: an app to a database, a pipeline to Azure, a cluster to a container registry (12.54). Managed identities remove the secret entirely:

pipeline -> Azure          OIDC federated credential (no client secret)
cluster -> registry        cluster identity + AcrPull role assignment
app -> Key Vault / Storage managed identity + a data-plane role
app -> Azure SQL           Entra ID authentication with the app's identity

Row by row:

  • pipeline -> Azure: OIDC (14.3, 16.15) - Azure trusts the pipeline's short-lived token; there is no password.
  • cluster -> registry: the cluster's machines log in to ACR (Azure Container Registry) with their managed identity, which gets the built-in AcrPull role (may download images).
  • app -> Key Vault or storage: the app's identity gets a data-plane role - a role about the data inside (read secrets, read files) rather than about managing the resource.
  • app -> Azure SQL (Azure's managed database): the database accepts Entra ID (Microsoft's login service) logins, so the app logs in as its identity.

The best secret is the one that does not exist.

Rule 5: protect what is unavoidable

  • State in a backend with RBAC, encryption and versioning, readable only by the pipeline identities (15.4).
  • sensitive = true on every output and variable that carries a secret, so plans and logs show (sensitive value).
  • Short retention on plan artifacts; no TRACE logs in pipelines.
  • Rotate on exposure. If a value reached a log or git, it is compromised. With Terraform-generated secrets, rotation is one command:
terraform apply -replace=random_password.db

-replace=ADDRESS forces Terraform to destroy and recreate that one resource (15.26). A new random password is generated, the Key Vault secret gets a new version, and apps using versionless references pick it up.

Newer Terraform: ephemeral values and write-only arguments

Terraform 1.10 added ephemeral values and 1.11 write-only arguments - the first language features that keep secrets out of state and plan files altogether. The exam (version 1.12) includes them; the lab runs 1.9 and does not:

variable "db_admin_password" {
  type      = string
  sensitive = true
  ephemeral = true          # 1.10+: available during the run, never persisted
}

ephemeral "azurerm_key_vault_secret" "admin" {   # 1.10+: an ephemeral resource
  name         = "sql-admin"
  key_vault_id = azurerm_key_vault.this.id
}

resource "azurerm_mssql_server" "this" {
  # ...
  administrator_login_password_wo         = ephemeral.azurerm_key_vault_secret.admin.value  # 1.11+ write-only
  administrator_login_password_wo_version = 1
}
  • An ephemeral variable or resource exists only during the plan/apply that uses it. Terraform refuses to store it in state or in the plan. Here, the ephemeral block reads a secret from Key Vault for the length of the run.
  • A write-only argument (the _wo suffix, where the provider offers one) is sent to Azure but never recorded. Because Terraform does not remember the value, it cannot tell when it changed - so you bump _wo_version (1 -> 2) when you want it to send a new one.

Know they exist, what problem they solve, and that they need both a new Terraform and provider support. Which resources offer ephemeral variants and _wo arguments varies by provider version, so treat the snippet above as the shape of the feature and check the provider docs for the exact names.

Sensitive vs secret, one more time

sensitive = true      redacts CLI output                  still in state and plan files
ephemeral = true      (1.10+) never in state or plan      only usable in ephemeral contexts
Key Vault reference   the value never passes through the app configuration or Terraform
managed identity      there is no secret at all

From weakest to strongest: hiding the value on screen, keeping it out of state, keeping it out of the app's configuration, and not having a secret at all.

What you can now do:

  • List the places a Terraform secret leaks, and which one sensitive fixes.
  • Generate a secret into Key Vault and give an app a Key Vault reference plus a read-only role on that vault.
  • Rotate a Terraform-generated secret with -replace.

Why it helps

Leaked secrets are one of the most common findings in platform security reviews, and Terraform repos are a favourite place to find them. Situations: someone commits db_password in tfvars; rewriting git history does not help, the secret must be rotated, and with a random_password into Key Vault design that is one terraform apply -replace=random_password.db. A pipeline passes -var db_password=... and the value shows in logs. An app stores its password in plaintext app settings, visible to anyone with Reader. Knowing the reference and identity patterns lets you fix the design, not just the leak. Exam objective 4h asks sensitive vs ephemeral directly.

On the blog

FAQ

If I mark everything sensitive, am I safe?

No. sensitive = true redacts values in CLI output, which stops them appearing in plan logs. They are still stored in state and in saved plan files, and in git if they came from a committed tfvars file. Real protection is keeping secrets out of the path: identities, Key Vault references, generated secrets, plus a locked-down backend for what remains.

Why use the versionless secret URI?

A Key Vault reference with the versionless URI always resolves to the current version of the secret. When the secret is rotated in Key Vault, the app picks up the new value (after its refresh interval or restart) without any Terraform change. A versioned URI would pin the app to the old value until someone updates the reference.

Why use a user-assigned identity for Key Vault references?

A user-assigned identity is its own resource, so you can create it and grant it Key Vault Secrets User on the vault before the app exists, and it survives the app being recreated. App Service then needs key_vault_reference_identity_id set to that identity; without it, App Service tries a system-assigned identity the app may not have, and the reference fails to resolve.

What are ephemeral values and write-only arguments?

Terraform 1.10 added ephemeral variables, outputs and resources, which exist only during one plan or apply and are never stored in state or plan files. 1.11 added write-only arguments (usually a _wo suffix), which are sent to the provider but never recorded, with a _wo_version to signal when to send a new value. Both need provider support that varies by resource. The lab's 1.9 does not have them; the 004 exam does.

A secret was committed to git. Is deleting it from history enough?

No. Once pushed, assume it is compromised: clones, forks, CI caches and logs may have it. Rotate the secret first, then clean up. With the pattern where Terraform generates the value into Key Vault, rotation is terraform apply -replace=random_password.db, and apps using versionless references pick up the new value. Then move the design to references or identities so it cannot recur.

In an interview Mid

How do you keep secrets out of Terraform?

First know where they leak: committed tfvars (git history, forever), -var on a command line (CI logs), outputs without sensitive, state and saved plans (every value in plain text), TF_LOG=TRACE. sensitive = true only fixes the screen output.

Then design them away, strongest first:

  1. No secret at all - managed identities: the pipeline logs in with OIDC, the app reads storage or the database as its identity.
  2. Generated where they live - random_password written straight into Key Vault with azurerm_key_vault_secret (with an expiry); no human sees it.
  3. References, not values - the app setting holds a Key Vault reference to the secret's versionless_id, and the app's identity gets Key Vault Secrets User on that vault only. Rotation then needs no Terraform change.
  4. Protect what is unavoidable - state in a locked-down backend, short plan retention.

Rotate on exposure: terraform apply -replace=random_password.db. Newer Terraform adds ephemeral values and write-only arguments that never reach state.

Also asked: Where can secrets leak in a Terraform setup? · Does sensitive = true keep a value out of state? · What should you do when a password appears in a CI log?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.