Terraform secret leaked in a CI log: sensitive = true, TF_LOG and rotating first

Security posts in the platform channel: last night's pipeline log for orders-prod contains what looks like a database password.

$ grep -n 'password' artifacts/ci-run-4812.log | cut -c1-90
4:2026-09-23T21:04:42Z smoke: connecting to sql-orders-prod.database.windows.net as orders

(The cut keeps the line short on purpose: the rest of it is the password.)

The configuration follows "the right pattern": Terraform generates the password with random_password, stores it in Key Vault, and the app reads it through a Key Vault reference. The output is even marked sensitive:

$ terraform output
db_password = <sensitive>

And still it leaked.

What sensitive = true really does

sensitive = true is a display filter. Terraform will not print the value in plan and apply output, and shows (sensitive value) instead. That is all it does. The value is still:

• in the state, in plain text, with every other attribute of every resource;
• printed by terraform output -raw NAME and terraform output -json on request;
• in a saved plan file, and in plain text in terraform show -json tfplan;
• possibly in TF_LOG=DEBUG or TRACE logs, which include the raw requests Terraform and the provider send to the cloud API.

Anything that can read state or run terraform output can read the secret. Pipeline secret masking does not help either: Azure DevOps and GitHub Actions mask the values of their own secret variables in logs (***), not a value Terraform generated during the run.

The diagnosis path

1. Is the leaked value the live one? Check without printing it

A terminal can be recorded, shared or scrolled back by someone else, so do not cat the password to compare. Compare it by count:

$ terraform init >/dev/null
$ grep -cF -- "$(terraform output -raw db_password)" artifacts/ci-run-4812.log
1

-F makes grep treat the password as a fixed string (passwords contain $, # and other characters a regex would interpret), -- stops it being read as an option, and -c prints a count instead of the line. 1 means the log holds the current password.

2. Find the leak path

The log line came from the smoke test step:

$ cat ci/smoke.sh
#!/usr/bin/env bash
# post-deploy smoke test (added in PLAT-2291 "debug flaky DB login")
set -euo pipefail
host="sql-orders-prod.database.windows.net"
pw="$(terraform output -raw db_password)"
echo "smoke: connecting to $host as orders_app with password $pw"
echo "smoke: ok"

A debug line, added to chase a login problem, that reads the output on purpose and echoes it. And the output exists only to make that possible:

$ grep -n -B1 -A4 'output "db_password"' main.tf
95-
96:output "db_password" {
97-  value     = random_password.db.result
98-  sensitive = true
99-}

Other common leak paths, worth checking in the same incident: set -x in a script that handles secrets, -var db_password=... on a command line the CI prints, TF_LOG left on in a pipeline, and saved plans uploaded as long-lived artifacts.

The fix: remove the leak path, then rotate

Order matters. Rotate first and the next pipeline run prints the new password too.

Remove the lines that read and print it, and delete the output - nothing outside Terraform needs the value, because the app reads Key Vault:

$ sed -i '/terraform output -raw db_password/d; s/ with password \$pw//' ci/smoke.sh
$ sed -i '/^output "db_password" {/,/^}/d' main.tf

Now rotate: replace the secret with a new one so the leaked value stops working. Because Terraform generates it, that is one flag. Review the plan first:

$ terraform plan -replace=random_password.db | grep -E "#|Plan:|db_password ="
  # random_password.db will be replaced, as requested
  # azurerm_key_vault_secret.db_password will be updated in-place
        # (5 unchanged attributes hidden)
Plan: 1 to add, 1 to change, 1 to destroy.
  - db_password = (sensitive value) -> null

A new random password, a new version of the Key Vault secret, the output removed - and the web app not in the list. It reads the secret through a versionless reference:

DB_PASSWORD = "@Microsoft.KeyVault(SecretUri=${azurerm_key_vault_secret.db_password.versionless_id})"

so it picks up the new version without a change (App Service refreshes references periodically, or at once on restart).

$ terraform apply -replace=random_password.db -auto-approve | tail -2

Apply complete! Resources: 1 added, 1 changed, 1 destroyed.
$ terraform plan | grep -E "No changes|Plan:"
No changes. Your infrastructure matches the configuration.

In a real incident the work continues outside Terraform: the database login must get the new password too (a Key Vault secret is only half of it), the CI log must be purged or restricted, and someone should ask why a smoke-test step could read prod outputs at all.

Keeping it from coming back

• No outputs for secrets. If something needs the value, it reads the secret store with its own identity.
• Treat state as a secret. Restrict who can read the state container: the pipeline identity, and break-glass access. Everyone who can read state can read every password in it.
• Short-lived plan artifacts, readable only by the apply stage.
• No TF_LOG in pipelines except for a one-off debug run whose log is deleted afterwards.
• Newer Terraform keeps secrets out of state: ephemeral values (1.10+) and write-only _wo arguments (1.11+, where the provider offers them) are used during the run and never stored.

A leaked secret is one of several ways a pipeline can hurt production. The others in this series: a plan reading prod's state from a dev folder, a stale state lock after a cancelled apply, and a count list that re-indexes. The smoke script above starts with set -euo pipefail, which is good practice but no safety net for secrets - and not even for every failure: why set -e does not catch a failing pipe.

Practise it

Incident: the prod database password is in a CI log (14.22) is this configuration: confirm the leak without printing it, remove the path, rotate, and leave the app untouched.