Ansible printed the password in the job log: no_log, ansible-vault and rotating the secret

The security ticket: "The orders DB password is visible in the deploy job log. The job log is readable by every engineer. Treat the password as compromised." The password lives in an encrypted vault file, and it is still in the log twice:

$ grep -n "Bl4ckP3pper-77" logs/deploy-2026-10-05.log
20:    "msg": "postgres://orders:Bl4ckP3pper-77@db-1:5432/orders"
31:            "after": "# Ansible managed\n[database]\nhost = db-1\nuser = orders\npassword = Bl4ckP3pper-77\n",

The vault did its job. Nobody ever leaked the vault file.

What is happening: the vault protects the file, not the run

ansible-vault encrypts data at rest: in Git, on disk, in the artifact.

$ head -3 group_vars/db/vault.yml
$ANSIBLE_VAULT;1.1;AES256
63646235383536376232333630336531303062323232393766353534326661396561376166303038
3039636130313032383833366430336635623065373734350a386230306137303533343230636264

At run time Ansible decrypts it with the vault password (here vault_password_file = ~/.vault_pass in ansible.cfg), and from then on vault_db_password is an ordinary variable. Anything that prints a variable prints the secret. Here is the playbook:

  tasks:
    - name: Show the connection string
      ansible.builtin.debug:
        msg: "postgres://{{ db_user }}:{{ db_password }}@db-1:5432/orders"

    - name: Config directory
      ansible.builtin.file:
        path: /etc/app
        state: directory
        mode: "0755"

    - name: Database credentials
      ansible.builtin.template:
        src: db.conf.j2
        dest: /etc/app/db.conf
        owner: root
        group: root
        mode: "0600"

Two leaks, two mechanisms:

1. debug prints by design. Someone added it to check the connection string once, and it shipped.
2. The template's result. In the log, the "Database credentials" result carries a diff with the whole rendered file:

TASK [Database credentials] ****************************************************
task path: /builds/platform/orders-ansible/site.yml:15
changed: [db-1] => {
    "changed": true,
    "checksum": "4d7c1f0bb2a0e8f65b0d59c6d5b8f1a29e5a3c10",
    "dest": "/etc/app/db.conf",
    "diff": [
        {
            "after": "# Ansible managed\n[database]\nhost = db-1\nuser = orders\npassword = Bl4ckP3pper-77\n",

Diff mode (--diff, or diff_always in the CI's config) puts the new file content into the result, and verbose output (-v and up) prints results in full. CI jobs often run with both "to have good logs", and those logs keep everything for months.

The other common leaks work the same way: a failing task that echoes its command line (shell: mysql -p{{ password }}), a register printed later, -vvv showing a module's arguments, or a callback plugin that ships results to a log store.

Diagnosis

1. Find every place. grep the log for the secret itself, then look at which task printed each line. Check every log from the same pipeline, not only the one in the ticket.
2. Run it again, verbosely, on purpose. Run the playbook the way CI does (same -v level, same diff setting) and grep the output:

$ ansible-playbook site.yml -v 2>&1 | grep -n -i 'Bl4ck\|censored\|msg'
10:    "msg": "postgres://orders:Bl4ckP3pper-77@db-1:5432/orders"

Even at -v, the debug task prints it.

The fix, in the right order

1. Rotate first. The secret is compromised the moment it reached a shared log. Cleaning the log does not undo who already read it, and fixing the playbook only protects the next secret. Change the password in the database and in the vault, then deploy:

$ printf 'vault_db_password: "Gr33nT34-2026"\n' > group_vars/db/vault.yml && ansible-vault encrypt group_vars/db/vault.yml
Encryption successful

(ansible-vault edit does the same without the plaintext ever touching the disk. Use it when you are not in a throwaway lab.)

2. Make the tasks unable to print it. Delete the debug task, because it exists only to print the secret. Mark every task that handles the secret with no_log: true:

    - name: Database credentials
      ansible.builtin.template:
        src: db.conf.j2
        dest: /etc/app/db.conf
        owner: root
        group: root
        mode: "0600"
      no_log: true

no_log replaces the whole result, at every verbosity and with diff mode on:

$ ansible-playbook site.yml -v
...
TASK [Database credentials] ****************************************************
ok: [db-1] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}

And at -vvv, which is what the job ran:

TASK [Database credentials] ****************************************************
task path: /home/learner/oncall-lab/labs/1a-ansible/inc-leak/site.yml:11
...
changed: [db-1] => {
    "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result",
    "changed": true
}

The cost is real: when that task fails, you see no error either. Keep no_log on the few tasks that touch secrets, not on whole plays. To debug one, run it with no_log: "{{ not debug_secrets | default(false) }}" on a private terminal, never in CI. For a template you can also turn off diff output for just that task with diff: false.

3. Remove the leaked logs from the CI system and from anywhere they were copied (artifacts, log shipping). Ask the CI admins: many systems keep job logs after you delete the run.

Keeping it from coming back

• Search the logs in CI. A pipeline step that greps the job output for known secret patterns, or the CI's own masking. GitHub Actions masks registered secrets, and GitLab masks masked variables. Masking only knows the values you registered, so it misses a decrypted vault value. no_log is still the fix.
• Lint for it. ansible-lint's no-log-password rule flags tasks that pass a password without no_log (it is in the opt-in rule list, so enable it).
• Keep -v/-vvv and --diff out of scheduled runs, or keep those logs as tightly controlled as the secrets themselves.
• Prefer short-lived secrets. A credential that Vault issues for an hour is worth far less in a leaked log than a password that lives for years. That is the Vault Agent pattern, and Terraform has the same leak with the same order of cleanup. If you move the secret into Vault, mind the paths in the policy: KV v2 reads at data/.

Practise it

The Ansible chapter has Incident: the password is in the job log: find both leaks, fix the playbook so nothing can print the secret, rotate it in the vault, prove a -vvv run prints neither the old nor the new password, and delete the log. The lesson Secrets: ansible-vault and no_log covers vault IDs, encrypt_string and every way a decrypted value gets out.