OnCallReadyStart learning

Security and secrets Lesson 23.2 · Kubernetes Security (CKS) · 8 min read

Audit logging: who did what to the API

In plain words

The API server is the front desk of the cluster: every request goes through it. Audit logging is the desk's visitor book. It writes down who came, what they asked for, at what time, and whether they got it. The audit Policy decides how much detail goes into the book, and the most important rule is that you never copy a Secret's contents into it.

The API server can write down every request

Every kubectl command is an HTTP request to kube-apiserver. With auditing on, the apiserver writes one JSON event per request (per stage) to a log: who (user, groups), what (verb, resource, namespace, name), when, and the answer (code, the RBAC decision). After a breach this log is how you find out what the stolen credentials touched.

The Policy decides how much is written. Rules are checked in order and the first match wins; a request no rule matches is not logged. Levels: None (nothing), Metadata (who/what/when, no bodies), Request (plus the request body), RequestResponse (plus the response body).

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  # no events for the noisy, harmless reads
  - level: None
    resources:
      - group: ""
        resources: ["events"]
  # who touched a Secret: metadata only - never log the values
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
  # everything else: metadata
  - level: Metadata

Never log Secrets above Metadata: a Request level would write the Secret values into a plain log file. omitStages: ["RequestReceived"] halves the volume: only the finished request is logged (stage ResponseComplete).

Turning it on is an edit to the static pod manifest on the control plane (chapter 20):

    - kube-apiserver
    - --audit-policy-file=/etc/kubernetes/audit/policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=30
...
    volumeMounts:
    - mountPath: /etc/kubernetes/audit/policy.yaml
      name: audit
      readOnly: true
    - mountPath: /var/log/kubernetes/audit/
      name: audit-log
      readOnly: false
...
  volumes:
  - hostPath:
      path: /etc/kubernetes/audit/policy.yaml
      type: File
    name: audit
  - hostPath:
      path: /var/log/kubernetes/audit/
      type: DirectoryOrCreate
    name: audit-log

The apiserver runs in a container: the flags point at paths inside it, so the policy file and the log directory must be mounted - a hostPath volume (the node's path) and a volumeMount (where the container sees it) for each. Forget either and the apiserver cannot read its policy: it exits at start, kubectl gets "connection refused", and the reason is only in the container log on cp-1 - the classic exam trap:

sudo crictl ps -a --name kube-apiserver
sudo crictl logs --tail 1 <the newest container id>
E1021 09:14:03.480211       1 run.go:72] "command failed" err="loading audit policy file: failed to read file path \"/etc/kubernetes/audit/policy.yaml\": open /etc/kubernetes/audit/policy.yaml: no such file or directory"

A typo in a level (Metadatas) ends the same way, with rules[0].level: Unsupported value. Mount the log directory, not the file: the apiserver creates and rotates the files in it. Without that mount the apiserver still runs, but writes the log inside its own container where no one reads it.

The policy is read at start. Editing the policy file later changes nothing until the apiserver restarts. A change to the manifest restarts it (the kubelet sees a new pod); for a policy-only change, stop the container (sudo crictl stop <id>) and the kubelet starts a new one that reads the file.

One event (formatted with jq):

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","verb":"get",
 "user":{"username":"kubernetes-admin","groups":["kubeadm:cluster-admins","system:authenticated"]},
 "objectRef":{"resource":"secrets","namespace":"payments","name":"db-creds","apiVersion":"v1"},
 "responseStatus":{"metadata":{},"code":200},"stage":"ResponseComplete", ...}

A request the cluster refused is logged with the answer it got. Authorization still says allow when admission said no; responseStatus then holds the API's error: "code":400 and the webhook's message for an admission webhook's denial, 422 (or 403) with "reason", "message" and "details" for the built-in admission policies (later in this chapter). Admission adds its own annotations too: a built-in policy that only audits writes validation.policy.admission.k8s.io/validation_failure, and every mutating webhook the API server called writes mutation.webhook.admission.k8s.io/round_0_index_<n> with "mutated":true or false.

In an interview: "the audit log answers who did what to the cluster; it does not see what happens inside a container - that is runtime detection (Falco)."

On the exam: domain "Monitoring, logging and runtime security": enable auditing with a given policy, then find an event.

Why it helps

After a breach the first question is always what the stolen credentials touched. Without an audit log nobody can answer it, and the investigation becomes guesswork. Knowing the levels also prevents the classic mistake of writing Secret values into a plain log file that far more people and systems can read than the Secret itself, and enabling auditing correctly is a regular CKS task.

FAQ

What are the audit levels?

None writes nothing; Metadata records who, what, when and the answer; Request adds the request body; RequestResponse adds the response body as well. Higher levels give more detail and more risk, because bodies can contain Secret values and personal data.

Which rule applies to a request?

The first rule in the Policy that matches it, checked from the top. A request that no rule matches is not logged at all, which is why many policies end with a catch-all rule at Metadata level.

Why log Secrets at Metadata only?

Request and RequestResponse would copy the Secret values into the audit log. That log is read by more people and shipped to more systems than the Secret itself, so it would become a second, weaker copy of every password in the cluster.

What does omitStages: ["RequestReceived"] do?

It drops the event the apiserver writes the moment a request arrives, so each request is logged once, at ResponseComplete, with its result. That roughly halves the log volume without losing anything an investigation needs.

Why is nothing logged after I added the flags?

The apiserver runs in a container: the policy file and the log directory must also be mounted into it with hostPath volumes and volumeMounts. Without the mounts it cannot read its policy, and a real apiserver would not even start.

In an interview Mid

How do you find out what a compromised ServiceAccount did in the cluster?

From the API server audit log. With auditing on - an audit Policy, --audit-policy-file and --audit-log-path, and hostPath mounts for both - every request is an event with the user, verb, resource, namespace, name and response code, logged at Metadata or above. You grep the log for that ServiceAccount's username and list what it read or changed, then revoke its access.

Also asked: What are the audit levels and which should Secrets get? · What does omitStages do? · Why would an audit flag have no effect?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.