The API server can write down every request
Every kubectl command is an HTTP request to kube-apiserver. With auditing on, the apiserver writes one JSON event per request (per stage) to a log: who (user, groups), what (verb, resource, namespace, name), when, and the answer (code, the RBAC decision). After a breach this log is how you find out what the stolen credentials touched.
The Policy decides how much is written. Rules are checked in order and the first match wins; a request no rule matches is not logged. Levels: None (nothing), Metadata (who/what/when, no bodies), Request (plus the request body), RequestResponse (plus the response body).
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
- "RequestReceived"
rules:
# no events for the noisy, harmless reads
- level: None
resources:
- group: ""
resources: ["events"]
# who touched a Secret: metadata only - never log the values
- level: Metadata
resources:
- group: ""
resources: ["secrets", "configmaps"]
# everything else: metadata
- level: Metadata
Never log Secrets above Metadata: a Request level would write the Secret values into a plain log file. omitStages: ["RequestReceived"] halves the volume: only the finished request is logged (stage ResponseComplete).
Turning it on is an edit to the static pod manifest on the control plane (chapter 20):
- kube-apiserver
- --audit-policy-file=/etc/kubernetes/audit/policy.yaml
- --audit-log-path=/var/log/kubernetes/audit/audit.log
- --audit-log-maxage=30
...
volumeMounts:
- mountPath: /etc/kubernetes/audit/policy.yaml
name: audit
readOnly: true
- mountPath: /var/log/kubernetes/audit/
name: audit-log
readOnly: false
...
volumes:
- hostPath:
path: /etc/kubernetes/audit/policy.yaml
type: File
name: audit
- hostPath:
path: /var/log/kubernetes/audit/
type: DirectoryOrCreate
name: audit-log
The apiserver runs in a container: the flags point at paths inside it, so the policy file and the log directory must be mounted - a hostPath volume (the node's path) and a volumeMount (where the container sees it) for each. Forget either and the apiserver cannot read its policy: it exits at start, kubectl gets "connection refused", and the reason is only in the container log on cp-1 - the classic exam trap:
sudo crictl ps -a --name kube-apiserver
sudo crictl logs --tail 1 <the newest container id>
E1021 09:14:03.480211 1 run.go:72] "command failed" err="loading audit policy file: failed to read file path \"/etc/kubernetes/audit/policy.yaml\": open /etc/kubernetes/audit/policy.yaml: no such file or directory"
A typo in a level (Metadatas) ends the same way, with rules[0].level: Unsupported value. Mount the log directory, not the file: the apiserver creates and rotates the files in it. Without that mount the apiserver still runs, but writes the log inside its own container where no one reads it.
The policy is read at start. Editing the policy file later changes nothing until the apiserver restarts. A change to the manifest restarts it (the kubelet sees a new pod); for a policy-only change, stop the container (sudo crictl stop <id>) and the kubelet starts a new one that reads the file.
One event (formatted with jq):
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","verb":"get",
"user":{"username":"kubernetes-admin","groups":["kubeadm:cluster-admins","system:authenticated"]},
"objectRef":{"resource":"secrets","namespace":"payments","name":"db-creds","apiVersion":"v1"},
"responseStatus":{"metadata":{},"code":200},"stage":"ResponseComplete", ...}
A request the cluster refused is logged with the answer it got. Authorization still says allow when admission said no; responseStatus then holds the API's error: "code":400 and the webhook's message for an admission webhook's denial, 422 (or 403) with "reason", "message" and "details" for the built-in admission policies (later in this chapter). Admission adds its own annotations too: a built-in policy that only audits writes validation.policy.admission.k8s.io/validation_failure, and every mutating webhook the API server called writes mutation.webhook.admission.k8s.io/round_0_index_<n> with "mutated":true or false.
In an interview: "the audit log answers who did what to the cluster; it does not see what happens inside a container - that is runtime detection (Falco)."
On the exam: domain "Monitoring, logging and runtime security": enable auditing with a given policy, then find an event.