Everything else in this chapter checks who may come in. Falco is the motion sensor inside the building: it watches what programs in containers actually do and sounds an alarm when one does something no normal program should, like opening a shell by hand or reading the file of passwords. The alarm names the room it came from, so you know where to go.
When an attacker is already inside
Every control so far is about what may start. Falco watches what containers do (it reads system calls through an eBPF probe on every node) and raises an alert when a rule matches. Three default rules catch most hands-on attackers:
Rule
Priority
Fires when
Terminal shell in container
Notice
someone opens an interactive shell in a container
Read sensitive file untrusted
Warning
a process reads /etc/shadow, /etc/sudoers...
Drop and execute new binary in container
Critical
a program runs whose file was written after the container started: it is not in the image
An alert names the process, the command line and the pod:
10:42:07.123456789: Warning Sensitive file opened for reading by non-trusted program |
file=/etc/shadow ... process=cat command=cat /etc/shadow ... k8s_pod_name=debug-7f k8s_ns_name=tools
The third one matters most for applications: an attacker who gets code to run through the app (no shell, no kubectl) usually writes a program somewhere writable and starts it. The alert names it: Executing binary not part of base image | proc_exe=/tmp/... parent=java ....
(simulator) Falco writes to /var/log/falco/falco.log on the node the pod runs on (its file_output); it watches kubectl exec and what the labs' scenarios start.
Responding: find the pod and its node, keep the evidence (the alert, kubectl get pod -o yaml, kubectl logs, one-shot ps and netstat -tnp inside it), cut it off (a NetworkPolicy that denies all for that pod, or scale it to 0), then delete it and find how the access happened (the audit log: who ran exec? No exec at all means the way in was the application itself).
A new policy does not cut a connection that already exists. Calico checks a NetworkPolicy on a flow's first packet; conntrack then lets the rest of that flow through. So right after your deny policy, netstat -tnp in the pod still shows the old connection ESTABLISHED. On the pod's node, sudo conntrack -L -d <ip> shows its entry and sudo conntrack -D -d <ip> deletes it: the next packet is a new flow, meets the policy and is dropped.
On the exam: "find which pod triggered a Falco rule and delete it / scale its deployment to 0".
Why it helps
Attackers who get past the other controls are found by their behaviour, not by policy. Reading a Falco alert and turning it into the pod, the node and a response is a CKS task and an everyday on-call skill. It also shows why runtime detection and the audit log answer different questions and why you need both during an incident.
FAQ
How does Falco see what containers do?
It reads the system calls on every node, through an eBPF probe, and matches them against its rules. Because it watches at the kernel level, a process inside the container cannot hide what it does from Falco.
Which default rules catch most attackers?
Terminal shell in container, priority Notice, and Read sensitive file untrusted, priority Warning. They fire when someone opens an interactive shell in a container or a program reads files such as /etc/shadow. Most hands-on attacks trip at least one of them.
How do I find the pod from an alert?
The alert line carries k8s_pod_name and k8s_ns_name, plus the process name and the full command line. It is written on the node where the pod runs, so you look on that node or in your log pipeline.
What is the first response?
Keep the evidence, contain the pod with a deny-all NetworkPolicy or by scaling it to zero, remove it, then find who opened the shell in the API audit log, fix that access and rotate any credentials the pod could reach.
Does Falco replace the audit log?
No. Falco sees what happened inside a container; the audit log sees who asked the API for it, for example the exec request that opened the shell. An investigation needs both to answer what and who.
In an interview Mid
A Falco alert says a shell was spawned in a production container. What do you do?
Read the alert for the pod, namespace and command, keep the evidence (the alert, the pod YAML, the node), contain the pod with a deny-all NetworkPolicy or by scaling it to zero, remove it, then search the API audit log for the exec request to learn which identity opened the shell, fix that access and rotate the credentials it could reach. Falco tells you what happened; the audit log tells you who.
Also asked: What can Falco see that the audit log cannot? · How do you contain a compromised pod without losing evidence? · How would you write a custom Falco rule?