OnCallReadyStart learning

Security and secrets Lesson 23.11 · Kubernetes Security (CKS) · 14 min read

A firewall on the nodes: what a kubeadm node must accept, and what ufw cannot see

In plain words

A node is a house with several doors: one for the admins (SSH), one for the control plane to ask the kubelet things, one the neighbours use to swap address books (Calico's BGP), and on cp-1 the big front door of the API. A host firewall is a doorman with a guest list: a door stays shut unless the list names who may use it. Write the list before you lock up, or you lock yourself out. And one door the doorman never sees: NodePort visitors are led around the house through the back garden.

Who has to reach a node

A NetworkPolicy filters traffic to and from pods. The node itself runs processes that listen on its own address - sshd, the kubelet, on cp-1 the apiserver and etcd - and nothing in Kubernetes filters those. That is the job of a host firewall on the node. Before you write one rule, list who needs what. The Kubernetes docs page "Ports and Protocols" does it for a kubeadm cluster:

NodePortPurposeUsed by
control plane6443/tcpKubernetes API serverall: kubelets, kubectl, pods
control plane2379-2380/tcpetcd client and peer APIkube-apiserver, etcd
control plane10257, 10259/tcpcontroller-manager, schedulerself (kubeadm binds them to 127.0.0.1)
every node10250/tcpKubelet API (logs, exec, metrics)self, control plane
worker10256/tcpkube-proxy health checkself, load balancers
worker30000-32767/tcp+udpNodePort Servicesall

Add what the docs leave to you: SSH (22/tcp) for the admins, and the CNI. This lab runs Calico with BGP and IP-in-IP (IPIP Always): BGP is TCP 179 between all nodes, IP-in-IP is IP protocol 4. In the lab: oncall-lab is 10.64.0.2, cp-1 10.64.0.10, the workers .11 and .12, all in 10.64.0.0/24. On this single control plane etcd talks only to the apiserver next to it, so nothing outside cp-1 needs 2379-2380.

ufw in five commands

Ubuntu ships ufw (ufw 0.36.2 on these 24.04 nodes): installed, inactive. It writes iptables rules for you; every command needs root.

learner@worker-1:~$ sudo ufw default deny incoming
Default incoming policy changed to 'deny'
(be sure to update your rules accordingly)
learner@worker-1:~$ sudo ufw allow from 10.64.0.2 to any port 22 proto tcp
Rules updated
learner@worker-1:~$ sudo ufw allow from 10.64.0.10 to any port 10250 proto tcp
Rules updated
learner@worker-1:~$ sudo ufw allow from 10.64.0.0/24 to any port 179 proto tcp
Rules updated
learner@worker-1:~$ sudo ufw enable
Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
Firewall is active and enabled on system startup
learner@worker-1:~$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW IN    10.64.0.2
10250/tcp                  ALLOW IN    10.64.0.10
179/tcp                    ALLOW IN    10.64.0.0/24
  • Rules are checked in order and the first match wins; what matches none gets the default policy. deny = DROP: the client waits and times out. reject answers at once ("Connection refused"). sudo ufw status numbered numbers them, sudo ufw delete 2 removes one, sudo ufw insert 1 RULE puts one first.
  • Added while inactive, a rule only reaches the files ("Rules updated"); once active, ufw allow answers "Rule added". The rules live in /etc/ufw/user.rules, enable sets ENABLED=yes in /etc/ufw/ufw.conf: it comes back after a reboot.
  • A rule without a source address is added for IPv6 too: "Rules updated (v6)".
  • Blocked packets are logged by the kernel: sudo journalctl -k -g "UFW BLOCK" shows SRC= and DPT= - who knocked on which port.
  • "Used by: control plane" is the minimum. metrics-server (kubectl top) scrapes every kubelet from its own pod, so its traffic arrives from the node it runs on: allow 10250 from the node network, or kubectl top nodes shows <unknown> for the locked node.

Do not lock yourself out

Allow SSH from your admin address before enable. The session you are typing in survives enable - its packets belong to an established connection - so a missing rule shows only at the next login: ssh: connect to host worker-1 port 22: Connection timed out. On a real VM the way back is the console; in a cluster, kubectl debug node/worker-1 -it --image=busybox:1.36 and chroot /host give you a root shell on the node - but only while the kubelet API (10250) still lets the control plane in.

The breakage that waits for a restart

That rule about established connections goes for the cluster too. A worker's kubelet keeps one connection open to the apiserver; Calico keeps a BGP session to every other node. Close 6443 on cp-1 and every node stays Ready - until a kubelet restarts and cannot reconnect:

kubelet_node_status.go:548] "Error updating node status, will retry" err="error getting node \"worker-2\": Get \"https://10.64.0.10:6443/api/v1/nodes/worker-2?resourceVersion=0&timeout=10s\": context deadline exceeded"

and 50 seconds later the node is NotReady. BGP is the same, with a twist: Calico's node-to-node mesh is one-directional - the node with the higher address connects to the other one's 179 (confd sets the lower side passive). Lock down worker-1 without 179 and nothing happens until a reboot; then worker-2 (.12) cannot open worker-1:179, the route between their pod networks is withdrawn and pods on the two nodes stop reaching each other. calico-node on worker-1 is not ready for its first 30 seconds ("BGP not established with 10.64.0.12"), then Ready again with one peer up - while the route is still missing. Test a firewall with a reboot, not only with the session you configured it from.

What ufw cannot see

ufw appends its chains to INPUT, FORWARD and OUTPUT. Calico inserts its own first and kube-proxy prepends its jumps, so on a Kubernetes node they run before ufw:

learner@worker-1:~$ sudo iptables -S INPUT
-P INPUT DROP
-A INPUT -m comment --comment "cali:Cz_u1IQiXIMmKD4c" -j cali-INPUT
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes load balancer firewall" -j KUBE-PROXY-FIREWALL
-A INPUT -m comment --comment "kubernetes health check service ports" -j KUBE-NODEPORTS
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A INPUT -j ufw-before-logging-input
-A INPUT -j ufw-before-input
...
  • NodePorts are not filtered by ufw. kube-proxy rewrites a NodePort packet to a pod address in PREROUTING (DNAT), so it never reaches INPUT; it leaves through FORWARD, where KUBE-FORWARD and Calico accept it before ufw's chains run. sudo ufw deny 30080/tcp changes nothing. To keep NodePorts to an admin network: do not use NodePort for internal services, limit the node addresses that serve them (kube-proxy nodePortAddresses), filter in front of the nodes (a cloud security group, a network firewall), or use Calico host endpoint policies with preDNAT: true, which run before the DNAT.
  • Pods talking to their own node pass Calico (FELIX_DEFAULTENDPOINTTOHOSTACTION=ACCEPT in calico.yaml); IP-in-IP from the other nodes is accepted by cali-INPUT ("Allow IPIP packets from Calico hosts"). ufw could not even name it: ufw allow proto ipip is "Unsupported protocol". Calico's own docs say to disable firewalld-style iptables managers and use host endpoint policies instead - on exam nodes and small clusters, ufw is what you will meet.

One rule set, three views - and IPv6

Ubuntu 24.04's iptables is iptables-nft (its version line ends in "(nf_tables)"): every rule kube-proxy, kubelet, Calico and ufw write is an nftables rule in the kernel, and nft lists them all, decoded its own way:

learner@worker-1:~$ sudo nft list chain ip filter INPUT
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy drop;
		 counter packets 0 bytes 0 jump cali-INPUT
		ct state new  counter packets 0 bytes 0 jump KUBE-PROXY-FIREWALL
		 counter packets 0 bytes 0 jump KUBE-NODEPORTS
		...
		counter packets 0 bytes 0 jump ufw-before-input
		...
	}
}
  • nft translates each iptables match it can: -m conntrack --ctstate NEW is ct state new, a ufw rule's -p tcp --dport 22 is tcp dport 22. -m comment prints as nothing (the double spaces), matches without a translation stay xt match "recent", and some translations lose detail: KUBE-FIREWALL's ! --ctstate RELATED,ESTABLISHED,DNAT reads ct status dnat. Read these rules with iptables -S / iptables-save; use nft list ruleset to see everything at once (both families). The warning means it: kube-proxy and ufw rewrite these tables, so never edit them with nft. (simulator) the counters stay at 0.
  • IPv6: a rule without a source (ufw allow 22/tcp) is also written for IPv6 ("Rules updated (v6)") and loaded into ufw6-* chains: sudo ip6tables -S ufw6-user-input. from 10.64.0.2 rules are IPv4 only - IPv6 then meets the default policy. This lab is IPv4-only (the nodes have link-local fe80:: addresses and nothing more), so no IPv6 traffic reaches them; on a dual-stack network check ufw status for the (v6) lines too.

Two layers, two jobs

Host firewall (ufw)NetworkPolicy
Protectsthe node's own ports: SSH, kubelet, apiserver, etcdpods
Writtenper node, by an adminin the cluster, per namespace
Seestraffic to the node's addresstraffic to and from pod addresses

Neither replaces the other: a NetworkPolicy cannot keep the internet off a node's SSH, and ufw cannot isolate two pods.

In an interview: "default deny incoming, then the ports from the Ports and Protocols page from the sources that need them, SSH from the admin network first - and NodePorts need a firewall in front of the nodes or Calico preDNAT policy, because ufw never sees them".

On the exam: domain "System hardening": "minimize external access to the network". Allow SSH before enable, and check sudo ufw status before you move on.

Why it helps

Kubernetes filters pod traffic with NetworkPolicies, but the node's own ports - SSH, the kubelet API, the apiserver, etcd - need a host firewall, and "minimize external access" is a CKS competency. Getting it wrong is quiet at first: established connections keep working, so a missing rule shows up at the next kubelet restart or reboot as a NotReady node or broken pod routes. Knowing which ports kubeadm needs, and what ufw cannot filter, is what makes the lockdown safe.

FAQ

Which ports does a kubeadm worker have to accept?

The kubelet API on 10250/tcp from the control plane, NodePorts 30000-32767 if you use them, and kube-proxy's health check on 10256 only if a load balancer checks the nodes. Add SSH from your admin network and the CNI's needs: for Calico with BGP that is 179/tcp between the nodes.

Why did my node stay Ready after I closed 6443 on cp-1?

Because the kubelet's connection to the apiserver was already established, and ufw lets established connections through. The node only goes NotReady when that kubelet reconnects - after a restart or a reboot - and cannot get in: its journal then says "Error updating node status, will retry".

Can ufw close a NodePort?

No. kube-proxy rewrites NodePort packets to a pod address before the INPUT chain, and they are accepted in FORWARD by kube-proxy's and Calico's rules, which run before ufw's appended chains. Filter NodePorts in front of the nodes, with kube-proxy nodePortAddresses, or with Calico preDNAT host endpoint policy.

What is the difference between deny and reject?

deny drops the packet, so the client waits until it times out ("Connection timed out", or i/o timeout in Kubernetes errors). reject answers at once, with a TCP reset for tcp rules, so the client sees "Connection refused". The default policy deny incoming drops.

How do I get back in after locking out SSH?

Your open session normally survives enable, so fix the rule from there. If it is gone: use the VM console, or kubectl debug node/NAME -it --image=busybox:1.36 and chroot /host for a root shell on the node - which only works while the kubelet API still accepts the control plane.

In an interview Mid

How would you firewall the nodes of a kubeadm cluster?

Start from the Kubernetes "Ports and Protocols" page: 6443 on the control plane, 10250 on every node from the control plane, 2379-2380 only between apiservers and etcd, NodePorts if used, plus the CNI (179/tcp for Calico BGP) and SSH from the admin network. With ufw: sudo ufw default deny incoming, one allow from ... to any port ... proto tcp per need, SSH before sudo ufw enable, then prove it with a reboot - established connections hide gaps. NodePorts bypass ufw, so filter them in front of the nodes.

Also asked: Why does ufw not filter NodePort traffic? · What is the difference between a host firewall and a NetworkPolicy? · Why should you test a node firewall with a reboot?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.