Chapter 17 Kubernetes: Scheduling, Health & Security
Requests vs limits, QoS and eviction, CPU throttling vs OOMKill, placement (affinity, taints, spread, priority), probes and the restart storm, rollouts, PDBs and drains, the HPA, RBAC and ServiceAccounts, securityContext, Pod Security Standards and Secrets.
In plain words
Imagine running a busy summer camp. You have to decide which cabin each kid sleeps in (some cabins are only for older kids, some kids must be with a friend, siblings shouldn't all be in one cabin). You check on the kids regularly: is anyone sick, is anyone ready for the activity? You swap groups slowly so there's always someone at the lake. And you hand out keys: counsellors get the kitchen key, kids don't, and the key to the first-aid cabinet is hidden properly, not under the doormat.
That is this chapter. Requests, limits and QoS decide how much room each pod gets and who is sent home first. Affinity, taints and topology spread decide which node it sleeps on. Probes, rollouts, PDBs and the HPA keep it healthy and available. RBAC, ServiceAccounts, securityContext, Pod Security and Secrets decide who holds which key.
Why it matters on call
This is the chapter behind most day-to-day platform tickets. "Our pod is Pending" (requests, taints, affinity), "it restarts every few hours with 137" (memory limits, the JVM trap), "it's slow but CPU looks fine" (throttling), "the node upgrade is stuck" (a PDB), "the rollout replaced every good pod with a broken one" (no readiness probe), "Forbidden: cannot list pods" (RBAC). Each has a precise mechanism, and knowing it turns a guess into a two-command answer.
It is also where the security review of every workload happens: non-root, dropped capabilities, restricted Pod Security, no list secrets for humans. On the CKA, workloads and scheduling plus troubleshooting are a large share of the points. It comes after networking and storage because PDBs, zones and probes all build on Services and volumes, and before cluster operations, where you drain the nodes these rules protect.
Lessons
- Requests and limits: what each one actually controls
- QoS classes, oom_score_adj, and who gets evicted first
- CPU throttling vs memory OOMKill: two failure modes, two fixes
- LimitRange and ResourceQuota: per-pod defaults vs namespace totals
- nodeSelector and node affinity
- Taints and tolerations: the node repels, the pod tolerates
- Pod affinity, anti-affinity and topology spread
- Reading FailedScheduling, and PriorityClass with preemption
- Liveness, readiness, startup: what each answers and what failure does
- Probe design: the restart storm, and why JVMs need startup probes
- Rolling updates: maxSurge, maxUnavailable, and the rollout verbs
- PodDisruptionBudgets and why they matter during a drain
- The HorizontalPodAutoscaler
- RBAC in full: Role, ClusterRole, RoleBinding, ClusterRoleBinding
- ServiceAccounts, tokens, and automountServiceAccountToken
- kubectl auth can-i as a debugging tool
- securityContext: the fields worth setting every time
- Pod Security Standards and Pod Security Admission
- Secrets, and why base64 is not a security measure
- Users are certificates: the CSR flow end to end
55 hands-on labs (missions, incidents and drills) run in the terminal: Open this chapter in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.
Questions people ask
What's the difference between requests and limits in one sentence?
Requests are what the scheduler reserves for the pod on a node and what decides its share of CPU under contention; limits are hard ceilings the kernel enforces through cgroups, where exceeding the CPU limit means throttling and exceeding the memory limit means an OOM kill. The scheduler never looks at limits or at actual usage.
Why is my pod Pending when the node's CPU is almost idle?
Because scheduling is bookkeeping on requests, not usage. If the pods on a node already request most of its allocatable CPU, a new pod asking for more doesn't fit, even at 3% real usage. k describe node shows "Allocated resources" (the ledger); k top node shows real usage. The gap is usually over-requested pods, and right-sizing them is a platform task.
Do I need liveness, readiness and startup probes on every container?
Readiness, almost always, for anything behind a Service: it controls traffic and protects rollouts. Startup, for anything slow to start, such as JVMs. Liveness only when the process can get stuck while still running, and it must be cheap and local. A badly designed liveness probe that checks the database is a classic cause of self-inflicted outages.
Is RBAC enough to protect Secrets?
No. RBAC controls API access, but anyone who can create pods in a namespace can mount any Secret there; secrets sit in etcd unencrypted unless you configure encryption at rest; mounted secrets are files on the node; and env vars leak through /proc and debug endpoints. In a bank the source of truth is an external secrets manager (a vault service), fetched by the pod with its own identity.
How much of this is on the CKA?
A lot. Workloads and scheduling (requests, affinity, taints, rollouts, HPA, probes) and cluster architecture (RBAC) together with troubleshooting make up well over half of the curriculum. Expect tasks like "make this pod schedule only on nodes labelled X", "create a Role and RoleBinding so this ServiceAccount can list pods", "fix the failing rollout", and reading FailedScheduling messages.