Kubernetes: Cluster Operations & Troubleshooting: interview questions

kubectl returns "connection refused" to the API server. How do you troubleshoot it? Mid

First check it is the real server: if the message says localhost:8080, you have no kubeconfig. If it names 10.64.0.10:6443, the apiserver is not answering - and kubectl cannot help, so go to the control-plane node:

1. ssh cp-1, then sudo crictl ps -a --name kube-apiserver.
2. Not there at all - the kubelet could not parse the static pod manifest: journalctl -u kubelet | grep -i manifest names the file and line in /etc/kubernetes/manifests/kube-apiserver.yaml.
3. Crash-looping (ATTEMPT climbing) - sudo crictl logs <id> gives the apiserver's own error: a bad flag, a missing certificate file, or etcd unreachable (then check etcd the same way).
4. Running, yet refused - is the kubelet itself running (systemctl status kubelet)?
5. Fix the file (back it up outside the manifests directory first); the kubelet restarts the pod within about 20 s. Prove it with kubectl get nodes.

If kubectl says x509: certificate has expired instead, it is the certificates: sudo kubeadm certs check-expiration.

Also asked: How do you upgrade a kubeadm cluster by one minor version? · How do you back up etcd? · A node is NotReady. How do you work out why?

A node shows NotReady. What do you check first on the node itself? Junior

The kubelet is a systemd service, not a pod, so the Linux tools apply. On the node:

1. systemctl status kubelet - active? Or activating (auto-restart) (it has Restart=always and no start limit, so a broken kubelet loops every 10 s instead of showing failed). enabled too, or it breaks at the next reboot.
2. journalctl -u kubelet -n 50 - read the klog E lines; the err= field is the why: a missing /var/lib/kubelet/config.yaml, cni plugin not initialized (nothing in /etc/cni/net.d), PLEG is not healthy (the runtime), or dial tcp 10.64.0.10:6443: connection refused (this node is fine - the apiserver is not).
3. systemctl status containerd - the runtime must be up.
4. sudo crictl ps -a - what the runtime is really running, even when the apiserver is dead.
5. df -h / - a full disk.

Start from kubectl describe node Conditions, then go to the machine when the message points there.

Also asked: What is the role of the container runtime, and how do you debug it when kubectl cannot help? · Where are the important kubeadm files on a node? · What is crictl, and when do you use it instead of kubectl?

Learn it: 18.1 Under kubectl: the nodes are Linux machines

What are static pods, and why does kubeadm use them for the control plane? Junior

A static pod is run by the kubelet straight from a manifest file in its staticPodPath - /etc/kubernetes/manifests on kubeadm - with no scheduler, no API server and no controller: file present = pod running, file removed = pod stopped.

kubeadm uses them to break the chicken-and-egg problem: the API server stores pods, but something must run the API server first. So systemd starts containerd and the kubelet, the kubelet starts etcd and kube-apiserver from files, then the controller-manager and scheduler connect to it.

The kubelet also creates a read-only mirror pod (kube-apiserver-cp-1, owner Node) so kubectl can see it; deleting the mirror changes nothing.

To change a component, edit its manifest - the kubelet restarts it within about 20 s. Back up outside the directory (a backup inside it is run as a second pod). A YAML parse error makes the pod vanish (the kubelet journal says why); a bad flag makes it crash-loop (crictl logs).

Also asked: How do you add a flag to the kube-apiserver on a kubeadm cluster? · A pod is Pending and describe shows no events at all. What is wrong? · How do you restart a static pod without changing it?

Learn it: 18.3 Static pods: how the control plane runs itself

What does kubeadm init actually do, and how does a node join? Mid

kubeadm init builds the control plane in phases:

• preflight - swap off, net.ipv4.ip_forward=1, modules, a runtime, ports free.
• certs - three CAs and every component certificate in /etc/kubernetes/pki (leaves valid one year).
• kubeconfig - admin.conf, controller-manager.conf, scheduler.conf, the kubelet's.
• etcd / control-plane - static pod manifests in /etc/kubernetes/manifests; the kubelet runs them.
• mark-control-plane - the NoSchedule taint.
• bootstrap-token - a token valid 24 h for joining.
• addons - CoreDNS and kube-proxy. Not the CNI: until you apply one, nodes are NotReady.

kubeadm join IP:6443 --token ... --discovery-token-ca-cert-hash sha256:... has two secrets: the token proves the node to the cluster; the CA hash proves the cluster to the node. The kubelet then does a TLS bootstrap - a CSR, auto-approved - and gets its own client certificate. Yesterday's join failing means the token expired: kubeadm token create --print-join-command.

Also asked: What prerequisites does a machine need before kubeadm can use it? · Why should the Kubernetes packages be held with apt-mark? · How do you remove a node from a cluster properly?

Learn it: 18.6 kubeadm init and join: what they actually do

All kubectl commands fail with "x509: certificate has expired". How do you fix it? Mid

kubeadm's leaf certificates are valid one year; a cluster nobody upgraded for a year hits this. "x509 expired" means the apiserver's serving certificate (your own client certificate expiring gives Unauthorized instead). On the control plane:

1. sudo kubeadm certs check-expiration - which ones, how long left (<invalid> = expired). Or openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates.
2. sudo kubeadm certs renew all - new leaves, same CAs and SANs.
3. Restart the static pods - they hold the old certificates in memory: move the manifests out of /etc/kubernetes/manifests and back, or crictl stop the containers.
4. Refresh every kubeconfig copy: sudo cp /etc/kubernetes/admin.conf ~/.kube/config, plus laptops and CI.

The kubelet's own client certificate rotates itself. The real prevention: upgrade on schedule (kubeadm upgrade apply renews everything) and alert at 30 days with openssl x509 -checkend from a timer.

Also asked: Why does a Kubernetes cluster use so many certificates? · What is the difference between "x509: certificate has expired" and "Unauthorized"? · Which certificates does the kubelet renew by itself?

Learn it: 18.11 kubeadm's certificates: who trusts whom, and for how long

Explain the Kubernetes version skew policy and what it means for upgrades. Mid

Each component has its own version, relative to kube-apiserver:

• kubelet and kube-proxy - never newer; up to 3 minor versions older.
• controller-manager, scheduler - never newer; at most 1 older.
• kubectl - within 1 minor, older or newer.
• several apiservers - within 1 minor of each other.

What follows: the control plane goes first (nothing may be newer than the apiserver), then the nodes; and one minor at a time - 1.33 to 1.35 is two full upgrades, and kubeadm refuses to skip. The generous kubelet window lets you upgrade the control plane twice and roll the nodes once, but it is for during an upgrade, not for living there. Patch versions can always be mixed.

Read the versions: kubectl version (Server = apiserver), the VERSION column of kubectl get nodes (each kubelet), kubeadm version.

Also asked: How do you find out which Kubernetes versions are running in a cluster? · Why can you not skip a minor version with kubeadm? · Why must kubeadm be upgraded before you run kubeadm upgrade?

Learn it: 18.15 Version skew: what may run with what

What are the steps to upgrade a kubeadm cluster by one minor version? Mid

First control-plane node:

1. Point apt at the next minor (/v1.35/deb/ in the repository line), apt-get update, unhold and install only kubeadm at the exact version, hold again.
2. sudo kubeadm upgrade plan - what will change, and that kubelets are your job.
3. sudo kubeadm upgrade apply v1.35.3 - the control-plane static pods one by one, etcd and CoreDNS, kube-proxy, certificates renewed, backups in /etc/kubernetes/tmp.
4. kubectl drain cp-1 --ignore-daemonsets, upgrade the kubelet and kubectl packages, systemctl daemon-reload, systemctl restart kubelet, kubectl uncordon cp-1.

Each worker, one at a time: repository line, kubeadm package, sudo kubeadm upgrade node, drain, kubelet + kubectl packages, daemon-reload, restart kubelet, uncordon, wait for Ready.

Mistakes to recognise: "version not found" (repository line not changed), a held package not upgraded, VERSION unchanged (kubelet not restarted), Ready,SchedulingDisabled (forgot to uncordon), a drain stuck on a PDB.

Also asked: What does kubeadm upgrade apply change, and what does it leave to you? · What is the difference between kubeadm upgrade apply and kubeadm upgrade node? · How do you make cluster upgrades routine and low-risk?

Learn it: 18.16 The kubeadm upgrade, step by step

How do you back up etcd in a kubeadm cluster? Junior

etcd holds every object in the cluster, so this is the cluster backup. On the control-plane node, with the TLS flags read from /etc/kubernetes/manifests/etcd.yaml:

sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/backup/etcd-$(date +%F).db

Then:

• check it: etcdutl snapshot status FILE (etcd 3.6 moved status and restore to the offline tool etcdutl);
• treat it like a credential - it contains every Secret: mode 600, encrypted storage;
• copy it off the node, on a schedule (a timer), with an alert when the newest is too old;
• and practise the restore.

Without sudo, or with http:// instead of https://, or without the client certificate, etcdctl fails - each with its own error.

Also asked: What is etcd's role in Kubernetes, and what happens if it goes down? · Why does etcd run 3 or 5 members, and not 4? · What is the difference between etcdctl and etcdutl?

Learn it: 18.20 etcd: the cluster in one database, and how to back it up

How do you restore etcd from a snapshot on a single control-plane kubeadm cluster? Mid

A restore rewinds the entire cluster to the snapshot - everything created since is gone. Use it when the state is lost, not to undo a deploy.

1. Restore into a new data directory: sudo etcdutl snapshot restore /opt/backup/etcd-snap.db --data-dir /var/lib/etcd-restore (without --data-dir it lands in ./default.etcd).
2. Back up etcd.yaml outside the manifests directory, then change the hostPath of the etcd-data volume to /var/lib/etcd-restore. Check with grep.
3. Wait: the kubelet restarts etcd on the restored data; the apiserver reconnects.
4. Verify: etcdctl endpoint health, kubectl get ns, the deleted objects are back.

The trap: changing only the --data-dir flag. That path is inside the container; pointing it at a directory on no volume makes etcd start an empty cluster - "the restore wiped the cluster", and it vanishes again on the next restart. With several control-plane nodes, every member is restored from the same snapshot with its own member settings.

Also asked: What are the risks of an etcd restore? · When would you not use an etcd restore to fix a problem? · How do you verify a restore worked?

Learn it: 18.22 Restoring etcd from a snapshot

How do you safely take a node out for maintenance, and what happens when a node dies unexpectedly? Junior

Planned: kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data - it cordons the node (no new pods) and evicts every pod through the Eviction API, respecting PDBs. --force (bare pods, deleted for good) is a decision, not a habit. Do the work, wait for Ready, kubectl uncordon - nothing moves back by itself.

Unplanned, minute by minute:

• The kubelet stops renewing its Lease; after ~50 s the node goes Ready=Unknown ("Kubelet stopped posting node status") and gets the unreachable NoExecute taint.
• Pods still show Running - the last status reported - but are no longer Ready, so Services stop sending them traffic.
• After 300 s (every pod's default toleration) they are evicted and their controllers recreate them elsewhere.
• StatefulSet pods do not come back: the old pod stays Terminating because no kubelet confirms it stopped, and the StatefulSet will not run two db-0s. Bring the node back, or kubectl delete node / force-delete once you are sure the machine is off.

Also asked: What is the difference between Ready False and Ready Unknown on a node? · Why does a StatefulSet pod not move off a dead node? · What does kubectl uncordon do, and what does it not do?

Learn it: 18.25 Node lifecycle: cordon, drain, NotReady and taint-based eviction

A pod is stuck in ImagePullBackOff. How do you diagnose it? Junior

ErrImagePull is a failed pull; ImagePullBackOff is the kubelet waiting to retry. kubectl describe pod POD - the Events contain the registry's answer verbatim. Read the end of it:

• not found - wrong tag or repository name: fix the image reference (kubectl set image deploy/web web=registry.lab/shop/web:1.2).
• pull access denied, 401 Unauthorized - a private registry without credentials: create a docker-registry Secret (kubectl create secret docker-registry) and reference it in imagePullSecrets.
• no such host, i/o timeout, dial tcp - the node cannot reach the registry: DNS, proxy, firewall. Test from the node with sudo crictl pull.
• 429 Too Many Requests - a registry rate limit: use a mirror or pull-through cache, or authenticate.

The kubelet keeps retrying with back-off, so once the cause is fixed there is no need to delete the pod.

Also asked: A pod is in CrashLoopBackOff. What do you do? · A namespace has been stuck in Terminating for a day. How do you resolve it? · What is a finalizer?

Learn it: 18.28 The failure catalogue I: pods that will not run

A Service has no endpoints. How do you find out why? Junior

The pods look fine and curl to the Service fails. Check what is behind it:

kubectl get endpointslices -l kubernetes.io/service-name=front
kubectl describe svc front
kubectl get pods --show-labels

Two causes, one check each:

1. The selector matches no pods - app=frontend in the Service, app=front on the pods. Test it: kubectl get pods -l app=frontend returns nothing. Fix the Service's selector (a Deployment's selector is immutable).
2. The pods match but are not Ready - 0/1 Running, and describe pod shows Readiness probe failed. Only Ready pods become ready endpoints. Fix the probe or the app.

And the look-alike: endpoints exist but the targetPort is wrong, so connections to the pod IP are refused - compare TargetPort with the container's port.

Also asked: Pods are being evicted with "The node was low on resource: ephemeral-storage". What do you do? · DNS fails inside every pod. How do you debug it? · What is the difference between a DNS timeout and NXDOMAIN?

Learn it: 18.34 The failure catalogue II: nodes, Services, DNS, certificates

What are your first commands when someone says "the cluster is broken"? Mid

Find the layer first - three commands:

1. kubectl get nodes - does the API server answer at all, and which nodes are sick?
2. kubectl get pods -A | grep -v Running - what is not running, and where?
3. kubectl get events -A --sort-by=.lastTimestamp | tail - what just happened?

Then branch:

• kubectl cannot connect - the control plane: ssh cp-1, sudo crictl ps -a --name kube-apiserver, crictl logs, the kubelet journal for manifest errors, kubeadm certs check-expiration for x509. Everything else is noise until this works.
• A node NotReady - describe node Conditions, then on the node: kubelet, containerd, /etc/cni/net.d, df -h /.
• Pods sick, nodes fine - the workload: describe pod Events, logs --previous.

Two rules: read the whole error - it usually names the file or flag; and fix the cause, not the symptom (restarting a kubelet with swap on works only until the reboot). A fix is done when the symptom is gone and the persistent setting is checked.

Also asked: How do you decide whether a problem is in the control plane, a node, or the workload? · How would you approach an incident where many services in a cluster fail at once? · How do you prove a fix will survive a reboot?

Learn it: 18.38 Working a cluster outage without looking anything up