Kubernetes: Architecture & Workloads: interview questions

What are the main components of a Kubernetes cluster, and what does each do? Junior

Control plane (the brain):

• kube-apiserver - the front door. Everything (kubectl, controllers, kubelets) talks only to it; it authenticates, authorizes, runs admission, validates and stores.
• etcd - the key-value database holding every object; only the API server touches it. Run 3 or 5 members for quorum.
• kube-scheduler - picks a node for each new pod (filter, score, bind) and writes only spec.nodeName.
• kube-controller-manager - the controllers: loops that make reality match the spec (Deployment -> ReplicaSets -> Pods, garbage collection...).

Every node:

• kubelet - a systemd service; starts the containers of pods bound to its node through CRI (containerd), runs probes, reports status.
• CNI plugin (Calico) - gives each pod its IP.
• kube-proxy - iptables rules that make Service addresses reach pods.

Add-ons like CoreDNS run on the cluster as ordinary workloads. The design is the reconciliation loop: you declare, controllers converge.

Also asked: What is the difference between a Pod and a Deployment? · What happens when you run kubectl apply -f deployment.yaml? · How do you troubleshoot a pod that is not running?

What is a kubeconfig file, and how does kubectl find it? Junior

A kubeconfig is the YAML file that tells kubectl where the cluster is and who you are. It has three lists and a pointer:

• clusters - API server URL plus the CA certificate to verify it
• users - credentials, e.g. a client certificate (client-key-data is a private key: treat the file like an SSH key, chmod 600, never commit it)
• contexts - named pairs "this cluster, as this user", optionally with a default namespace
• current-context - the one in use

kubectl looks at --kubeconfig=PATH, then the KUBECONFIG variable (several files merged), then ~/.kube/config. With none it falls back to localhost:8080 - so "The connection to the server localhost:8080 was refused" means no kubeconfig, not a down cluster (and sudo kubectl looks in /root).

kubectl config get-contexts, current-context and use-context read and switch; check the context before anything destructive.

Also asked: kubectl says "The connection to the server localhost:8080 was refused". What does that mean? · How do you avoid running a command against the wrong cluster? · What does kubectl version tell you, and what is version skew?

Learn it: 15.1 The lab cluster, kubectl and kubeconfig

How do you create Kubernetes manifests quickly without writing YAML from scratch? Junior

Let kubectl's generators write them, and print instead of create:

export do="--dry-run=client -o yaml"
k run nginx --image=nginx:1.27 $do > pod.yaml
k create deployment web --image=nginx:1.27 --replicas=3 $do > web.yaml
k create job pi --image=busybox:1.36 $do -- sh -c 'echo hi' > job.yaml

--dry-run=client builds the object without sending it; -o yaml prints it. Edit the file, then k apply -f web.yaml.

When a field is not covered by a flag, look it up instead of guessing: k explain deployment.spec.strategy (add --recursive for the whole subtree), and k api-resources for the short name (deploy, cm, sts), the apiVersion (apps/v1) and whether it is namespaced.

Every command has the same shape - kubectl <verb> <type> [<name>] [flags] - and the speed kit (alias k=kubectl, Tab completion for k) makes the rest fast.

Also asked: What does kubectl api-resources tell you? · How do you look up the fields of a Kubernetes object from the command line? · What are the four top-level parts of every manifest?

Learn it: 15.3 The speed kit, and the shape of every kubectl command

What does each control plane component do? Junior

Four components, one job each, and one rule: only the API server reads and writes etcd.

• kube-apiserver - the front door. Every request goes through authentication, authorization, admission (plugins that may change or reject the object), validation against the schema, then is stored in etcd with a new resourceVersion, and every watcher is told.
• etcd - the key-value database where every object lives (/registry/pods/...). Members agree with Raft; a write needs a quorum (majority), which is why it runs 3 or 5. Lose quorum and running containers keep going, but nothing can change.
• kube-scheduler - for each pod without a node: filter (resources by requests, taints, selectors), score, then write a Binding - it only sets nodeName. FailedScheduling events are its explanation.
• kube-controller-manager - dozens of controllers, each watching one kind and making reality match.

On kubeadm they are static pods from /etc/kubernetes/manifests/ on cp-1; change them by editing those files.

Also asked: Why does a production etcd cluster have 3 or 5 members? · What is a static pod? · A pod stays Pending. Which component do you suspect first, and why?

Learn it: 15.5 The control plane: who does what

What does the kubelet do? Junior

The kubelet is the node agent - a systemd service on every node, not a pod - and the only component that actually starts containers. It watches the API server for pods bound to its node, and for each one:

1. asks the runtime through CRI (containerd) for a pod sandbox; the CNI plugin gives it an IP
2. mounts the volumes, pulls the images
3. runs the init containers in order, then starts the app containers
4. runs the probes and restarts exited containers with backoff
5. writes pod status and the node's heartbeat back to the API server

It also runs the static pods from /etc/kubernetes/manifests.

When it stops, the node goes NotReady (describe node conditions go Unknown: "Kubelet stopped posting node status"). Debug it like any service: systemctl status kubelet, journalctl -u kubelet. On the node, sudo crictl ps lists containers - there is no docker.

Also asked: A node shows NotReady. How do you troubleshoot it? · What are CRI, CNI and kube-proxy? · What is the difference between allocatable and capacity on a node?

Learn it: 15.7 The node: kubelet, CRI, CNI, kube-proxy, CoreDNS

What is a controller in Kubernetes, and what is the reconciliation loop? Junior

Every object has a spec (what you want, which you write) and a status (what is, which the cluster writes). A controller is a loop that watches one kind of object and keeps acting until status matches spec:

loop: desired = spec; actual = observe; if different, act; repeat

The ReplicaSet controller keeps N pods with its labels, the Deployment controller keeps the ReplicaSets matching the template, the scheduler gives pods a node, the kubelet keeps containers running.

Consequences:

• You declare, you do not command - kubectl scale only changes spec.replicas; the loops do the rest.
• Level-triggered - controllers compare states, not events, so a missed event or a restart heals on the next pass.
• Fighting the loop loses - delete a Deployment's pod and a new one (with a new name) appears. If a change keeps undoing itself, find the owner: metadata.ownerReferences (Pod -> ReplicaSet -> Deployment). Deleting an owner garbage-collects its dependents.

Also asked: What are ownerReferences used for? · What does kubectl delete --cascade=orphan do? · Why does a pod you delete come back with a different name?

Learn it: 15.9 The reconciliation loop

What happens when you run kubectl apply -f deployment.yaml? Junior

1. kubectl sends the object to the API server (POST if new, PATCH if it exists) and keeps the file in the last-applied-configuration annotation.
2. kube-apiserver authenticates you, authorizes, runs admission plugins, validates the fields strictly, and writes it to etcd.
3. The deployment controller sees a Deployment without a matching ReplicaSet and creates one (ScalingReplicaSet event).
4. The replicaset controller creates the Pod objects, with no node yet - Pending (SuccessfulCreate).
5. The scheduler filters and scores nodes and binds each pod (Scheduled).
6. The kubelet on that node creates the sandbox through CRI, CNI gives it an IP, it pulls the image (Pulling, Pulled).
7. containerd creates and starts the container (Created, Started).
8. The kubelet reports it Ready; a Service's endpoints then include it and kube-proxy routes traffic to it.

Each hop leaves an Event, so when nothing runs, kubectl describe and the last event tell you which hop stopped.

Also asked: How do you find out why a pod is not running? · What do admission controllers do? · Why does kubectl get events need --sort-by?

Learn it: 15.11 From kubectl apply to a running container

A pod shows CrashLoopBackOff. How do you troubleshoot it? Junior

CrashLoopBackOff is not the cause: it means the container keeps exiting and the kubelet is waiting before the next restart (10s, 20s, 40s ... capped at 5 minutes), as restartPolicy: Always demands.

1. k get pod POD - RESTARTS and how long ago.
2. k logs POD --previous - the log of the run that crashed (the current one is often empty).
3. k describe pod POD - Last State: Terminated with the exit code and reason, plus the events.
4. Read the exit code: 1/2 the app's own error (config, a dependency) - the logs say which; 126/127 not executable / command not found; 137 killed by SIGKILL - OOMKilled means its memory limit; 0 the process simply finished, and Always restarts even that (a container needs a long-running foreground process).
5. Fix the cause, not the pod: the image, the command, the config or the limit in the Deployment.

Also asked: What is a Pod, and why is it the smallest unit rather than a container? · What happens step by step when a pod is deleted? · What is the difference between a pod's phase and the STATUS column?

Learn it: 15.14 Pods: the unit, its lifecycle, and how to read its status

What is the difference between a Deployment and a ReplicaSet? Junior

A ReplicaSet does one thing: keep exactly N pods matching its selector alive. A Deployment manages ReplicaSets - one per version of its pod template - and adds updates and history.

• Changing the pod template (image, env) starts a rollout: a new ReplicaSet (new pod-template-hash) scales up while the old one scales down, within maxSurge (pods above replicas, rounded up) and maxUnavailable (pods below, rounded down); default 25%/25%. maxUnavailable: 0 means no capacity loss but needs room for the surge pod.
• Changing replicas is just a scale - same ReplicaSet.
• Old ReplicaSets stay at 0 (up to revisionHistoryLimit), which is what k rollout undo scales back up.

You never create ReplicaSets yourself. Day to day: k set image, k rollout status --timeout (exits non-zero on failure), k rollout history, k rollout undo --to-revision=N. A stuck rollout hits progressDeadlineSeconds, but Kubernetes does not roll back on its own.

Also asked: How do you configure a Deployment for zero-downtime updates? · A rollout is stuck. How do you investigate and recover? · What is the difference between the RollingUpdate and Recreate strategies?

Learn it: 15.16 Deployments and ReplicaSets: rolling updates, the maths, history and rollback

When would you use a StatefulSet instead of a Deployment? Junior

When the pods are not interchangeable - any of:

• Stable identity - pods are named <name>-0, -1, -2 and keep the name when recreated. With a headless Service (clusterIP: None) each gets its own DNS name, db-0.db.<namespace>.svc.cluster.local, so members can find each other.
• Own persistent storage - volumeClaimTemplates creates one PVC per pod (data-db-0...) that always follows that pod, and survives deleting the StatefulSet.
• Order - with OrderedReady, db-1 starts only after db-0 is Ready; scale-down removes the highest ordinal first. Updates go from the highest ordinal down, and partition lets you update one member first.

So: databases, message brokers, consensus systems. Everything stateless - web servers, APIs - is a Deployment.

And ask whether the database belongs in the cluster at all: a managed database is often the better answer.

Also asked: What is a headless Service, and why does a StatefulSet need one? · What happens to a StatefulSet's PVCs when you delete it? · Why should you not force-delete a StatefulSet pod on a node you cannot reach?

Learn it: 15.19 StatefulSets: stable identity, ordered start, headless Services

What is a DaemonSet, and when would you use one? Junior

A DaemonSet runs one pod on every node (or every node matching a nodeSelector), adds one when a node joins and removes it when a node leaves. There is no replicas: the node count decides.

Use it for things every machine needs: log shippers reading /var/log/pods, node metrics agents, the network plugin (calico-node), kube-proxy.

The detail to know: it respects taints. On kubeadm the control plane has node-role.kubernetes.io/control-plane:NoSchedule, so a plain DaemonSet shows DESIRED 2 on a three-node cluster. To include the control plane, add a matching toleration to the pod template (kube-proxy tolerates everything with operator: Exists).

Updates roll node by node (maxUnavailable: 1), and rollout status/history/undo work as for Deployments. It is also why kubectl drain needs --ignore-daemonsets.

Also asked: Why does a DaemonSet not run on the control plane node by default? · How do you run a DaemonSet only on some nodes? · What is the difference between a taint and a toleration?

Learn it: 15.22 DaemonSets: one pod per node, and why the control plane is left out

What is the difference between a Job and a Deployment, and how do CronJobs work? Junior

A Deployment keeps pods running for ever (restartPolicy: Always); a finished process gets restarted. A Job runs pods until they succeed, then stops - for migrations, reports, backups. Its restartPolicy must be OnFailure or Never.

Job fields: completions (successes needed), parallelism (at the same time), backoffLimit (failed retries before giving up), activeDeadlineSeconds (a hard time limit), ttlSecondsAfterFinished (clean-up). With Never each retry is a new pod with its own logs; with OnFailure the kubelet restarts the container in place.

A CronJob creates a Job on a schedule, a 5-field cron expression in UTC unless timeZone is set. concurrencyPolicy decides what happens if the last run is still going: Allow, Forbid (right for backups) or Replace.

Operating it: k create job manual-run --from=cronjob/tick runs it now; suspend: true pauses it; k logs job/NAME reads the logs.

Also asked: How would you design a nightly backup as a CronJob? · What does backoffLimit do? · What is the difference between restartPolicy Never and OnFailure in a Job?

Learn it: 15.24 Jobs and CronJobs: run to completion, on a schedule

What are labels and selectors used for in Kubernetes? Junior

A label is a key/value pair in metadata.labels (app: web); a selector is a query over labels (app=web). They are the wiring of the cluster: a ReplicaSet finds its pods by selector, a DaemonSet too, and so do Services. There is no other link.

• On the command line: k get pods -l app=web,tier=frontend, set-based -l 'tier in (frontend,backend)', -L tier for a column, k label pod x env=prod --overwrite.
• In manifests: matchLabels and matchExpressions.
• Get one letter wrong and things silently stop being connected.

Useful consequences: relabelling a pod takes it out of its ReplicaSet (it gets replaced, and you can debug the old one in peace); a stray pod with matching labels can be adopted - a Deployment's ReplicaSets add pod-template-hash to their selector to prevent that.

Annotations are metadata for tools and humans, not selectable. Namespaces scope names, not network traffic.

Also asked: What is the difference between a label and an annotation? · What is a namespace, and what does it not isolate? · How do you change the default namespace of your kubectl context?

Learn it: 15.26 Labels, selectors, annotations, namespaces

What is a ConfigMap, and how can a pod use it? Junior

A ConfigMap is a named set of key/value pairs - each value a string or a whole file - kept outside the image, so the same image runs in dev and prod: k create configmap app-config --from-literal=LOG_LEVEL=info --from-file=default.conf.

A pod uses it as:

• environment variables - one key with valueFrom.configMapKeyRef, or all keys with envFrom;
• files - a configMap volume with volumeMounts; subPath for a single file.

The critical difference: env vars are read once, at container start - a change never reaches a running container. Mounted files update in place within about a minute (an atomic swap of the ..data symlink) - but subPath mounts never update, and the app must re-read the file.

Changing a ConfigMap never restarts pods. To roll it out: k rollout restart deploy/NAME, a checksum annotation in the pod template, or a versioned ConfigMap name. A missing ConfigMap leaves the pod in CreateContainerConfigError or stuck mounting.

Also asked: How do you roll out a configuration change safely? · Why does a ConfigMap mounted with subPath not update? · What does immutable: true do on a ConfigMap?

Learn it: 15.29 ConfigMaps: env vs volumes, and what happens when you change one

Are Kubernetes Secrets secure? Junior

Not by themselves. A Secret is a ConfigMap with better manners: values are base64-encoded, which is an encoding, not encryption - k get secret db-creds -o jsonpath='{.data.password}' | base64 -d prints the password. They are kept on tmpfs on the node, and describe hides values.

Where the protection really comes from:

• Who may read them: anyone who can get or list Secrets in a namespace - or create a pod there - can read them. That is a permissions question.
• Encryption at rest: without an EncryptionConfiguration (ideally backed by a KMS), etcd and its backups hold them as plain base64.
• Not in git: a manifest with stringData is a plaintext password.
• Files over env vars: env vars leak into crash dumps, /proc/<pid>/environ and child processes.

The careful pattern keeps them out of Kubernetes objects: a driver mounts secrets from an external store (a Key Vault) into the pod.

Also asked: Why is base64 not protection? · What is the classic bug when you base64-encode a password with echo? · What is the Downward API?

Learn it: 15.33 Secrets, base64, and the Downward API

What is the difference between an init container and a sidecar? Junior

• An init container (spec.initContainers) runs before the app containers, one at a time, in order, each to successful completion. If it fails, nothing after it starts; the pod shows Init:N/M or Init:Error. Use it for waiting on a dependency, a migration, rendering config into a shared emptyDir, fixing volume permissions.
• A sidecar runs beside the app for the pod's whole life, sharing its network (localhost) and volumes: a log shipper, a proxy.

Since 1.28 a native sidecar is an init container with restartPolicy: Always: it starts before the app, the kubelet does not wait for it to exit, it is restarted if it dies, it is stopped after the app, and a Job with one still completes. That fixed the two old problems of sidecars as plain containers: startup order and Jobs that never finished.

Debug either with k logs POD -c NAME - without -c you get the app container.

Also asked: A pod is stuck at Init:0/2. How do you debug it? · Why did Jobs with sidecar containers never complete before native sidecars? · What is an emptyDir volume used for?

Learn it: 15.35 Init containers and sidecars

How do you extract specific fields from Kubernetes objects with kubectl? Junior

Find the path once with -o yaml (metadata / spec / status), then ask for just that:

• jsonpath - k get pod web -o jsonpath='{.spec.nodeName}'. On a list, paths start with .items: '{.items[*].metadata.name}'. One line per item with {range .items[*]}...{"\n"}{end}. Filters: '{.items[*].status.conditions[?(@.type=="Ready")].status}'. Escape dots in keys (deployment\.kubernetes\.io/revision) and always single-quote.
• custom-columns - -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName for your own table.
• --sort-by and --field-selector (status.phase!=Running, spec.nodeName=worker-1) - filtered on the server.
• -o name - type/name per line, to feed into other commands.
• -o json | jq - when you need to count, sum or group, which jsonpath cannot.

Also asked: How would you list every container image running in the cluster, with how many pods use it? · How would you script a check that all nodes are Ready? · What is the difference between jsonpath and custom-columns output?

Learn it: 15.38 Reading the API fast: -o yaml, jsonpath, custom-columns, sort and filter

What is the difference between kubectl create, apply, patch and replace? Mid

• create -f - imperative: create, and fail if it exists.
• apply -f - declarative: create or update to match the file; idempotent, the production way (from git). It does a three-way merge of last-applied (the annotation), live and the new file: fields in the file are set; fields in last-applied but gone from the file are removed; fields only in live are left alone - so a replicas: in the manifest fights an autoscaler on every apply, and leaving it out does not.
• diff -f - what apply would change (exit 1 = differences): the terraform plan of kubectl.
• patch - change only the fields you send: strategic merge (merges lists by key, like containers by name), --type=merge (replaces lists wholesale), --type=json (exact add/remove/replace operations).
• replace -f - a PUT of the whole object; anything not in the file is lost. replace --force deletes and recreates, the only way to change immutable fields.
• edit - opens the live object; quick, but invisible to git.

Also asked: What is the difference between imperative and declarative management in Kubernetes? · Why can kubectl apply fight an autoscaler? · What does kubectl diff show, and what does its exit code mean?

Learn it: 15.40 Imperative vs declarative: create, apply, diff, patch, replace, edit

A pod is not working. What are your first steps? Junior

The same order every time, reading before acting:

1. k get pods -o wide - STATUS, READY, RESTARTS and the node. Pending = scheduling; ContainerCreating = sandbox or mounts; ImagePullBackOff = image; CrashLoopBackOff = the app exits; Running but 0/1 = readiness.
2. k describe pod POD - the Events at the bottom first, then Last State (exit code, reason).
3. k logs POD and 4. k logs POD --previous - the crashed run usually has the error.
4. k get events --sort-by=.lastTimestamp - what happened around it (they expire after an hour).
5. k exec -it POD -- sh - check assumptions: env, files, DNS, the port.
6. One level up - k get deploy,rs and describe deploy: a paused or stuck rollout, a ReplicaSet that cannot create pods.

Do not restart first: it destroys the evidence (previous logs, events).

Also asked: Map the common pod statuses to their likely causes. · What does FailedScheduling tell you? · How do you debug a container image that has no shell?

Learn it: 15.42 The debugging order