Lesson 15.11 · Kubernetes: Architecture & Workloads · 21 min read

From kubectl apply to a running container

Q: Why doesn't kubectl get events show things in time order?

By default, kubectl get events isn't sorted by time. Always add --sort-by=.lastTimestamp, or use the newer kubectl events, which sorts and can filter with --for pod/NAME or --types=Warning. Events are also deduplicated, shown as (x12 over 10m).

Q: Why are there no events for a pod that has been broken since yesterday?

Events expire, by default after one hour (the apiserver's --event-ttl). For older problems you have the current state in describe (Last State, exit codes, conditions), the container logs including --previous, and whatever your logging and monitoring stack retained. That's one reason to ship events to a log store in production.

Q: What happens when you run kubectl apply -f deployment.yaml?

1. kubectl sends the object to the API server (POST if new, PATCH if it exists) and keeps the file in the last-applied-configuration annotation. 2. kube-apiserver authenticates you, authorizes, runs admission plugins, validates the fields strictly, and writes it to etcd. 3. The deployment controller sees a Deployment without a matching ReplicaSet and creates one (ScalingReplicaSet event). 4. The replicaset controller creates the Pod objects, with no node yet - Pending (SuccessfulCreate). 5. The scheduler filters and scores nodes and binds each pod (Scheduled). 6. The kubelet on that node creates the sandbox through CRI, CNI gives it an IP, it pulls the image (Pulling, Pulled). 7. containerd creates and starts the container (Created, Started). 8. The kubelet reports it Ready; a Service's endpoints then include it and kube-proxy routes traffic to it. Each hop leaves an Event, so when nothing runs, kubectl describe and the last event tell you which hop stopped.

In plain words

When you post a letter, it goes through several hands: the postbox, the sorting office, the van, the local postman, your friend's letterbox. Each one stamps the envelope. If the letter never arrives, you look at the last stamp to know where it got stuck.

kubectl apply travels the same way: kubectl to the apiserver (authenticate, authorise, admit, validate, store in etcd), then the deployment controller creates a ReplicaSet, the replicaset controller creates Pod objects, the scheduler binds each to a node, the kubelet on that node creates the sandbox via CRI, CNI gives an IP, the image is pulled, containers start, probes pass, and endpoints plus kube-proxy make it reachable. Each hop leaves an Event, and the last event tells you where it stopped.

Why narrate the path

"A deploy succeeded but nothing is running" is one of the most common Kubernetes tickets. Between kubectl apply and a running container there are about eight hand-offs between different components; the fix is always "find the hop that stopped". This lesson walks every hop with the evidence each one leaves, so you can point at the one that failed. "What happens when you run kubectl apply?" is also a classic interview question - practise saying it out loud.

What you need to know already: the API server's checks (15.5), the scheduler and controllers (15.5), the kubelet, CRI and CNI (15.7), the loop and ownerReferences (15.9), docker pull (10.5).

The narration

kubectl apply -f web.yaml

kubectl apply -f web.yaml = "make the cluster match this file" (-f = from this file). Say the file describes a Deployment with 3 replicas.

1. kubectl reads the file and looks up the kind (Deployment, group apps/v1) in the list of types the API server advertised (the same list api-resources prints, 15.3). It sends the object to the API server: a PATCH if it exists, a POST if not. It also saves a copy of the file in an annotation (a note on the object) called kubectl.kubernetes.io/last-applied-configuration, so next time it can tell what you removed (15.40).

2. kube-apiserver checks who you are (the client certificate in your kubeconfig -> user kubernetes-admin), whether you may create Deployments, runs the admission plugins, validates the fields strictly, and writes the object to etcd (15.5). Evidence: the object now exists, with a uid, resourceVersion and creationTimestamp the server assigned:

$ k create deployment web --image=nginx:1.27 --replicas=3     # AlreadyExists is fine
deployment.apps/web created
$ k get deploy web -o jsonpath='{.metadata.uid} {.metadata.resourceVersion} {.metadata.creationTimestamp}{"\n"}'
6a0f3c51-2c1d-4a6e-9f4b-0d2e7b9c8a13 184210 2026-09-23T10:31:07Z

(uid = the object's unique id; resourceVersion = the write counter; creationTimestamp = when it was stored, in UTC.)

3. The deployment controller (inside kube-controller-manager) is watching Deployments. It sees one with no ReplicaSet for its pod template, creates web-<hash> and sets it to 3. It leaves an Event - a short, timestamped log line attached to the object it acted on:

Normal  ScalingReplicaSet  12s  deployment-controller  Scaled up replica set web-6b8d9c7f5d from 0 to 3

4. The replicaset controller sees a ReplicaSet wanting 3 pods and owning

It creates 3 Pod objects - with no nodeName. Nothing is running yet;

these are just records in etcd. A pod in this state shows as Pending.

Normal  SuccessfulCreate  12s  replicaset-controller  Created pod: web-6b8d9c7f5d-8xk2p

5. kube-scheduler watches for pods with an empty spec.nodeName. For each: filter the nodes, score the survivors, write a Binding (15.5) - this sets the pod's nodeName.

Normal  Scheduled  12s  default-scheduler  Successfully assigned default/web-6b8d9c7f5d-8xk2p to worker-2

6. The kubelet on worker-2 watches for pods bound to its node. It creates the pod sandbox through CRI (containerd), which calls the CNI plugin for an IP; then it mounts volumes and pulls the image:

Normal  Pulling  11s  kubelet  Pulling image "nginx:1.27"
Normal  Pulled   8s   kubelet  Successfully pulled image "nginx:1.27" in 2.576s (2.576s including waiting). Image size: 71617741 bytes.

7. containerd creates and starts the container; the kubelet reports it:

Normal  Created  8s  kubelet  Created container: nginx
Normal  Started  8s  kubelet  Started container nginx

8. The kubelet checks the container is healthy and writes the pod status

Ready=True once its readiness check passes (or immediately if it has none).

The pod is Running.

Two more hops turn "running" into "serving traffic". They need a Service - a stable address in front of the pods:

9. A controller sees a Ready pod matching a Service's labels and adds the pod's IP to that Service's list of backends (an EndpointSlice object).

10. kube-proxy on every node sees the list change and rewrites its iptables rules (15.7), so traffic to the Service address now reaches this pod.

Later (Ch 16): Services, EndpointSlices and kube-proxy in depth - hops 9 and 10.

Events: the audit trail of every hop

Every component above records Events against the object it acted on. They are the single most useful thing to read when something is wrong, because the last event before things stop tells you which hop failed. k describe shows an object's events at the bottom; $(k get pods -l app=web -o name | head -1) picks the first app=web pod:

$ k describe $(k get pods -l app=web -o name | head -1)
...
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  12s   default-scheduler  Successfully assigned default/web-6b8d9c7f5d-8xk2p to worker-2
  Normal  Pulling    11s   kubelet            Pulling image "nginx:1.27"
  Normal  Pulled     8s    kubelet            Successfully pulled image "nginx:1.27" in 2.576s (2.576s including waiting). Image size: 71617741 bytes.
  Normal  Created    8s    kubelet            Created container: nginx
  Normal  Started    8s    kubelet            Started container nginx

column	means
Type	`Normal` (informational) or `Warning` (something went wrong)
Reason	a one-word code: `Scheduled`, `Pulling`, `FailedScheduling`, `BackOff`...
Age	how long ago it happened
From	the component that wrote it: default-scheduler, then kubelet
Message	the human-readable detail

Read it as the path above. The Deployment and ReplicaSet events live on those objects - k describe deploy web and k describe rs - because events are attached to the object that was acted on.

All events in a namespace, oldest first:

$ k get events --sort-by=.lastTimestamp
LAST SEEN   TYPE     REASON              OBJECT                      MESSAGE
14s         Normal   ScalingReplicaSet   deployment/web              Scaled up replica set web-6b8d9c7f5d from 0 to 3
14s         Normal   SuccessfulCreate    replicaset/web-6b8d9c7f5d   Created pod: web-6b8d9c7f5d-8xk2p
14s         Normal   Scheduled           pod/web-6b8d9c7f5d-8xk2p    Successfully assigned default/web-6b8d9c7f5d-8xk2p to worker-2
13s         Normal   Pulling             pod/web-6b8d9c7f5d-8xk2p    Pulling image "nginx:1.27"
...

get events is not sorted by time by default - always add --sort-by=.lastTimestamp (sort by the lastTimestamp field). The newer kubectl events command sorts for you and can filter:

k events --for pod/web-6b8d9c7f5d-8xk2p     # only this object's events
k events --types=Warning -A                 # only warnings, all namespaces

Two properties to know:

Events are deduplicated: a repeated event shows once as 2m (x12 over 10m) - twelve times, last 2 minutes ago, first seen ten minutes ago.
Events expire after 1 hour. A pod broken since last night has no events explaining why - you get the current state and the logs, not the history.

What a failed hop looks like

The last event tells you where it stopped. You do not need every term in this table yet - each row gets its own incident in this chapter:

stopped at    last thing you see                          meaning
-----------   -----------------------------------------   ------------------------------
apiserver     kubectl prints an error                      validation/admission/permission
controller    Deployment exists, RS FailedCreate           e.g. missing ServiceAccount, quota
scheduler     pod Pending, FailedScheduling                no feasible node
scheduler     pod Pending, NO events at all                the scheduler is not running
kubelet       FailedMount                                   missing ConfigMap/Secret/PVC
kubelet       Failed ... ErrImagePull / ImagePullBackOff    image name, tag, registry auth
kubelet       Error: CreateContainerConfigError             env refers to a missing key
container     Started, then BackOff / CrashLoopBackOff      the app itself exits

FailedScheduling - the scheduler found no node that fits (15.5).
FailedMount - a file the pod needs (a ConfigMap, Secret or disk, taught later in this chapter) does not exist.
ErrImagePull / ImagePullBackOff - the image cannot be pulled: wrong name or tag, or the registry refused (10.54). The kubelet retries, waiting longer each time.
CreateContainerConfigError - the container's settings point at something missing (15.29).
CrashLoopBackOff - the container starts and then exits, again and again (15.14).

Every "pod not running" question starts with describe, and the answer is almost always in the last three events.

What you can now do:

narrate kubectl apply hop by hop, naming the component and its evidence
read an Events table and k get events --sort-by=.lastTimestamp
say which hop failed from the last event

Why it helps

"What happens when you run kubectl apply?" is one of the most common Kubernetes interview questions, and the plan asks you to be able to narrate it out loud. More practically, it's the map for every "my pod isn't running" ticket: an error from kubectl means the apiserver refused; FailedCreate on the ReplicaSet means a controller-level problem like quota; FailedScheduling means no feasible node; no events at all means the scheduler isn't running; FailedMount, ImagePullBackOff or CreateContainerConfigError mean the kubelet hop. And remembering that events expire after an hour explains why a pod broken since last night shows nothing useful in describe.

FAQ

Why doesn't kubectl get events show things in time order?

By default, kubectl get events isn't sorted by time. Always add --sort-by=.lastTimestamp, or use the newer kubectl events, which sorts and can filter with --for pod/NAME or --types=Warning. Events are also deduplicated, shown as (x12 over 10m).

Why are there no events for a pod that has been broken since yesterday?

Events expire, by default after one hour (the apiserver's --event-ttl). For older problems you have the current state in describe (Last State, exit codes, conditions), the container logs including --previous, and whatever your logging and monitoring stack retained. That's one reason to ship events to a log store in production.

Where do I see the Deployment and ReplicaSet steps?

Events attach to the object that was acted on. The pod's describe shows scheduler and kubelet events; the Deployment's describe shows ScalingReplicaSet; the ReplicaSet's shows SuccessfulCreate or FailedCreate. If a Deployment has no pods at all, look at describe rs, where quota or admission failures appear.

What does "Pending with no events" mean?

The pod object exists but the scheduler never touched it: no Scheduled, no FailedScheduling. Most likely the scheduler isn't running or can't reach the apiserver, or the pod names a custom scheduler that doesn't exist (schedulerName). Check the scheduler pod in kube-system, and on kubeadm the static pod on the control plane.

Is a pod that says Running serving traffic?

Not necessarily. Running means the containers started. It receives Service traffic only when it's Ready, meaning its readiness probe passes, and the endpoint controllers have added it to the EndpointSlice, and kube-proxy has updated routing. READY 0/1 with STATUS Running means probes are failing, and describe shows "Readiness probe failed".

In an interview Junior

What happens when you run kubectl apply -f deployment.yaml?

kubectl sends the object to the API server (POST if new, PATCH if it exists) and keeps the file in the last-applied-configuration annotation.
kube-apiserver authenticates you, authorizes, runs admission plugins, validates the fields strictly, and writes it to etcd.
The deployment controller sees a Deployment without a matching ReplicaSet and creates one (ScalingReplicaSet event).
The replicaset controller creates the Pod objects, with no node yet - Pending (SuccessfulCreate).
The scheduler filters and scores nodes and binds each pod (Scheduled).
The kubelet on that node creates the sandbox through CRI, CNI gives it an IP, it pulls the image (Pulling, Pulled).
containerd creates and starts the container (Created, Started).
The kubelet reports it Ready; a Service's endpoints then include it and kube-proxy routes traffic to it.

Each hop leaves an Event, so when nothing runs, kubectl describe and the last event tell you which hop stopped.

Also asked: How do you find out why a pod is not running? · What do admission controllers do? · Why does kubectl get events need --sort-by?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.