OnCallReady

Lesson 30.26 · AWS II: VPC, EC2, ELB & EKS · 13 min read

EKS: what AWS runs and what you run

In plain words

EKS is Kubernetes where AWS runs the brain and you run the body. The brain - the API servers and their database - lives in AWS's own building, spread over three districts, and you only see its front door (the endpoint). The body is your own rented machines (nodes) that join the brain, and every pod gets a real address on your private network, like a desk with its own phone number in your office.

EKS: what AWS runs and what you run

You have run Kubernetes clusters with kubeadm and used AKS. EKS is Amazon's managed Kubernetes, and the interesting part is not that it is Kubernetes - it is where the AWS parts plug in: the control plane you cannot see, the nodes that are EC2 instances in an Auto Scaling group, pods that get real VPC addresses, and IAM in front of the API server.

Need to know: AWS runs the control plane (API servers, etcd) in its own account across three AZs and gives you an HTTPS endpoint; you never see the masters. Your nodes are EC2 instances, usually a managed node group (an ASG of EKS-optimized AMIs that EKS creates and upgrades), or Fargate, or Auto Mode. The VPC CNI gives every pod an IP address from your subnets. kubectl authenticates with IAM: aws eks update-kubeconfig writes an exec plugin that runs aws eks get-token.

The cluster from the AWS side

$ aws eks describe-cluster --name try-eks --query 'cluster.{status:status,version:version,platform:platformVersion,endpoint:endpoint,public:resourcesVpcConfig.endpointPublicAccess,private:resourcesVpcConfig.endpointPrivateAccess,auth:accessConfig.authenticationMode,oidc:identity.oidc.issuer}' --output yaml
status: ACTIVE
version: '1.34'
platform: eks.12
endpoint: https://28271C9502380905913D771E936FB670.gr7.eu-central-1.eks.amazonaws.com
public: true
private: true
auth: API_AND_CONFIG_MAP
oidc: https://oidc.eks.eu-central-1.amazonaws.com/id/652F529758912AA10762385189F4A624
$ aws eks list-nodegroups --cluster-name try-eks
{
    "nodegroups": [
        "general"
    ]
}
$ aws eks describe-nodegroup --cluster-name try-eks --nodegroup-name general --query 'nodegroup.{status:status,instances:instanceTypes,ami:amiType,scaling:scalingConfig,asg:resources.autoScalingGroups[0].name}' --output yaml
status: ACTIVE
instances:
- t4g.medium
ami: AL2023_ARM_64_STANDARD
scaling:
  minSize: 1
  maxSize: 4
  desiredSize: 2
asg: eks-general-b2c36652-1d91-4f38-bb12-26839f452e68

The control plane costs $0.10 per hour (~$73 a month) per cluster, $0.60 per hour once its version leaves standard support (14 months after release) - the "extended support" bill is how many teams learn about EKS's version calendar.

The cluster from the Kubernetes side

$ aws eks update-kubeconfig --name try-eks --alias try-eks
Updated context try-eks in /home/learner/.kube/config
$ kubectl get nodes -o wide
NAME                                           STATUS   ROLES    AGE     VERSION               INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION                    CONTAINER-RUNTIME
ip-10-99-10-30.eu-central-1.compute.internal   Ready    <none>   7m26s   v1.34.1-eks-113cf36   10.99.10.30   <none>        Amazon Linux 2023.9.20260929   6.12.46-66.121.amzn2023.aarch64   containerd://2.1.4
ip-10-99-11-19.eu-central-1.compute.internal   Ready    <none>   7m26s   v1.34.1-eks-113cf36   10.99.11.19   <none>        Amazon Linux 2023.9.20260929   6.12.46-66.121.amzn2023.aarch64   containerd://2.1.4
$ kubectl get pods -n kube-system -o wide
NAME                       READY   STATUS    RESTARTS   AGE    IP            NODE                                           NOMINATED NODE   READINESS GATES
aws-node-k7xpp             1/1     Running   0          4m1s   10.99.10.30   ip-10-99-10-30.eu-central-1.compute.internal   <none>           <none>
aws-node-lmdwp             1/1     Running   0          4m1s   10.99.11.19   ip-10-99-11-19.eu-central-1.compute.internal   <none>           <none>
coredns-5ptprgn2bx-ck7zp   1/1     Running   0          25m    10.99.10.81   ip-10-99-10-30.eu-central-1.compute.internal   <none>           <none>
coredns-5ptprgn2bx-tp5ng   1/1     Running   0          25m    10.99.11.81   ip-10-99-11-19.eu-central-1.compute.internal   <none>           <none>
kube-proxy-bk9st           1/1     Running   0          4m1s   10.99.10.30   ip-10-99-10-30.eu-central-1.compute.internal   <none>           <none>
kube-proxy-hmm2c           1/1     Running   0          4m1s   10.99.11.19   ip-10-99-11-19.eu-central-1.compute.internal   <none>           <none>

Nodes are named after their private DNS names (ip-10-99-10-x.eu-central-1.compute.internal) and there are no control-plane nodes: kubectl get nodes only ever shows workers. In kube-system the add-ons EKS installed:

Add-onWhat it does
aws-node (vpc-cni)DaemonSet: gives pods VPC IPs (secondary IPs / prefixes on the node's ENIs)
kube-proxyDaemonSet: Service routing (iptables/IPVS) as in any cluster
corednsDeployment: cluster DNS
eks-pod-identity-agentDaemonSet (when added): credentials for pods (the next lesson)

Add-ons are versioned EKS objects (aws eks list-addons, describe-addon-versions); upgrading the cluster means upgrading them too.

Pods get VPC addresses

With the VPC CNI there is no overlay network: a pod's IP is a real address in a private subnet, reachable from anything in the VPC (security groups permitting). Two consequences:

  1. Subnet size matters: every pod uses a VPC IP. A /24 per AZ fills up long before the nodes do - EKS node subnets are usually /19 or /20, or use a secondary CIDR (100.64.0.0/10) for pods.
  2. Max pods per node depends on the instance type's ENIs and IPs per ENI: a t4g.medium has 3 ENIs x 6 IPs -> 17 pods (35 or 110 with prefix delegation). Pods stuck Pending with "Too many pods" on small nodes is this limit, not CPU.
$ kubectl get nodes -o custom-columns=NAME:.metadata.name,PODS:.status.allocatable.pods,TYPE:.metadata.labels.node\\.kubernetes\\.io/instance-type
NAME                                           PODS   TYPE
ip-10-99-10-30.eu-central-1.compute.internal   17     t4g.medium
ip-10-99-11-19.eu-central-1.compute.internal   17     t4g.medium
$ kubectl get pods -A -o wide | head -6
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE    IP            NODE                                           NOMINATED NODE   READINESS GATES
kube-system   aws-node-k7xpp             1/1     Running   0          4m1s   10.99.10.30   ip-10-99-10-30.eu-central-1.compute.internal   <none>           <none>
kube-system   aws-node-lmdwp             1/1     Running   0          4m1s   10.99.11.19   ip-10-99-11-19.eu-central-1.compute.internal   <none>           <none>
kube-system   coredns-5ptprgn2bx-ck7zp   1/1     Running   0          25m    10.99.10.81   ip-10-99-10-30.eu-central-1.compute.internal   <none>           <none>
kube-system   coredns-5ptprgn2bx-tp5ng   1/1     Running   0          25m    10.99.11.81   ip-10-99-11-19.eu-central-1.compute.internal   <none>           <none>
kube-system   kube-proxy-bk9st           1/1     Running   0          4m1s   10.99.10.30   ip-10-99-10-30.eu-central-1.compute.internal   <none>           <none>

kubectl and IAM

Look at what update-kubeconfig wrote:

$ kubectl config view --minify -o jsonpath='{.users[0].user.exec}' ; echo
{"apiVersion":"client.authentication.k8s.io/v1beta1","args":["--region","eu-central-1","eks","get-token","--cluster-name","try-eks","--output","json"],"command":"aws"}
$ aws eks get-token --cluster-name try-eks --query 'status.expirationTimestamp' --output text
2026-09-22T20:14:04Z
$ kubectl auth whoami
ATTRIBUTE                                              VALUE
Username                                               arn:aws:iam::111122223333:user/learner
UID                                                    aws-iam-authenticator:111122223333:AIDAEVXTZTVCSMBGJ2WDW
Groups                                                 [system:authenticated]
Extra: accessKeyId                                     [AKIAQ3EGPLAB4LEARNER]
Extra: arn                                             [arn:aws:iam::111122223333:user/learner]
Extra: canonicalArn                                    [arn:aws:iam::111122223333:user/learner]
Extra: principalId                                     [AIDAEVXTZTVCSMBGJ2WDW]
Extra: sessionName                                     []
Extra: sigs.k8s.io/aws-iam-authenticator/principalId   [AIDAEVXTZTVCSMBGJ2WDW]

Every kubectl request runs aws eks get-token, which presigns an STS GetCallerIdentity request with your current AWS credentials (profile, env vars, role) and sends it as a bearer token (valid 15 minutes). The API server asks STS who signed it - so whatever identity your CLI has, that is who you are in the cluster. What that identity may do there is decided separately (next lesson).

Versions and upgrades

EKS supports a Kubernetes version for 14 months of standard support plus 12 of extended support. Upgrades go one minor version at a time: control plane first (update-cluster-version), then the add-ons, then the node groups (update-nodegroup-version, which drains and replaces nodes like an instance refresh). Check deprecated APIs before each step - the cluster upgrade does not convert your manifests.

In an interview: "What does AWS manage in EKS and what do you manage?" - AWS runs and patches the control plane (API servers, etcd) across three AZs and provides managed add-ons; you own the nodes (or choose Fargate/Auto Mode), the networking (VPC, subnets, the CNI's IP budget), IAM access, the add-on versions, the workloads, and the upgrade schedule.

What you can do now

Why it helps

EKS is the most common way companies run Kubernetes on AWS, and the problems on call are at the AWS seams, not in Kubernetes itself: pods stuck Pending because a subnet ran out of IPs or a node hit its pod limit, kubectl refusing access, an upgrade forced by the support calendar. Knowing what AWS runs, what you run and how kubectl authenticates makes those seams visible.

Commands in this lesson

aws kubectl

FAQ

Why do I not see any control-plane nodes?

AWS runs the API servers and etcd in its own account, across three AZs, and only gives you an endpoint. kubectl get nodes lists your worker nodes only. You cannot SSH into the control plane or read etcd; you see its effects through the API and its logs if you enable them.

Why do pods get IP addresses from my subnets?

The VPC CNI plugin gives each pod a secondary IP address of the node's network interfaces, so pods are real members of the VPC, reachable from anything allowed by security groups. That makes load balancers able to target pods directly, but it uses many addresses: plan large subnets or a secondary CIDR for pods.

Why are my pods Pending with "Too many pods"?

The node reached its maximum pods, which with the VPC CNI depends on the instance type's network interfaces and addresses per interface: a t4g.medium allows 17. Use larger instances, enable prefix delegation (more addresses per interface), or add nodes. It is not about CPU or memory.

How does kubectl log in to EKS?

update-kubeconfig writes an exec plugin that runs aws eks get-token with your current AWS credentials. The token is a presigned STS GetCallerIdentity request; the API server asks STS who signed it. Whatever identity your CLI has is who you are in the cluster, and access entries decide what you may do.

How often do I need to upgrade?

Each Kubernetes version gets 14 months of standard support and 12 of extended support, which costs six times more per hour. Upgrades go one minor version at a time: the control plane first, then the add-ons, then the node groups, after checking for removed APIs in your manifests.

In an interview Mid

What does AWS manage in EKS and what do you manage?

AWS runs the control plane - API servers and etcd across three AZs - patches it and offers managed add-ons (vpc-cni, coredns, kube-proxy, eks-pod-identity-agent). You own the nodes (a managed node group is an Auto Scaling group of EKS-optimized AMIs; or Fargate/Auto Mode), the VPC and the IP budget the VPC CNI consumes, who may use the cluster (access entries), pod identities, add-on versions, upgrades before standard support ends, and the workloads. kubectl authenticates with IAM through aws eks get-token.

Also asked: Why does the VPC CNI matter for subnet sizing? · How do you upgrade an EKS cluster? · What is the difference between managed node groups and Fargate?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.