Lesson 30.33 · AWS II: VPC, EC2, ELB & EKS · 12 min read

The AWS Load Balancer Controller: Ingress and Service to ALB and NLB

Q: How do you expose a service on EKS with an AWS load balancer?

Install the AWS Load Balancer Controller with an IAM role (Pod Identity or IRSA), tag the subnets kubernetes.io/role/elb (public) or internal-elb (private), then create an Ingress with ingressClassName: alb and annotations for the scheme and target-type: ip - the controller builds an ALB whose targets are the pod IPs. For TCP, a Service of type LoadBalancer with the NLB annotations. kubectl describe ingress shows the controller's events when it cannot build it.

In plain words

The AWS Load Balancer Controller is a helper living in the cluster that reads your Kubernetes requests for a front door (an Ingress or a LoadBalancer Service) and builds a real AWS receptionist desk for each one - an ALB or NLB - wired straight to your pods. It needs its own badge to order those desks from AWS, and it looks for signs on your subnets to know where to put them.

The AWS Load Balancer Controller: Ingress and Service to ALB and NLB

On a kubeadm cluster an Ingress needs an ingress controller running inside the cluster (you used ingress-nginx and Gateway API in the mesh chapter). On EKS the usual answer is different: the AWS Load Balancer Controller watches Ingresses and Services and creates real AWS load balancers for them - ALBs and NLBs you can see with aws elbv2, health-checking your pods directly.

Need to know: the controller (v3.6, a Deployment in kube-system, installed with Helm) needs IAM permissions for EC2 and ELB (its own role, via Pod Identity or IRSA). An Ingress with ingressClassName: alb becomes an ALB; a Service of type LoadBalancer with service.beta.kubernetes.io/aws-load-balancer-type: external becomes an NLB. Annotations choose scheme (internet-facing or internal) and target-type (ip = pod IPs directly, instance = the nodes' NodePort). It finds subnets by tags: kubernetes.io/role/elb=1 (public) and kubernetes.io/role/internal-elb=1 (private).

The controller

$ kubectl -n kube-system get deploy aws-load-balancer-controller
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
aws-load-balancer-controller   2/2     2            2           5m1s
$ kubectl -n kube-system get deploy aws-load-balancer-controller -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}{.spec.template.spec.containers[0].args}{"\n"}'
public.ecr.aws/eks/aws-load-balancer-controller:v3.6.0
["--cluster-name=try-eks","--ingress-class=alb","--aws-region=eu-central-1","--aws-vpc-id=vpc-0fab6173e91281414"]
$ kubectl get ingressclass
NAME   CONTROLLER            PARAMETERS   AGE
alb    ingress.k8s.aws/alb   <none>       1s
$ aws eks list-pod-identity-associations --cluster-name try-eks --query 'associations[].[namespace,serviceAccount]' --output text
kube-system	aws-load-balancer-controller

Its permissions are a long IAM policy (the project's iam_policy.json) - create/modify/delete load balancers, target groups, listeners, rules and security groups, describe almost everything. Without them the controller still runs, and every Ingress gets events like FailedDeployModel ... AccessDenied.

It picks subnets by their tags, which is why the network for an EKS cluster is tagged on purpose:

$ aws ec2 describe-subnets --filters Name=tag:Name,Values='try-*' --query 'Subnets[].[Tags[?Key==`Name`]|[0].Value,Tags[?Key==`kubernetes.io/role/elb`]|[0].Value,Tags[?Key==`kubernetes.io/role/internal-elb`]|[0].Value]' --output text
try-public-a	1	None
try-public-b	1	None
try-private-a	None	1
try-private-b	None	1

The Ingress below points at the Service web; first, its ports:

Quick check

$ kubectl -n try get svc web -o jsonpath='{.spec.ports[0].port} -> {.spec.ports[0].targetPort}{"\n"}'

The Ingress will use target-type ip and point at this Service on port 80. Which port will the ALB's targets use?

80, the Service port
8080, the pods' targetPort
a NodePort (30000-32767)

Answer

8080, the pods' targetPort - IP targets are the pods themselves, on the container port the Service points at (its targetPort); only target-type instance goes through a NodePort on the nodes.

An Ingress becomes an ALB

$ cd ~/oncall-lab/labs/aws/try
$ printf 'apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n  name: web\n  namespace: try\n  annotations:\n    alb.ingress.kubernetes.io/scheme: internet-facing\n    alb.ingress.kubernetes.io/target-type: ip\n    alb.ingress.kubernetes.io/healthcheck-path: /healthz\nspec:\n  ingressClassName: alb\n  rules:\n  - http:\n      paths:\n      - path: /\n        pathType: Prefix\n        backend:\n          service:\n            name: web\n            port:\n              number: 80\n' > ingress.yaml
$ kubectl apply -f ingress.yaml
ingress.networking.k8s.io/web created
$ sleep 90
$ kubectl -n try get ingress web
NAME   CLASS   HOSTS   ADDRESS                                                            PORTS   AGE
web    alb     *       k8s-try-web-9e8727401b-1894709991.eu-central-1.elb.amazonaws.com   80      90s
$ kubectl -n try describe ingress web | tail -3
  Type    Reason                  Age   From     Message
  ----    ------                  ----  ----     -------
  Normal  SuccessfullyReconciled  86s   ingress  Successfully reconciled

ADDRESS is the ALB's DNS name. What the controller built, seen from AWS:

$ aws elbv2 describe-load-balancers --query 'LoadBalancers[?starts_with(LoadBalancerName, `k8s-`)].[LoadBalancerName,Scheme,State.Code]' --output text
k8s-try-web-9e8727401b	internet-facing	active
$ TG=$(aws elbv2 describe-target-groups --query 'TargetGroups[?TargetType==`ip`]|[0].TargetGroupArn' --output text)
$ aws elbv2 describe-target-health --target-group-arn $TG --query 'TargetHealthDescriptions[].[Target.Id,Target.Port,TargetHealth.State]' --output text
10.99.10.155	8080	healthy
10.99.11.155	8080	healthy
$ kubectl -n try get pods -o wide | awk '{print $1, $6}'
NAME IP
web-v68rvhdwbh-pm5df 10.99.10.155
web-v68rvhdwbh-px6ns 10.99.11.155

The targets are the pod IPs on 8080 (the Service's targetPort), not the nodes: with target-type: ip traffic goes ALB -> pod directly, skipping kube-proxy and the NodePort hop, and the ALB's health checks see each pod. Names are k8s-<namespace>-<ingress>-<hash>; tags on every object (elbv2.k8s.aws/cluster, ingress.k8s.aws/stack) let the controller find and garbage-collect them.

The annotations you will use

Annotation	Values	Effect
`alb.ingress.kubernetes.io/scheme`	`internet-facing` / `internal`	which subnets (elb / internal-elb tags) and whether it has public IPs
`alb.ingress.kubernetes.io/target-type`	`ip` / `instance`	pod IPs vs NodePorts
`alb.ingress.kubernetes.io/healthcheck-path`	`/healthz`	the target group's health check
`alb.ingress.kubernetes.io/listen-ports`	`[{"HTTPS":443}]`	listeners
`alb.ingress.kubernetes.io/certificate-arn`	an ACM certificate	TLS on the ALB
`alb.ingress.kubernetes.io/group.name`	`shop`	IngressGroup: several Ingresses share one ALB (rules merged) - one ALB per team instead of one per Ingress

An ALB costs ~$20 a month before traffic; without group.name every Ingress makes its own.

Services become NLBs

apiVersion: v1
kind: Service
metadata:
  name: api
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
spec:
  type: LoadBalancer
  ports:
  - port: 443
    targetPort: 8443

Without the external annotation the old in-tree cloud provider makes a Classic Load Balancer instead - one more reason to check what you got with aws elbv2 describe-load-balancers.

When it does not work

The controller reports through events on the Ingress/Service and its own log:

Event	Usual cause
`FailedBuildModel ... couldn't auto-discover subnets`	the subnets lack the `kubernetes.io/role/elb` tag
`FailedDeployModel ... AccessDenied`	the controller's role misses a permission (or Pod Identity is not set up)
`FailedBuildModel ... unable to resolve ... service`	backend Service name/port wrong
no events at all	wrong `ingressClassName`, or the controller is not running

In an interview: "How do you expose a service on EKS with an AWS load balancer?" - install the AWS Load Balancer Controller with an IAM role, tag the subnets, then an Ingress with class alb (or a Service of type LoadBalancer with the NLB annotations); with target-type ip the load balancer sends traffic straight to the pod IPs.

What you can do now

turn an Ingress into an ALB and a Service into an NLB, and read the result with the AWS CLI;
choose ip or instance targets and share an ALB with IngressGroups;
find out from events why an Ingress has no address.

Why it helps

Exposing services from EKS is daily platform work, and the failures are specific: an Ingress without an address because the subnets lack tags or the controller lacks permissions, a surprise bill from one ALB per Ingress, traffic hopping through NodePorts. Knowing how the controller maps Ingress and Service annotations to AWS objects, and how to read its events, makes those quick to fix.

Commands in this lesson

kubectl aws cd printf sleep

FAQ

Why does my Ingress have no address?

Look at its events with kubectl describe ingress. The usual causes: the controller is not running or the ingressClassName is wrong (no events at all), the subnets lack the kubernetes.io/role/elb or internal-elb tags (cannot auto-discover subnets), the controller's IAM role lacks a permission (AccessDenied), or the backend service name or port is wrong.

ip or instance targets?

With target-type ip the load balancer's targets are the pod IPs, so traffic goes straight to pods and health checks see each pod; it needs the VPC CNI's VPC addresses for pods. With instance the targets are the nodes on a NodePort, adding a kube-proxy hop. Prefer ip on EKS.

How do I avoid one ALB per Ingress?

Give Ingresses the same alb.ingress.kubernetes.io/group.name annotation: the controller merges their rules into one ALB (an IngressGroup). Each ALB has a fixed hourly price, so a cluster with dozens of Ingresses and no groups can cost hundreds a month in idle load balancers.

How do I get an NLB for a Service?

Use a Service of type LoadBalancer with the annotation service.beta.kubernetes.io/aws-load-balancer-type: external, plus nlb-target-type ip and the scheme annotation. Without the external annotation the legacy in-tree cloud provider creates a Classic Load Balancer instead, so check with aws elbv2 describe-load-balancers what you got.

Why does the controller need IAM permissions?

It creates and changes load balancers, target groups, listeners, rules and security groups through the AWS API, and describes subnets and instances. Its policy (the project's iam_policy.json) is attached to a role that its service account gets through Pod Identity or IRSA. Without it the controller runs but every Ingress reports AccessDenied events.

In an interview Mid

How do you expose a service on EKS with an AWS load balancer?

Install the AWS Load Balancer Controller with an IAM role (Pod Identity or IRSA), tag the subnets kubernetes.io/role/elb (public) or internal-elb (private), then create an Ingress with ingressClassName: alb and annotations for the scheme and target-type: ip - the controller builds an ALB whose targets are the pod IPs. For TCP, a Service of type LoadBalancer with the NLB annotations. kubectl describe ingress shows the controller's events when it cannot build it.

Also asked: What is an IngressGroup and why use one? · Why would an Ingress stay without an address? · What is the difference between target-type ip and instance?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.