EKS identity: access entries, IRSA, Pod Identity

EKS identity: access entries, IRSA, Pod Identity

Identity in EKS runs in two directions, and confusing them is the root of most EKS permission tickets: people (and CI) into the cluster - an IAM identity becomes a Kubernetes user - and pods out to AWS - a workload gets an IAM role. Each direction has an old way and a new way.

Need to know: Into the cluster: access entries (EKS API) map an IAM principal to a Kubernetes identity and attach access policies (AmazonEKSClusterAdminPolicy, AmazonEKSAdminPolicy, AmazonEKSEditPolicy, AmazonEKSViewPolicy), cluster-wide or per namespace; the old way is the aws-auth ConfigMap. Out to AWS: EKS Pod Identity (an agent add-on + an association namespace/service account -> role, trust pods.eks.amazonaws.com) or IRSA (the cluster's OIDC provider in IAM, a trust policy on :sub and :aud, a service-account annotation). Without either, a pod falls back to the node's role - or nothing, with hop limit 1.

Into the cluster: access entries

$ aws eks describe-cluster --name try-eks --query cluster.accessConfig
{
    "bootstrapClusterCreatorAdminPermissions": true,
    "authenticationMode": "API_AND_CONFIG_MAP"
}
$ aws eks list-access-entries --cluster-name try-eks
{
    "accessEntries": [
        "arn:aws:iam::111122223333:user/learner",
        "arn:aws:iam::111122223333:role/oncall-eks-node"
    ]
}
$ aws eks list-access-policies --query 'accessPolicies[].name' --output text
AmazonEKSAdminPolicy	AmazonEKSAdminViewPolicy	AmazonEKSClusterAdminPolicy	AmazonEKSEditPolicy	AmazonEKSViewPolicy

Two entries already exist: the node group's role (type EC2_LINUX - EKS creates it so kubelets can join) and whoever created the cluster, if bootstrap admin was on. Everything else you add:

aws eks create-access-entry --cluster-name try-eks --principal-arn arn:aws:iam::111122223333:role/shop-dev
aws eks associate-access-policy --cluster-name try-eks \
    --principal-arn arn:aws:iam::111122223333:role/shop-dev \
    --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSEditPolicy \
    --access-scope type=namespace,namespaces=shop

An entry can also carry Kubernetes group names (--kubernetes-groups) for your own RBAC RoleBindings - the access policies are optional. The three authentication modes:

Modes only move forward (CONFIG_MAP -> API_AND_CONFIG_MAP -> API). aws-auth is a YAML document in kube-system with mapRoles/mapUsers; one bad edit (an indentation error, a deleted node role) locks people out or makes every node NotReady, and only someone already inside can fix it. That is why access entries exist.

Out to AWS: what a pod gets by default

$ kubectl -n try exec deploy/reports -- aws sts get-caller-identity --query Arn --output text
arn:aws:sts::111122223333:assumed-role/oncall-eks-node/i-05c48c61b8679868a

The node role: the SDK in the pod found no credentials of its own, asked IMDS (one hop further than on the node, allowed by hop limit 2) and got the node's role - the same for every pod on that node. That is the thing to fix: either block it (hop limit 1, so pods get nothing) or give each workload its own role.

EKS Pod Identity

1. Install the eks-pod-identity-agent add-on (a DaemonSet listening on 169.254.170.23).
2. A role whose trust policy allows pods.eks.amazonaws.com to sts:AssumeRole and sts:TagSession (the session is tagged with cluster, namespace, service account - usable in permission policies as aws:PrincipalTag/kubernetes-namespace).
3. An association: cluster + namespace + service account -> role.
4. Pods created after that get AWS_CONTAINER_CREDENTIALS_FULL_URI and a token file injected by EKS's webhook; the SDK asks the agent, the agent asks EKS, EKS assumes the role.

$ aws eks create-addon --cluster-name try-eks --addon-name eks-pod-identity-agent --query addon.status --output text
CREATING
$ aws eks wait addon-active --cluster-name try-eks --addon-name eks-pod-identity-agent
$ aws eks create-pod-identity-association --cluster-name try-eks --namespace try --service-account reports --role-arn arn:aws:iam::111122223333:role/try-reports --query 'association.[associationId,namespace,serviceAccount]' --output text
a-9b6016a1eb5a4340	try	reports
$ kubectl -n try rollout restart deployment reports
deployment.apps/reports restarted
$ sleep 20
$ kubectl -n try exec deploy/reports -- env | grep AWS_ | sort
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token
AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials
AWS_DEFAULT_REGION=eu-central-1
AWS_REGION=eu-central-1
AWS_STS_REGIONAL_ENDPOINTS=regional
$ kubectl -n try exec deploy/reports -- aws sts get-caller-identity --query Arn --output text
arn:aws:sts::111122223333:assumed-role/try-reports/eks-try-eks-reports-9prbgslp8h-2f6w5-9a1c3b99-4941-4d33-a16b-242

The role's trust policy is generic (no cluster, no namespace in it), so the same role can be used by many clusters; the mapping lives in EKS.

IRSA (IAM roles for service accounts)

The older way, still everywhere, and the only way on self-managed Kubernetes on EC2:

1. The cluster has an OIDC issuer (identity.oidc.issuer in describe-cluster). Register it in IAM as an identity provider (aws iam create-open-id-connect-provider, audience sts.amazonaws.com).
2. A role whose trust allows sts:AssumeRoleWithWebIdentity from that provider, with conditions on the token: <issuer>:sub = system:serviceaccount:<namespace>:<serviceaccount> and <issuer>:aud = sts.amazonaws.com.
3. Annotate the service account: eks.amazonaws.com/role-arn: arn:aws:iam::...:role/....
4. New pods get a projected service-account token (audience sts.amazonaws.com) and AWS_ROLE_ARN / AWS_WEB_IDENTITY_TOKEN_FILE; the SDK calls AssumeRoleWithWebIdentity itself.

$ aws eks describe-cluster --name try-eks --query cluster.identity.oidc.issuer --output text
https://oidc.eks.eu-central-1.amazonaws.com/id/652F529758912AA10762385189F4A624

The failure everyone meets: "Not authorized to perform sts:AssumeRoleWithWebIdentity". It is always the trust policy side - the provider is not registered, the :sub names another namespace or service account (a workload moved), or :aud does not match. The role's permissions are not even looked at yet. The incident in this chapter is exactly that.

In an interview: "How does a pod on EKS get AWS permissions without access keys?" - with EKS Pod Identity (agent add-on, a role trusted by pods.eks.amazonaws.com, an association for the namespace and service account) or IRSA (the cluster's OIDC provider, a trust policy on the token's sub and aud, an annotated service account); either way the pod's SDK gets temporary credentials for its own role instead of the node's.

What you can do now

• give people and CI exactly the cluster access they need with access entries and access policies;
• tell what a pod's AWS identity is and why;
• set up Pod Identity, and debug an IRSA trust policy from the error message.