Lesson 17.33 · Kubernetes: Scheduling, Health & Security · 16 min read

ServiceAccounts, tokens, and automountServiceAccountToken

Q: Can I change a running pod's ServiceAccount?

No. serviceAccountName is part of the pod spec and can't be changed in place. For a Deployment you change the pod template (kubectl set serviceaccount deploy/x sa-name), which triggers a rollout with new pods. A pod naming a ServiceAccount that doesn't exist is rejected at admission, visible as FailedCreate on the ReplicaSet.

Q: What is a ServiceAccount, and how does a pod use it?

A ServiceAccount is the identity a pod uses when it calls the Kubernetes API (a controller, a CI runner, an app using a Kubernetes client library). RBAC sees it as system:serviceaccount:<namespace:<name. - Every namespace has a default SA, and pods that do not name one run as it. - The kubelet mounts a projected, bound, short-lived token (about an hour, rotated, tied to the pod) with the cluster CA at /var/run/secrets/kubernetes.io/serviceaccount/; client libraries read it from there. Good practice: - A dedicated SA per workload that calls the API (kubectl create serviceaccount podwatcher, serviceAccountName in the pod template) with a Role for exactly what it needs. - automountServiceAccountToken: false for everything that never calls the API - otherwise any permission someone grants to default reaches every pod. kubectl create token SA --duration=10m issues a token by hand; 401 means the token is bad or expired, 403 means RBAC said no.

In plain words

Imagine every worker in a factory gets a visitor badge automatically, even the ones who never need to enter the office. The badge is tied to the worker's shift: it expires after an hour and stops working the moment they leave. Most workers never use it, but if a manager ever gives "all visitor badges" access to the safe, every worker can open it.

A ServiceAccount is a pod's identity when it calls the Kubernetes API. Every namespace has a default one, and every pod gets a projected, bound, short-lived token mounted at /var/run/secrets/kubernetes.io/serviceaccount. Good hygiene: automountServiceAccountToken: false for pods that don't call the API, and a dedicated ServiceAccount with a narrow Role for those that do.

Every pod has an identity

The problem. Some programs inside the cluster talk to the apiserver themselves: a tool that lists pods, a job that restarts Deployments, a controller. They need an identity RBAC can grant things to - and ordinary apps, which never call the API, should not carry one at all.

What you need to know already: RBAC subjects, Roles and bindings (17.30), Secrets, base64 and the Downward API (15.33), volumes (16.37), cut and pipes (7.4).

A ServiceAccount (SA) is the identity a pod uses when it talks to the apiserver (a controller, a CI runner - the program that runs automated builds and tests - or any app using a Kubernetes client library such as client-go for Go or the Java client). Every namespace gets a default SA automatically, and every pod that does not name one runs as it:

# an illustration: shop/web and a podwatcher Deployment (the ServiceAccount mission)
kubectl get sa -n shop
NAME      SECRETS   AGE
default   0         30d
kubectl get pod web-5d8f-2kq9x -n shop -o jsonpath='{.spec.serviceAccountName}'
default

kubectl get sa = list ServiceAccounts. SECRETS 0 - since 1.24 no long-lived token Secret is created for an SA. Instead the kubelet mounts a projected, bound, short-lived token into every pod (a projected volume combines several sources - here a token, a ConfigMap and Downward API fields - into one directory). The grep -A12 below prints the volume and its next 12 lines:

# an illustration: shop/web and a podwatcher Deployment (the ServiceAccount mission)
kubectl get pod web-5d8f-2kq9x -n shop -o yaml | grep -A12 'kube-api-access'
  - name: kube-api-access-7x2kq
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              fieldPath: metadata.namespace
            path: namespace
kubectl exec web-5d8f-2kq9x -n shop -- ls /var/run/secrets/kubernetes.io/serviceaccount
ca.crt
namespace
token

bound: the token is tied to this pod (and SA); delete the pod and it stops being valid, even before it expires.
short-lived: ~1 hour (3607s), the kubelet rotates the file before expiry. Client libraries re-read it.
audience: the apiserver. A token stolen from a log file is worth an hour at most.
ca.crt is the cluster's certificate authority, so the app can check it is really talking to the apiserver (TLS trust, 9.15); namespace is the pod's namespace.

Don't hand out identities that nobody needs

Most application pods never call the Kubernetes API, yet by default each one carries a token for the namespace's default SA. If anyone ever grants default something (people do - "just give default view, it was quicker"), every pod in the namespace gets it. Two habits:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: shop
automountServiceAccountToken: false      # SA level: pods of this SA get no token by default
---
spec:
  automountServiceAccountToken: false    # pod level (wins over the SA setting)

...and a dedicated SA per workload that does call the API, with a Role for exactly what it needs:

# an illustration: shop/web and a podwatcher Deployment (the ServiceAccount mission)
kubectl create serviceaccount podwatcher -n shop
serviceaccount/podwatcher created
kubectl set serviceaccount deploy/podwatcher podwatcher -n shop
deployment.apps/podwatcher serviceaccount updated

kubectl create serviceaccount NAME makes the SA; kubectl set serviceaccount deploy/NAME SA sets serviceAccountName in the Deployment's pod template.

serviceAccountName is part of the pod template, so changing it is a rollout; a pod's SA cannot be changed in place. A pod naming an SA that does not exist is rejected at admission:

Error creating: pods "podwatcher-6d9f-" is forbidden: error looking up service account shop/podwatcher: serviceaccount "podwatcher" not found

Tokens by hand: kubectl create token

The TokenRequest API issues a fresh token on demand - for CI, for debugging, for a kubeconfig. kubectl create token SA -n NS --duration=10m prints one (default lifetime 1 hour):

# an illustration: shop/web and a podwatcher Deployment (the ServiceAccount mission)
kubectl create token podwatcher -n shop --duration=10m
eyJhbGciOiJSUzI1NiIsImtpZCI6Ik5sM1dTa1dUSjRxZ1RlMWVOQ0R6dWxTa2c0UzFoUDBvTkdxV3YzSzhmMkEifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXS...

The token is a JWT (JSON Web Token): three base64url-encoded parts joined by dots, header.payload.signature. The payload is not encrypted - anyone can read it. Below, cut -d. -f2 takes the second dot-separated field and base64 -d decodes it:

# an illustration: shop/web and a podwatcher Deployment (the ServiceAccount mission)
TOKEN=$(kubectl create token podwatcher -n shop)
echo "$TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null; echo
{"aud":["https://kubernetes.default.svc.cluster.local"],"exp":1790253600,"iat":1790250000,"iss":"https://kubernetes.default.svc.cluster.local","jti":"...","kubernetes.io":{"namespace":"shop","serviceaccount":{"name":"podwatcher","uid":"..."}},"nbf":1790250000,"sub":"system:serviceaccount:shop:podwatcher"}

(base64 -d may complain about missing padding - JWTs strip the =; the JSON is still printed.) The fields: aud = audience (who should accept it), exp/iat/nbf = expires / issued at / not before (Unix timestamps), iss = issuer. sub is the user name RBAC will see. The signature is what makes it trustworthy: signed with the cluster's service-account key, verified by the apiserver.

Use it:

# an illustration: shop/web and a podwatcher Deployment (the ServiceAccount mission)
kubectl get pods -n shop --token="$TOKEN"
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:shop:podwatcher" cannot list resource "pods" in API group "" in the namespace "shop"

--token replaces the kubeconfig's credentials for that one command - you are now exactly what the pod is. A token for a deleted (and recreated) SA fails authentication, not authorization:

error: You must be logged in to the server (Unauthorized)

(401 Unauthorized = authentication failed: bad, expired or revoked token. 403 Forbidden = authenticated fine, RBAC said no. Different fixes.)

For a CI system outside the cluster, prefer a login it gets automatically and that expires by itself over a token you generated by hand; a long --duration token pasted into the CI settings is a credential nobody will rotate (replace regularly).

What you can now do

Give a workload its own ServiceAccount with a minimal Role, and turn tokens off for pods that do not need one.
Create a token by hand, read its payload, and use it with kubectl --token.
Tell 401 Unauthorized (who are you?) from 403 Forbidden (not allowed).

Why it helps

ServiceAccounts are how your operators, controllers, CI runners and any app using client-go talk to the cluster, so you'll create and scope them constantly. The security side matters just as much: the default ServiceAccount token in every pod is a gift to an attacker if someone once granted default extra rights "because it was quicker".

You'll also debug the auth side of things: 401 versus 403 (a deleted and recreated ServiceAccount fails authentication, not authorization), pods rejected because they name a ServiceAccount that doesn't exist, and CI jobs with a long-lived token pasted into a variable. kubectl create token is also handy for testing permissions.

FAQ

Why does my ServiceAccount show SECRETS 0?

Since Kubernetes 1.24, no long-lived token Secret is created for ServiceAccounts. Instead the kubelet mounts a projected token from the TokenRequest API into each pod: bound to that pod, valid for about an hour, rotated automatically. You can still create a long-lived token Secret by hand, but you almost never should.

What happens to the token when the pod is deleted?

It stops being valid, even before its expiry. Projected tokens are bound to the pod object (and the ServiceAccount), and the API server checks that the pod still exists. A token copied out of a deleted pod, or out of a log file after the hour, is worthless. That's a big improvement over the old never-expiring Secret tokens.

Can I change a running pod's ServiceAccount?

No. serviceAccountName is part of the pod spec and can't be changed in place. For a Deployment you change the pod template (kubectl set serviceaccount deploy/x sa-name), which triggers a rollout with new pods. A pod naming a ServiceAccount that doesn't exist is rejected at admission, visible as FailedCreate on the ReplicaSet.

Is the JWT token encrypted?

No. A JWT is three base64url parts: header, payload and signature. Anyone can decode the payload and read the audience, expiry, namespace and sub (system:serviceaccount:ns:name). The signature, made with the cluster's service-account key, is what makes it trustworthy. Treat tokens as secrets, because possession is all that's needed to use them.

What's the difference between 401 and 403 with a token?

401 Unauthorized means authentication failed: the token is invalid, expired, revoked, or for a ServiceAccount that was deleted and recreated. No Role can fix that. 403 Forbidden means the API server knows who you are and RBAC said no; adding the right Role and binding fixes it.

In an interview Junior

What is a ServiceAccount, and how does a pod use it?

A ServiceAccount is the identity a pod uses when it calls the Kubernetes API (a controller, a CI runner, an app using a Kubernetes client library). RBAC sees it as system:serviceaccount:<namespace>:<name>.

Every namespace has a default SA, and pods that do not name one run as it.
The kubelet mounts a projected, bound, short-lived token (about an hour, rotated, tied to the pod) with the cluster CA at /var/run/secrets/kubernetes.io/serviceaccount/; client libraries read it from there.

Good practice:

A dedicated SA per workload that calls the API (kubectl create serviceaccount podwatcher, serviceAccountName in the pod template) with a Role for exactly what it needs.
automountServiceAccountToken: false for everything that never calls the API - otherwise any permission someone grants to default reaches every pod.

kubectl create token SA --duration=10m issues a token by hand; 401 means the token is bad or expired, 403 means RBAC said no.

Also asked: How do you give a CI job access to deploy to a cluster? · Why set automountServiceAccountToken: false, and where? · What is inside a ServiceAccount token, and is it encrypted?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.