Ask the authorizer directly
The problem. "It says Forbidden" - but for whom, for what, and which binding is missing? Guessing and re-deploying wastes an afternoon. kubectl auth can-i asks the authorizer the exact question and answers yes or no.
What you need to know already: RBAC (17.30), ServiceAccounts (17.33), exit codes and $? (1.7).
kubectl auth can-i VERB RESOURCE sends a SelfSubjectAccessReview (a "may I do this?" question object) - or, with --as, the same question impersonating someone - and prints the authorizer's decision. Exit code 0 for yes, 1 for no - scriptable:
$ kubectl auth can-i create deployments -n prod
yes
$ kubectl auth can-i create deployments -n prod --as=jane
no
$ kubectl auth can-i list pods -n dev --as=system:serviceaccount:dev:app
yes
$ kubectl auth can-i get secrets -n dev --as=system:serviceaccount:dev:app -q; echo $?
1
--as works because you (cluster-admin) have the impersonate verb. It is the fastest way to answer the Notion question "how do you check what a specific ServiceAccount is allowed to do": you do not need its token, just its name in the system:serviceaccount:NS:NAME form.
The variations you will need:
$ kubectl auth can-i get pods --subresource=log -n dev --as=jane # subresources
$ kubectl auth can-i create pods/exec -n dev --as=jane # also accepted for create
$ kubectl auth can-i get configmap/app-config -n dev --as=release-bot # a named object (resourceNames)
$ kubectl auth can-i list pods -A --as=jane # cluster-wide: only ClusterRoleBindings count
$ kubectl auth can-i list nodes --as=pager --as-group=oncall # group membership
$ kubectl auth can-i '*' '*' # am I cluster-admin?
$ kubectl auth can-i get /healthz --as=jane # non-resource URLs
Careful with kubectl auth can-i get pods/log: kubectl reads TYPE/NAME, so that asks about a pod named "log". Use --subresource=log.
Everything an identity can do: --list
$ kubectl auth can-i --list -n dev --as=system:serviceaccount:dev:app
Resources Non-Resource URLs Resource Names Verbs
pods [] [] [get list watch]
deployments.apps [] [] [get list watch]
selfsubjectreviews.authentication.k8s.io [] [] [create]
selfsubjectaccessreviews.authorization.k8s.io [] [] [create]
selfsubjectrulesreviews.authorization.k8s.io [] [] [create]
[/.well-known/openid-configuration] [] [get]
[/api] [] [get]
[/api/*] [] [get]
[/healthz] [] [get]
...
The rows you added are on top; the selfsubject* and non-resource URL rows come from the default system:basic-user, system:discovery, system:public-info-viewer and system:service-account-issuer-discovery bindings that every authenticated identity has. --list is per namespace (-n); cluster-wide grants show up in every namespace.
From "Forbidden" to the missing rule
Read the error, then write the rule in the same words:
Error from server (Forbidden): deployments.apps "api" is forbidden: User "system:serviceaccount:ci:deployer" cannot patch resource "deployments" in API group "apps" in the namespace "prod"
- apiGroups: ["apps"] # "in API group apps"
resources: ["deployments"] # "resource deployments"
verbs: ["patch"] # "cannot patch" (kubectl apply/set/scale send an HTTP PATCH; replace sends PUT, 9.21)
# bound by a RoleBinding in prod to ServiceAccount ci/deployer
The opposite question: who can do this?
"Who can delete pods in prod?" has no built-in kubectl answer (the kubectl who-can plugin does it; krew is kubectl's plugin installer). By hand, you check the bindings (-A = all namespaces, -o wide adds the ROLE and subject columns):
# an illustration: the ci/deployer bindings of the incident
kubectl get rolebindings,clusterrolebindings -A -o wide | grep -E 'ci|deployer'
$ kubectl get rolebindings -n prod -o custom-columns=NAME:.metadata.name,ROLE:.roleRef.name,SUBJECTS:.subjects[*].name
and describe the roles they reference. Because RBAC is additive, every binding that matches the subject counts - including group bindings the subject belongs to.
401 or 403
error: You must be logged in to the server (Unauthorized) <- 401: authentication failed (token/cert)
Error from server (Forbidden): ... cannot list resource ... <- 403: authenticated, RBAC said no
A 401 is never fixed with a Role. Check the credential: expired token, deleted SA, client certificate from another cluster, a kubeconfig pointing at the wrong CA (certificate authority - the issuer your client trusts, 9.15).
What you can now do
- Ask yes/no questions with
can-i(as anyone, for subresources, named objects, groups). - List everything an identity may do with
--list, and find the binding that grants it. - Turn a Forbidden message into the exact missing rule.