Lesson 17.43 · Kubernetes: Scheduling, Health & Security · 37 min read

Users are certificates: the CSR flow end to end

Q: Why base64 the CSR in the manifest when it is already text?

The field is bytes in the API ([]byte), and bytes are always base64 in JSON and YAML. Pasting the PEM directly breaks the YAML (it is multi-line) or fails with "illegal base64 data". base64 -w0 jane.csr gives the single line you need. The same applies in reverse to status.certificate, which you pipe through base64 -d.

Q: Why use --context instead of use-context when testing?

kubectl --context jane get pods runs one command as jane and leaves your admin context current. With use-context you switch your whole shell to a user who probably cannot do anything, and every following command fails with Forbidden until you notice. On the exam that confusion costs minutes and sometimes tasks done in the wrong context.

In plain words

Imagine a festival where nobody keeps a guest list. Instead, the organisers own a special stamp. You write your name and your crew on a wristband, hand it in, a supervisor checks it and stamps it, and from then on every gate reads the wristband: the name says who you are, the crew says which areas the crew may enter. The gates never phone anyone. They trust the stamp.

Kubernetes users work exactly like that. There is no user database: a client certificate signed by the cluster CA is the wristband. Its CN is the username and every O is a group. You make a key and a request with openssl, submit it as a CertificateSigningRequest, someone approves it, the controller-manager stamps it, and RBAC decides what that name and those groups may do.

A user is a certificate

The problem. Until now you tested people's access with --as=jane. A real new colleague needs her own credential that the cluster trusts - and you need to know, when she says "kubectl does not work", whether the problem is her credential or her permissions.

What you need to know already: RBAC users, groups, 401 vs 403 (17.30, 17.33), TLS certificates, CAs and openssl (9.15, 9.17), kubeconfig and contexts (15.1), base64 (15.33), heredocs (6.18).

Kubernetes has no User objects. kubectl get users does not exist, and there is no API to "create jane". A human user is whatever an authenticator says it is, and on a kubeadm cluster the built-in one is x509 client certificates (x509 = the standard format of TLS certificates; a client certificate proves who the client is, the same way a server certificate proves who the server is): the API server trusts any certificate signed by the cluster CA (--client-ca-file=/etc/kubernetes/pki/ca.crt) and reads the identity out of the certificate's subject:

Subject: CN = jane, O = dev, O = oncall
            |          |         |
         username    group     group

CN (common name) becomes the username: jane.
Every O (organization) becomes a group: dev, oncall. Repeat O for more groups.
The apiserver adds system:authenticated to every authenticated request.

That is all authentication does. Whether jane may do anything is RBAC's job (authorization), and it looks only at those strings: a RoleBinding to User jane or to Group dev. Your own kubeconfig works the same way: kubernetes-admin is a certificate with CN=kubernetes-admin, O=kubeadm:cluster-admins, and a ClusterRoleBinding gives that group cluster-admin.

In production you rarely hand out certificates to humans (managed clusters log people in through OIDC - the company login system), because certificates have one fatal property: they cannot be revoked (cancelled before they expire). The apiserver does not check revocation lists (CRLs - published lists of cancelled certificates) for client certificates. A leaked certificate is valid until it expires, and your only lever is to remove the RBAC bindings its user and groups get. That is why you give them short lives, and why you never, ever issue one with O=system:masters (that group bypasses RBAC entirely: no binding to delete).

The CKA (Certified Kubernetes Administrator, the hands-on Kubernetes exam) still asks for the certificate flow, because it shows you understand authentication vs authorization end to end. Seven steps.

1. A private key

openssl genrsa -out FILE 2048 generates an RSA private key of 2048 bits (genrsa = "generate RSA", -out = where to write it):

$ openssl genrsa -out jane.key 2048
$ ls -l jane.key
-rw------- 1 learner learner 1704 Sep 25 09:12 jane.key
$ head -1 jane.key
-----BEGIN PRIVATE KEY-----

OpenSSL 3 prints nothing and writes the key with mode 600: only you can read it. The key never leaves the user's machine; everything after this step is about the public half.

2. A certificate signing request

A CSR (certificate signing request) contains the public key and the subject you are asking for, signed with the private key (proof you hold it). openssl req -new makes one: -key = your key, -out = the file, -subj = the subject; -noout -subject then prints only the subject back:

$ openssl req -new -key jane.key -out jane.csr -subj "/CN=jane/O=dev"
$ openssl req -in jane.csr -noout -subject
subject=CN=jane, O=dev

The -subj syntax is /type=value/type=value, and it must start with a slash:

$ openssl req -new -key jane.key -out jane.csr -subj "CN=jane,O=dev"
req: subject name is expected to be in the format /type0=value0/type1=value1/type2=... where characters may be escaped by \. This name is not in that format: 'CN=jane,O=dev'
$ openssl req -new -key jane.key -out jane.csr -subj "/CN"
req: Hit end of string before finding the '='

Without -subj, openssl asks you for every field interactively (Country Name, State..., with defaults like AU and Internet Widgits Pty Ltd that you do not want in a username). Always pass -subj. Two groups: -subj "/CN=jane/O=dev/O=oncall".

Key and request in one command, if you prefer (-nodes: do not encrypt the key, kubectl cannot prompt for a passphrase):

$ openssl req -new -newkey rsa:2048 -nodes -keyout jane.key -out jane.csr -subj "/CN=jane/O=dev"

openssl req -in jane.csr -noout -text shows the whole request: Version: 1 (0x0), the subject, the 2048-bit RSA public key, and the signature.

3. The CertificateSigningRequest object

Instead of copying the CSR to the control plane and signing it with the CA key by hand, you submit it to the API as a CertificateSigningRequest object (Kubernetes' own "please sign this" object; kubectl get csr lists them). The manifest (this is the one on the kubernetes.io page "Certificates and Certificate Signing Requests", section Normal user - bookmark it for the exam):

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: jane                     # cluster-scoped: no namespace
spec:
  request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K...   # base64 of jane.csr, one line
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 86400       # one day (minimum 600)
  usages:
  - client auth

request is the base64 of the whole PEM file, on one line: base64 -w0 jane.csr (or cat jane.csr | base64 | tr -d "\n", as the docs write it).
signerName: kubernetes.io/kube-apiserver-client asks for a client certificate the apiserver will accept. (Other built-in signers are for kubelets: kube-apiserver-client-kubelet, kubelet-serving.)
usages: [client auth] - with a space. digital signature and key encipherment are also allowed for this signer; server auth is not.
expirationSeconds caps the certificate's life. Without it you get the controller's default, --cluster-signing-duration, which is one year.

The fastest way to write it without leaving the terminal is a heredoc that does the base64 for you (unquoted EOF, so $(...) runs):

$ cat > jane-csr.yaml <<EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: jane
spec:
  request: $(base64 -w0 jane.csr)
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 86400
  usages:
  - client auth
EOF
$ kubectl apply -f jane-csr.yaml
certificatesigningrequest.certificates.k8s.io/jane created

The mistakes, and what they look like:

# pasted the PEM itself instead of base64 (multi-line: not even valid YAML)
error: error parsing jane-csr.yaml: error converting YAML to JSON: yaml: line 8: could not find expected ':'

# base64 with a stray character (or the -----BEGIN line pasted in)
Error from server (BadRequest): error when creating "jane-csr.yaml": CertificateSigningRequest in version "v1" cannot be handled as a CertificateSigningRequest: illegal base64 data at input byte 3

# usages with a dash, expiry too short
The CertificateSigningRequest "jane" is invalid:
* spec.usages[0]: Unsupported value: "client-auth": supported values: "signing", "digital signature", ... "client auth", ...
* spec.expirationSeconds: Invalid value: 300: may not specify a duration less than 600 seconds (10 minutes)

4. Look at it, approve it

$ kubectl get csr
NAME   AGE   SIGNERNAME                            REQUESTOR          REQUESTEDDURATION   CONDITION
jane   12s   kubernetes.io/kube-apiserver-client   kubernetes-admin   24h                 Pending

REQUESTOR is not something you wrote: the apiserver stamps spec.username and spec.groups with the identity that submitted the object (you), so an approver can see who asked. kubectl describe csr jane shows the requested subject - check it before you approve, because approving signs whatever CN and O the request contains:

$ kubectl describe csr jane
Name:               jane
...
Requesting User:    kubernetes-admin
Signer:             kubernetes.io/kube-apiserver-client
Requested Duration: 24h
Status:             Pending
Subject:
         Common Name:    jane
         Serial Number:
         Organization:   dev
Events:  <none>

Then:

kubectl certificate approve NAME (or deny NAME) records your decision on the request:

$ kubectl certificate approve jane
certificatesigningrequest.certificates.k8s.io/jane approved
$ kubectl get csr jane
NAME   AGE   SIGNERNAME                            REQUESTOR          REQUESTEDDURATION   CONDITION
jane   40s   kubernetes.io/kube-apiserver-client   kubernetes-admin   24h                 Approved,Issued

Two different things happened there. Approved is a condition you (the approver) added. Issued means the signer wrote a certificate into status.certificate - that is the csrsigning controller in kube-controller-manager, which kubeadm starts with --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt and --cluster-signing-key-file=/etc/kubernetes/pki/ca.key. So:

CONDITION stuck at Approved (no Issued)? The controller-manager is down, or the signerName is one nobody signs.
Approved,Failed? The signer refused the request. kubectl get csr jane -o yaml shows a condition with reason: SignerValidationFailure and a message such as invalid usage for client certificate: server auth. Delete it and submit a correct one: the spec is immutable.
kubectl certificate deny jane adds Denied - and a CSR cannot be both:

$ kubectl certificate deny jane
error: certificate signing request "jane" is already Approved

CSR objects do not stay around: the controller-manager garbage-collects approved and denied ones after an hour and pending ones after 24 hours. Copy the certificate out right away. (The simulator keeps them.)

Approving is itself an RBAC permission (update on certificatesigningrequests/approval plus approve on the signer name), which is exactly why the API beats handing people the CA key: you can let a team lead approve client certificates without giving them anything else.

5. Get the certificate out

openssl x509 -in FILE -noout reads a certificate without printing it back; -subject, -issuer, -dates pick the fields to show:

$ kubectl get csr jane -o jsonpath='{.status.certificate}' | base64 -d > jane.crt
$ openssl x509 -in jane.crt -noout -subject -issuer -dates
subject=CN=jane, O=dev
issuer=CN=kubernetes
notBefore=Sep 25 09:08:31 2026 GMT
notAfter=Sep 26 09:13:31 2026 GMT

status.certificate is base64 like the request was, so decode it. Read the output: issuer CN=kubernetes is the kubeadm cluster CA; the validity is the 24 hours you asked for, with notBefore backdated five minutes (the signer allows for clocks that disagree slightly). openssl x509 -in jane.crt -noout -text shows X509v3 Extended Key Usage: TLS Web Client Authentication.

6. A kubeconfig user and context

jane's team works in its own namespace. Make it, with something running in it for her to look at later:

$ kubectl create namespace devteam
namespace/devteam created
$ kubectl create deployment api -n devteam --image=nginx:1.27
deployment.apps/api created

kubectl config set-credentials NAME adds a user entry to your kubeconfig; kubectl config set-context NAME --cluster=... --user=... --namespace=... adds a context (a named cluster + user + default namespace combination, 15.1):

$ kubectl config set-credentials jane --client-key=jane.key --client-certificate=jane.crt --embed-certs=true
User "jane" set.
$ kubectl config set-context jane --cluster=kubernetes --user=jane --namespace=devteam
Context "jane" created.

--embed-certs=true copies the file contents into the kubeconfig as client-certificate-data / client-key-data. Without it, the kubeconfig stores the paths, and the day someone moves the files:

# a user set without --embed-certs, after jane.crt moved
$ kubectl --context jane get pods
error: unable to read client-cert /home/learner/jane.crt for jane due to open /home/learner/jane.crt: no such file or directory

Test with --context jane instead of use-context jane: one command as jane, and your own admin context stays current. (Forgetting to switch back from a user context is a classic way to lose ten exam minutes to "Forbidden" everywhere.) Who does the apiserver think you are?

$ kubectl --context jane auth whoami
ATTRIBUTE   VALUE
Username    jane
Groups      [dev system:authenticated]

7. Authenticated is not authorized

$ kubectl --context jane get pods
Error from server (Forbidden): pods is forbidden: User "jane" cannot list resource "pods" in API group "" in the namespace "devteam"

That is a 403, and it is good news: the certificate works (authentication passed) and RBAC says no, because nothing grants jane anything yet. Grant it - to the group if the permission belongs to the team, to the user if it is personal:

$ kubectl create rolebinding dev-edit -n devteam --clusterrole=edit --group=dev
rolebinding.rbac.authorization.k8s.io/dev-edit created
$ kubectl --context jane get pods
NAME                   READY   STATUS    RESTARTS   AGE
api-6d4b9c7f8d-2lq9x   1/1     Running   0          12m
$ kubectl --context jane auth can-i list nodes
no

Binding the group is what makes certificates manageable: the next developer gets a certificate with O=dev and needs no RBAC change at all. The flip side: to take access away from one person with a group certificate you have to wait for the expiry (or change the binding for everyone).

When it fails: read the error

error	layer	cause
`error: tls: private key does not match public key`	kubectl, before any request	`--client-key` is not the key the CSR was made from
`error: You must be logged in to the server (Unauthorized)`	authentication (401)	certificate expired, or signed by a CA this cluster does not trust (a self-signed one, another cluster's)
`unable to read client-cert ... no such file or directory`	kubectl	kubeconfig points at a path; embed or fix the path
`invalid configuration: client-key-data or client-key must be specified for jane to use the clientCert authentication method`	kubectl	the user entry has a certificate but no key
`Error from server (Forbidden): ... User "jane" cannot ...`	authorization (403)	authenticated fine; RBAC: bind a role to the user or one of its groups

A 401 is never fixed with a RoleBinding, and a 403 is never fixed with a new certificate. And watch the spelling: RBAC compares strings, so a RoleBinding for User Jane does not match CN jane, and nothing warns you.

The manual way (and why not)

Before the CSR API you signed with the CA key directly, on the control plane node. The request has to get there first:

$ scp -q jane.csr cp-1:/tmp/
$ ssh cp-1 sudo openssl x509 -req -in /tmp/jane.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out /tmp/jane.crt -days 30
Certificate request self-signature ok
subject=CN=jane, O=dev

It works - the apiserver only cares who signed it. But it needs root on a control plane node and the CA private key (whoever holds ca.key can mint system:masters), it leaves no audit trail in the API, and nothing records the approval. The CSR API gives you all of that and needs only RBAC. Use the manual way only when the controller-manager is broken.

A certificate you sign with any other CA is simply not trusted. Make one, and a user and context that carry it (the real jane stays as she is - her certificate is embedded in the kubeconfig, so a new file on disk would change nothing for her):

$ openssl req -new -key jane.key -x509 -subj /CN=fake-ca -out fake-ca.crt
$ openssl x509 -req -in jane.csr -CA fake-ca.crt -CAkey jane.key -out fake-jane.crt
Certificate request self-signature ok
subject=CN=jane, O=dev
$ kubectl config set-credentials fake-jane --client-key=jane.key --client-certificate=fake-jane.crt --embed-certs=true
User "fake-jane" set.
$ kubectl config set-context fake-jane --cluster=kubernetes --user=fake-jane
Context "fake-jane" created.
$ kubectl --context fake-jane get pods
error: You must be logged in to the server (Unauthorized)

What you can now do

Create a user's key and CSR with openssl, submit, approve and extract the certificate.
Add a kubeconfig user and context, and test with --context without switching.
Tell a kubectl-side error, a 401 and a 403 apart - and fix each at the right layer.

Why it helps

It is a guaranteed CKA task shape ("create user X in group Y and give it this Role"), and doing it fast means knowing every step by heart: openssl, the manifest with the base64 request, approve, extract, kubeconfig, RoleBinding. It is also the clearest possible lesson in authentication versus authorization: after the certificate works you still get Forbidden, and knowing which layer failed is half of every access ticket.

In real platforms humans usually come through OIDC (the company login), but certificates remain for break-glass kubeconfigs, automation and kubeadm itself. Knowing that they cannot be revoked, and that O=system:masters bypasses RBAC, is what stops a leaked admin certificate from becoming a year-long incident.

Commands in this lesson

openssl ls head cat kubectl scp ssh

FAQ

Where do I create a user object?

Nowhere: there isn't one. Kubernetes only stores Roles and bindings that name users and groups as strings. The identity comes from the authenticator, here the client certificate's CN (user) and O fields (groups). A RoleBinding to User "jane" is valid even if no such certificate exists yet, and nothing warns you about a misspelled name.

Why base64 the CSR in the manifest when it is already text?

The field is bytes in the API ([]byte), and bytes are always base64 in JSON and YAML. Pasting the PEM directly breaks the YAML (it is multi-line) or fails with "illegal base64 data". base64 -w0 jane.csr gives the single line you need. The same applies in reverse to status.certificate, which you pipe through base64 -d.

What is the difference between Approved and Issued?

Approved is a condition a person (or an approver controller) adds with kubectl certificate approve. Issued means the signer actually wrote the certificate into status.certificate. For kubernetes.io/kube-apiserver-client the signer is the csrsigning controller in kube-controller-manager. Approved without Issued means the signer is down or refused; Approved,Failed shows the reason in the Failed condition.

How do I take access away from a certificate user?

You cannot revoke the certificate: the apiserver checks no revocation list. Remove or change the RBAC bindings for the user and for every group in the certificate, and rely on a short expirationSeconds for the rest. That is why group certificates are awkward to offboard and why O=system:masters must never be issued: that group needs no binding at all.

Why use --context instead of use-context when testing?

kubectl --context jane get pods runs one command as jane and leaves your admin context current. With use-context you switch your whole shell to a user who probably cannot do anything, and every following command fails with Forbidden until you notice. On the exam that confusion costs minutes and sometimes tasks done in the wrong context.

In an interview Mid

How do you give a new person access to a kubeadm cluster with a client certificate?

Kubernetes has no User objects: the certificate is the user. The API server trusts certificates signed by the cluster CA; CN becomes the username, each O a group.

Key and request: openssl genrsa -out jane.key 2048, openssl req -new -key jane.key -out jane.csr -subj "/CN=jane/O=dev".
Submit a CertificateSigningRequest object: request = base64 -w0 jane.csr, signerName: kubernetes.io/kube-apiserver-client, usages: [client auth], a short expirationSeconds.
Check the subject (kubectl describe csr jane), then kubectl certificate approve jane; the controller-manager issues it.
Extract: kubectl get csr jane -o jsonpath='{.status.certificate}' | base64 -d > jane.crt.
kubeconfig: kubectl config set-credentials jane --client-key=jane.key --client-certificate=jane.crt --embed-certs=true, plus set-context.
Authorize: a RoleBinding, preferably to the group (--group=dev).
Test with kubectl --context jane auth whoami and get pods: 401 = certificate problem, 403 = RBAC.

Certificates cannot be revoked, so keep them short-lived - and never O=system:masters.

Also asked: A user's kubectl returns errors. How do you tell whether it is the certificate or RBAC? · Why are client certificates risky for human access in production? · What does a CSR stuck in Approved without Issued tell you?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.