OnCallReadyStart learning

Security and secrets Lesson 23.21 · Kubernetes Security (CKS) · 11 min read

Supply chain: scan, sign, and enforce at admission

In plain words

Before a delivery enters the building there are three different checks. A scanner looks inside the box for known dangerous items. A label shows which factory sent it and that nobody resealed it. And a guard at the door refuses any box that failed either check, so nobody can carry one in through a side door. Scanning, signing and admission are those three checks.

Only run images you built, without known holes

Scan. trivy image lists known vulnerabilities (CVEs) in an image's packages, with the version that fixes each. First a summary, one row per target: the OS packages, then every jar (0 = clean, - = that scanner did not run); then a section for each target with findings:

Report Summary

┌────────────────────────────────────────────────┬────────┬─────────────────┬─────────┐
│                     Target                     │  Type  │ Vulnerabilities │ Secrets │
├────────────────────────────────────────────────┼────────┼─────────────────┼─────────┤
│ registry.lab/shop/legacy-api:1.0 (debian 12.1) │ debian │        2        │    -    │
├────────────────────────────────────────────────┼────────┼─────────────────┼─────────┤
│ app/legacy-api.jar                             │  jar   │        0        │    -    │
└────────────────────────────────────────────────┴────────┴─────────────────┴─────────┘
...
registry.lab/shop/legacy-api:1.0 (debian 12.1)
==============================================
Total: 2 (HIGH: 1, CRITICAL: 1)

│ Library  │ Vulnerability  │ Severity │ Status │ Installed Version │   Fixed Version   │ ...
│ libc6    │ CVE-2023-4911  │ HIGH     │ fixed  │ 2.36-9+deb12u1    │ 2.36-9+deb12u3    │ ...
│ libcurl4 │ CVE-2023-38545 │ CRITICAL │        │ 7.88.1-10+deb12u1 │ 7.88.1-10+deb12u4 │ ...

A cell equal to the one above it is left blank (Status: both fixed). In CI, trivy image --exit-code 1 --severity HIGH,CRITICAL <image> fails the pipeline: the gate. (simulator) the lab uses a small vulnerability database; the two CVEs are real Debian 12 advisories (glibc "Looking Glass", curl SOCKS5), with the severity trivy shows (NVD's: the Debian tracker rates neither).

Inside the app too. trivy also reads the libraries packed in the image's jars (catalog.jar is a fat jar, with them inside under BOOT-INF/lib/): one summary row per jar, and in its section a library shows as org.apache.logging.log4j:log4j-core (catalog.jar). A jar with log4j-core 2.14.1 is Log4Shell (CVE-2021-44228, CRITICAL, fixed in 2.15.0; 2.17.1 fixes the whole family). A new base image clears the OS rows, never these: only a rebuild with the fixed library does.

Inventory. An SBOM (software bill of materials) lists every package in an image. syft <image> prints it; syft <image> -o spdx-json > sbom.json writes a standard SPDX document you keep with the release, so when the next big CVE lands you can answer "which images contain it?" without rescanning.

Sign. A scan says "no known holes"; a signature says "this is the image we built". cosign (v3):

cosign generate-key-pair                      # cosign.key (private) + cosign.pub
cosign sign --key cosign.key -y registry.lab/shop/api:2.3
cosign verify --key cosign.pub registry.lab/shop/api:2.3

cosign v3 packs the signature, and what is needed to check it, into a Sigstore bundle that it attaches to the image in the registry, and it records the signature in Rekor, Sigstore's public transparency log. So sign first prints the Sigstore privacy statement and asks Are you sure you would like to continue? [y/N]; -y answers yes. verify checks the log entry, then the signature:

Verification for registry.lab/shop/api:2.3 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key

An image nobody signed: Error: no signatures found, exit code 10. Signed, but with another key: the log entry names that key (transparency log certificate does not match), exit code 1. Older guides add --tlog-upload=false; cosign v3 refuses it while it uses Sigstore's signing config (the default).

Sign by digest (image@sha256:...) in real use: a tag can be moved to another image after you signed it (cosign warns when you sign a tag, and says signing by tag will go away).

Enforce. Nobody runs cosign verify by hand before every deploy. An admission policy does it on every Pod. With Kyverno 1.19 that is an ImageValidatingPolicy: it picks the images to check (matchImageReferences), names who may sign them (attestors), and a CEL validation calls verifyImageSignatures, which returns how many attestors verified the image.

apiVersion: policies.kyverno.io/v1
kind: ImageValidatingPolicy
metadata:
  name: verify-shop-images
spec:
  validationActions: [Deny]              # Audit = only report
  matchConstraints:
    resourceRules:
    - apiGroups: ['']
      apiVersions: ['v1']
      operations: ['CREATE', 'UPDATE']
      resources: ['pods']
  matchImageReferences:
  - glob: "registry.lab/shop/*"
  attestors:
  - name: cosign
    cosign:
      key:
        data: |
          -----BEGIN PUBLIC KEY-----
          ...
  validations:
  - expression: >-
      images.containers.map(image, verifyImageSignatures(image, [attestors.cosign])).all(e, e > 0)
    message: failed the image Signature verification

An unsigned image is refused by Kyverno's image webhook, which names the policy:

Error from server: admission webhook "ivpol.validate.kyverno.svc-fail" denied the request: Policy
verify-shop-images failed: failed the image Signature verification

(A policy with matchConditions gets a webhook of its own: ivpol.validate.kyverno.svc-fail-finegrained-<policy>.)

Digests by default. spec.validationConfigurations has three switches, all on unless you turn them off: mutateDigest rewrites every matched image to <image>:<tag>@sha256:... at admission (in Audit too), verifyDigest fails an image that has no digest, and required fails an image no policy verified. So the pod that gets in runs registry.lab/shop/api:2.3@sha256:...: the build that was checked, not whatever the tag points at tomorrow. A signature belongs to that digest, so a new tag on a hand-built image has none. Autogen applies the same policy to Deployments, StatefulSets, DaemonSets, Jobs and CronJobs: kubectl set image deployment/... is checked itself, not only its pods.

Older clusters use the same check as a verifyImages rule in a kyverno.io/v1 ClusterPolicy (imageReferences, attestors[].entries[].keys.publicKeys). Kyverno 1.19 (August 2026) deprecates ClusterPolicy and plans to remove it in 1.20; a cluster that still has them prints a deprecation warning on every apply. Write new policies with the policies.kyverno.io types.

In an interview: "scanning, signing and admission are three different controls: known vulnerabilities, provenance, enforcement".

On the exam: domain "Supply chain security" - scan with trivy, find the image with a given CVE, write an admission policy.

Why it helps

Most clusters run images that nobody in the team built themselves, and every image carries hundreds of packages. Scanning, keeping an SBOM, signing and enforcing at admission are separate controls with separate purposes, and confusing them is a common interview mistake. The supply chain is also one of the most heavily weighted CKS domains. This lesson covers each control on its own, then shows how they fit together.

On the blog

FAQ

What does trivy image report?

The known vulnerabilities (CVEs) in the image's packages, with the installed version, the version that fixes each one, and a severity. It answers whether the image contains holes that are already public, not whether it is the image you built.

How does trivy become a CI gate?

With --exit-code 1 and --severity HIGH,CRITICAL: the scan exits non-zero when such vulnerabilities are found, which fails the pipeline step. Lower severities are still reported but do not block the build. Keep the gate in every pipeline that builds images.

What is an SBOM for?

A software bill of materials lists every package in an image. syft writes one, for example as SPDX JSON, and you keep it with the release, so when the next big CVE lands you can find every affected image without rescanning them all.

What does a cosign signature prove?

That the image is the one signed with your key, its provenance. It says nothing about vulnerabilities inside, which is why signing complements scanning instead of replacing it. Sign by digest, because a tag can be moved after signing.

What does a Kyverno ImageValidatingPolicy do?

At admission it checks the images that match matchImageReferences: verifyImageSignatures counts the attestors, such as your cosign key, that verified each one. With validationActions Deny an unsigned image is refused; Audit only records it, which is how you start a new policy safely.

In an interview Mid

How do you make sure only trusted images run in a cluster?

Scan images in CI with trivy image and fail on HIGH and CRITICAL, keep an SBOM with syft, sign the images that pass with cosign (by digest), and enforce the signature at admission with a Kyverno ImageValidatingPolicy in Deny mode, so an unsigned or foreign image is rejected however it was deployed.

Also asked: What is the difference between scanning and signing? · Why start a new policy in Audit mode? · What would you do with an SBOM during a new CVE?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.