OnCallReady

Lesson 10.54 · Images & Builds · 21 min read

Registries, tagging strategy and multi-arch images

In plain words

Think of a registry as a library for images. You need a library card to borrow or donate (docker login). When you donate a book, you write the library's name and the shelf on the cover first (docker tag registry.lab/team/orders:2.14.1), then hand it in (docker push). A careful library never lets anyone swap the pages of a book with the same title and edition: a new edition gets a new number.

And books come in languages. Your Mac and this VM read arm64; most servers read amd64. A multi-arch image is one title with both translations on the shelf, and each reader automatically gets the one they understand. Hand an arm64-only book to an amd64 server and it says exec format error.

Why this matters

An image on your laptop helps nobody. Shipping means pushing it to a registry under a name that will still mean the same thing next month - and built for the CPU of the machine that will run it. Your Mac and this VM are arm64; most servers are amd64. This lesson is the last mile.

What you need to know already: image references, tags, digests and latest ("Images, tags and digests"), arm64 vs amd64 and exec format error ("What a container actually is"), input redirection with < (Ch 1), JSON (Ch 7).

Logging in

A private registry wants to know who you are. docker login stores your credentials so later pulls and pushes can use them:

# the registry mission below creates the token and the image
docker login registry.lab -u learner --password-stdin < ~/.registry-token

WARNING! Your password will be stored unencrypted in /home/learner/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credential-stores

Login Succeeded

-u is the user; --password-stdin reads the password (here a token, a long random password made for machines) from stdin, which < fills from a file. Believe the warning: ~/.docker/config.json now holds your credentials as base64, which is an encoding (a way to write bytes as plain letters), not encryption.

# after that login
cat ~/.docker/config.json
{
	"auths": {
		"registry.lab": {
			"auth": "bGVhcm5lcjpsYWItcmVnaXN0cnktdG9rZW4="
		}
	}
}
$ echo bGVhcm5lcjpsYWItcmVnaXN0cnktdG9rZW4= | base64 -d
learner:lab-registry-token

(base64 -d decodes.) On a workstation use a credential helper, a small program that keeps the password in the system keychain instead (docker-credential-pass, osxkeychain on the Mac); on build servers use short-lived tokens. And --password-stdin rather than -p PASSWORD, which leaves the password in your shell history and in ps (Ch 3).

Errors you will see:

unauthorized: authentication required                      push without login
no basic auth credentials                                   pull from a private registry without login
denied: requested access to the resource is denied          logged in, but not allowed (or wrong repo name)

Tag, then push

An image is pushed to the name it is tagged with, so the registry host goes into the tag:

docker tag orders:slim registry.lab/team/orders:2.14.1
docker push registry.lab/team/orders:2.14.1
The push refers to repository [registry.lab/team/orders]
5f1e2d3c4b5a: Pushed
a8b7c6d5e4f3: Pushed
2eeab876c9ff: Mounted from library/eclipse-temurin
2.14.1: digest: sha256:7c2d... size: 1570

docker tag SOURCE NEW-NAME does not copy anything; it adds a name to the same image ID. docker images shows both names with one IMAGE ID.

A tagging strategy

registry.lab/team/orders:2.14.1          semver, one per release, NEVER moved
registry.lab/team/orders:2.14.1-3f9c2ab  version + git short SHA, traceable to a commit
registry.lab/team/orders:3f9c2ab         every automated build of main
registry.lab/team/orders:2.14            moving pointer for "latest patch" - humans only
registry.lab/team/orders:latest          do not deploy this. Ever.

semver (semantic versioning) is MAJOR.MINOR.PATCH: 2.14.1 is major 2, minor 14, patch 1. The git short SHA is the first 7 characters of a commit id (git log --oneline, Ch 1). The properties you want: every deployed tag maps to exactly one commit, nothing you deployed can change underneath you, and a rollback is "deploy the previous tag", which still means what it meant yesterday.

Registries can enforce that with immutable tags: once pushed, a tag can never point anywhere else. Most private registries support it. This lab's registry rejects overwriting a semver tag:

docker push registry.lab/team/orders:2.14.1
unknown: The image tag '2.14.1' already exists in the 'team/orders' repository and cannot be overwritten because the repository is immutable.

A fixed image is a new version (2.14.2), not a re-push.

Pulling by digest

# digest shortened
docker pull registry.lab/team/orders@sha256:7c2d...
docker inspect -f '{{index .RepoDigests 0}}' registry.lab/team/orders:2.14.1
registry.lab/team/orders@sha256:7c2d...

Whatever starts your containers in production should reference the digest, or a tag the registry makes immutable.

Multi-arch: your Mac is not your server

This VM is arm64, like your Mac. Most servers are amd64. Every image has an architecture:

$ docker image inspect -f '{{.Architecture}}' orders:slim
arm64

Build on the Mac, push, run on an amd64 server, and:

exec /opt/java/openjdk/bin/java: exec format error

exec format error = the kernel cannot execute this binary's instruction set. The reverse happens here: pull an image someone built only for amd64 and run it on this box:

# during the registry mission (logged in to registry.lab)
docker run --rm registry.lab/tools/reporter:1.4
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
exec /usr/local/bin/reporter: exec format error

A platform is OS + CPU, written linux/amd64, linux/arm64. The fix is to publish multi-arch images: one tag that points at an index (also called a manifest list: a small file with one image per platform). The client pulls the one that matches its CPU:

$ docker buildx imagetools inspect nginx:1.27
Name:      docker.io/library/nginx:1.27
MediaType: application/vnd.oci.image.index.v1+json
...
Manifests:
  Platform:    linux/amd64
  Platform:    linux/arm64/v8

(docker buildx imagetools inspect reads an image's index straight from the registry, without pulling it.) Building one needs a builder that can produce several platforms (the default docker driver cannot):

# in a project directory, logged in to registry.lab
docker build --platform linux/amd64,linux/arm64 -t x .
ERROR: Multi-platform build is not supported for the docker driver.
Switch to a different driver, or turn on the containerd image store, and try again.
docker buildx create --name multi --driver docker-container --use
docker buildx build --platform linux/amd64,linux/arm64 -t registry.lab/team/orders:2.14.1 --push .

buildx create --driver docker-container --use starts a BuildKit builder in its own container and makes it the default (--use); --platform lists the targets; --push uploads the result straight to the registry (a multi-arch index cannot be stored in the local image list).

RUN steps for a foreign platform execute under QEMU emulation (a program that pretends to be the other CPU; install it with docker run --privileged --rm tonistiigi/binfmt --install all), which is slow. For compiled languages, cross-compile instead: run the compiler on your own platform and tell it to produce the other one's code:

FROM --platform=$BUILDPLATFORM golang:1.23 AS build
ARG TARGETOS TARGETARCH
RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH go build -o /out/app .

BUILDPLATFORM (where the build runs), TARGETOS and TARGETARCH (what it is building for) are set automatically by BuildKit; you only declare them with ARG to use them. GOOS/GOARCH tell the Go compiler the target. Java is easier still: a JAR runs on any CPU and the JRE base images are already multi-arch.

What you can now do

Why it helps

This is the daily plumbing of shipping software. The unauthorized versus denied errors tell you whether the problem is login or permissions when a pipeline fails at push. The base64 in ~/.docker/config.json is why a leaked CI runner or laptop leaks registry credentials, and why short-lived tokens are the norm on build servers. A tagging strategy with immutable semver tags is what makes "roll back to the previous version" safe during an incident. And exec format error is the classic Apple Silicon problem: you build on your Mac, push, and the container on an amd64 server crashes instantly. Knowing buildx, --platform and cross-compilation fixes it in minutes instead of an afternoon.

Commands in this lesson

echo docker

FAQ

Is the password in ~/.docker/config.json encrypted?

No. Without a credential helper, docker login stores user:password as base64 in the auth field, and base64 is encoding: base64 -d reverses it. Anyone who can read the file has your credentials. Configure a credential helper (osxkeychain on the Mac, pass or secretservice on Linux) via credsStore, and in CI use short-lived tokens that expire after the job.

What is the difference between unauthorized and denied?

unauthorized: authentication required and no basic auth credentials mean the registry does not know who you are: you are not logged in to that host, or the token expired. denied: requested access to the resource is denied means it knows who you are but you lack permission, or the repository name is wrong, which on Docker Hub is often a missing namespace. The first is fixed by logging in; the second by roles or the right name.

Does docker tag copy the image?

No. docker tag adds another name pointing at the same image ID; no bytes are copied, and docker images shows both names with one IMAGE ID. Pushing uploads to the registry named in the tag, and only the layers the registry does not have yet: shared base layers show "Layer already exists" or "Mounted from", which is why pushes of small top layers are fast.

Why is latest a bad tag to deploy?

latest is just the default tag name, not "the newest version". It moves whenever someone pushes without a tag, so it tells you nothing about which code is running, two servers can run different content, and rolling back to "the previous latest" is impossible. And a server set to always pull on start upgrades silently on a routine restart. Deploy immutable version tags, ideally resolved to digests.

Why does my image fail with exec format error on the server?

The binaries in the image were built for a different CPU architecture than the server. An image built on your Apple Silicon Mac or this arm64 VM is linux/arm64 by default; most servers are amd64. Check with docker image inspect -f '{{.Architecture}}'. Build for the target with docker buildx build --platform linux/amd64, or publish a multi-arch image with both platforms under one tag.

In an interview Junior

Describe a good image tagging strategy.

Tags that never move, each traceable to one commit:

Make the registry enforce it with immutable tags, so a re-push of 2.14.1 is rejected. Record the digest docker push prints, and have production reference the digest (or an immutable tag). Then a rollback is "deploy the previous tag", and it still means what it meant yesterday.

The mechanics: docker login registry.lab --password-stdin (not -p), docker tag orders:slim registry.lab/team/orders:2.14.1, docker push.

Also asked: How do you push an image to a private registry? · What does "exec format error" mean when a container starts? · What is a multi-arch image, and how do you build one?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.