OnCallReady

Lesson 25.21 · CI/CD Pipelines & Helm · 18 min read

Values: where every number comes from

In plain words

Imagine ordering a pizza. The menu has a default: medium, tomato, cheese. You write your wishes on a note: "large". Your friend adds another note: "no cheese". Then you shout at the counter "extra olives!". The cook starts from the menu and applies each note in order, and the shout always comes last, so it beats the notes. If you write "extra onoins", the cook just ignores what he cannot read.

Helm values work like that. The chart's values.yaml is the menu. -f files are the notes, applied left to right, so the last file wins. --set is the shout and beats every file. Maps merge key by key, lists are replaced whole, null deletes a default, and a misspelled key like replica is silently ignored unless values.schema.json forbids it.

The problem

A change "did nothing": someone set a value, Helm said Upgrade complete, and the app still behaves the old way. Nine times out of ten the value lost to another layer, went to a key the chart never reads, or was dropped by the next upgrade. To answer "where does this number come from?", you need the merge rules.

What you need to know already: 25.19 (chart, values, release, revision), 15.29 (ConfigMaps), 17.1 (requests and limits).

Words for this lesson

-f FILE (or --values FILE) passes a YAML file of values; --set KEY=VALUE passes one value on the command line, with dots for nesting (--set image.tag=1.4.1 = image: {tag: 1.4.1}).

The merge, in order

For one install or upgrade, Helm builds the values from these layers, each one overriding the one before:

1. the chart's values.yaml                      (defaults)
2. for a subchart: the parent's values under the subchart's name (and global:, 25.29)
3. -f / --values files, left to right           (the last file wins)
4. --set-json        } by KIND, in this order,
5. --set             } whatever order you typed them;
6. --set-string      } left to right within a kind
7. --set-file
8. --set-literal

So helm upgrade app ./chart -f base.yaml -f prod.yaml --set replicaCount=5 always ends with 5 replicas, and -f prod.yaml -f base.yaml lets base win over prod - the order of -f is the most common real mistake. --set beats any file, which is why pipelines that pass --set for "just the image tag" and then someone adds image.tag to a values file get a surprise: the file loses.

Maps merge, lists do not

In YAML a map is a set of key: value pairs; a list is a sequence of - item entries (or [a, b]). The two files below side by side:

# values.yaml (chart)             # prod.yaml
resources:                         resources:
  requests:                          requests:
    cpu: 50m                           memory: 384Mi
    memory: 256Mi
  limits:
    memory: 512Mi

Result: requests: {cpu: 50m, memory: 384Mi}, limits: {memory: 512Mi}. Maps are merged key by key, recursively. Lists are not: if the chart default is tolerations: [a, b] (tolerations, Ch 17) and your file says tolerations: [c], you get [c]. There is no way to append to a list from values - that is why good charts use maps keyed by name (extraEnv: {FOO: bar}) and render the list in the template.

null deletes. null is YAML for "no value". To remove a default entirely, set it to null:

resources:
  limits: null        # no limits block at all in the output

or --set resources.limits=null. An empty map limits: {} does not remove the chart's keys - it merges nothing into them.

--set, typed

--set has its own tiny language, and it guesses the type of each value (integer, true/false boolean, or string = text). int64 below is a whole-number type:

--set replicaCount=3              int64 3
--set enabled=true                bool true      ("True", "TRUE" too)
--set tag=1.10                    string "1.10"  (floats are NOT converted)
--set tag=0123                    string "0123"  (leading zero: stays a string)
--set x=null                      deletes x
--set-string replicaCount=3       string "3"     (a Deployment will reject it)
--set list={a,b,c}                a list
--set env[0].name=FOO             index into a list (creates it)
--set 'nodeSelector.kubernetes\.io/os=linux'   a dot in a key must be escaped
--set 'args={--port,8080}'        braces: a list; a comma inside a VALUE needs \,
--set-json 'resources={"limits":{"memory":"1Gi"}}'
--set-file config=./app.properties  the file's content as a string

YAML files have their own traps, and they bite harder because nobody reads the type: tag: 1.10 in a values file is the float 1.1 (the image app:1.1); country: NO and enabled: on are YAML 1.1 booleans in some parsers; mode: 0755 may become 493. Quote anything that is a version, an id, or a code: tag: "1.10".

Numbers from values files are decoded through JSON, so every number in values.yaml is a float64 (a decimal-number type, like 1.5) - and Go prints big floats in exponent form: maxBytes: 1500000 renders as 1.5e+06, 1000000 as 1e+06. Small ones look fine (3 prints 3), which is why this bites months later when someone raises a limit. {{ .Values.maxBytes | int }} (pipe it through int to turn it back into a whole number) (or a quoted string in the values) fixes it. --set integers are int64 and print normally - the same key can render differently depending on where it was set.

A wrong key is not an error

Helm has no idea what keys a chart reads. This is accepted silently:

helm upgrade payments-api ./chart --set replica=5          # the chart reads replicaCount
helm upgrade payments-api ./chart -f timeout.yaml           # timeout.yaml nests under payments-api:

The release gets a new revision, STATUS: deployed, Upgrade complete - and nothing changes. Two defences:

  1. Compare helm get values (what was supplied) with what the templates read.
  2. Ship a values.schema.json (a JSON Schema, 25.19: it lists which keys exist and what type each must be). Helm validates the merged values on install, upgrade, lint and template:
{
  "$schema": "https://json-schema.org/draft-07/schema#",
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "replicaCount": { "type": "integer", "minimum": 1 },
    "image": { "type": "object" },
    "config": {
      "type": "object",
      "additionalProperties": false,
      "properties": {
        "profile": { "type": "string" },
        "gatewayUrl": { "type": "string", "pattern": "^https?://" },
        "gatewayTimeout": { "type": "string", "pattern": "^[0-9]+(ms|s|m)$" }
      }
    }
  }
}

Reading it: the values must be an object with replicaCount (an integer of at least 1), image and config; pattern is a regular expression the string must match. With "additionalProperties": false a stray key fails the upgrade before anything is applied:

Error: UPGRADE FAILED: values don't meet the specifications of the schema(s) in the following chart(s):
payments-api:
- at '': additional properties 'replica' not allowed

Upgrade and the old values

What happens to the values of the previous revision on helm upgrade?

no values flag at all        the previous revision's supplied values are REUSED
any -f/--set given           ONLY this command's values (+ the new chart's defaults)
--reset-values               only this command's values, even if there are none
--reuse-values               previous supplied values + this command's, merged -
                             and the new chart's defaults are IGNORED (the old
                             computed values stand in for them)
--reset-then-reuse-values    new chart defaults + previous supplied + this command's

The second line is the one that loses production settings: the pipeline always passed -f prod.yaml, someone runs helm upgrade payments-api ./chart --set image.tag=1.4.1 by hand and prod drops to the chart defaults (2 replicas, no profile). The --reuse-values trap is subtler: a new chart version adds a default (say a readiness probe path) and your release never gets it. Since Helm 3.14, --reset-then-reuse-values is the flag you actually want for a manual change - and the real answer is that manual upgrades should not exist: the pipeline passes every file, every time.

Reading what a release got

helm get values payments-api -n pay-prod           # USER-SUPPLIED VALUES only
helm get values payments-api -n pay-prod --all     # COMPUTED VALUES: the full merge
helm get values payments-api -n pay-prod --revision 4
helm template payments-api ./chart -f prod.yaml --show-only templates/configmap.yaml

--all adds the chart defaults (the full merge); --revision 4 looks at an older revision. --show-only (-s) renders one template, which is the fastest way to answer "what would this value do" without a cluster.

What you can now do

Why it helps

Most Helm incidents are values incidents. Prod dropping to 2 replicas and no profile after someone ran helm upgrade --set image.tag=... by hand: supplying any value discards the previous revision's supplied values. A limit that renders as 1.5e+06: numbers from values files are float64. An image app:1.1 when you wrote tag: 1.10: YAML read a float. A change that "deployed" but did nothing: a wrong key, accepted silently.

When you review a Helm PR or a pipeline, you now check the order of -f files, whether the pipeline passes every file every time, and whether versions are quoted. helm get values --all versus plain get values is how you find out what a release really got. This is also a favourite interview topic: "what is the precedence of Helm values?"

FAQ

What is the order of precedence for Helm values?

From lowest to highest: the chart's values.yaml, then for subcharts the parent's section for them, then -f files left to right (the last wins), then the --set family by kind: --set-json, --set, --set-string, --set-file, --set-literal. Within one kind, left to right. So --set replicaCount=5 beats any file, and -f prod.yaml -f base.yaml lets base override prod.

Why did my list from values.yaml disappear instead of being extended?

Helm merges maps key by key, recursively, but replaces lists entirely. If the chart default is tolerations: [a, b] and your file has tolerations: [c], the result is [c]. There is no append. Well-designed charts use maps keyed by name, like extraEnv: {FOO: bar}, and build the list in the template, so users can add or override single entries.

What happens to previous values on helm upgrade?

With no -f or --set at all, Helm reuses the previous revision's supplied values. With any value flag, only this command's values plus the new chart's defaults count, so a quick --set image.tag=... drops the -f prod.yaml the pipeline passed. --reuse-values merges previous plus new but ignores the new chart's defaults. --reset-then-reuse-values (Helm 3.14+) is new defaults plus previous plus new.

Why is my number rendered as 1.5e+06?

Values from files are decoded through JSON, so every number is a float64, and Go prints large floats in exponent notation: 1500000 becomes 1.5e+06. Small numbers like 3 look fine, so it surfaces later. Fix it in the template with {{ .Values.maxBytes | int }}, or quote the value in the file. Integers passed with --set are int64 and print normally, so the same key can render differently depending on its source.

How do I catch a misspelled values key?

Helm does not know which keys templates read, so --set replica=5 is accepted and does nothing. Add a values.schema.json with types and "additionalProperties": false; Helm validates the merged values on install, upgrade, lint and template, and fails with additional properties 'replica' not allowed. Also compare helm get values with what templates actually use, and render with helm template in CI.

In an interview Mid

How do you manage different configurations for dev, test and prod with Helm, and what are the traps?

One chart, one values file per environment, every value passed by the pipeline every time: helm upgrade --install app ./chart -f values.yaml -f prod.yaml.

The merge rules: the chart's values.yaml < -f files left to right (the last wins) < --set. So the order of -f matters, and a --set in the pipeline beats any file.

The traps:

Check: helm get values (supplied) and --all (computed).

Also asked: Prod lost its settings after someone ran helm upgrade by hand. What happened and how do you prevent it? · What YAML type pitfalls do you watch for in Helm values files? · How do you see the values a release was actually deployed with?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.