OnCallReady

Lesson 21.5 · Spring Boot Runtime, Resilience & Python Ops · 14 min read

Externalised configuration and its precedence

In plain words

Imagine your mum packs your school bag with default things: a sandwich, water, a pencil. Then your dad adds a note: "take the umbrella, not the cap". Then your teacher says on the phone: "tomorrow bring the red book, not the blue one". The instruction closest to the moment, from the most specific person, wins.

Spring Boot reads settings the same way from many places, called property sources, in a fixed order. application.properties inside the jar is mum's packing; a file outside the jar overrides it; environment variables override every file; -D system properties beat environment variables; command-line arguments beat everything. That is why a Deployment can change SPRING_DATASOURCE_PASSWORD without a new image, and why /actuator/env/<name> shows which source won.

One property, many places

The problem. The config file says the pool size is 20, the app uses 10, and nobody knows why. A Spring Boot app can get the same setting from a dozen places; knowing which one wins turns a guessing game into one curl.

What you need to know already: Actuator and /actuator/env (21.1), env vars in systemd units and drop-ins (2.3, 2.21), env vars and Secrets in pods (15.29, 15.33), the JVM's -D flags and jcmd (20.2, 20.3).

A Spring Boot app reads configuration from many property sources and, for each property, uses the value from the highest-priority source that has it. The real order, highest first (Spring Boot reference, "Externalized Configuration"):

 1. Devtools global settings (dev machines only)
 2. @TestPropertySource / test properties                          tests only
 3. Command line arguments                --server.port=9090
 4. SPRING_APPLICATION_JSON               inline JSON in an env var or -D
 5. ServletConfig / ServletContext init params
 6. JNDI attributes
 7. Java System properties                -Dserver.port=9090
 8. OS environment variables              SERVER_PORT=9090
 9. random.* values
10. Config data files, most specific first:
      application-{profile}.properties OUTSIDE the jar  (./config/, ./, spring.config.location)
      application.properties           OUTSIDE the jar
      application-{profile}.properties INSIDE the jar   (classpath)
      application.properties           INSIDE the jar
11. @PropertySource on @Configuration classes
12. Default properties (SpringApplication.setDefaultProperties)

Things worth noticing:

Relaxed binding: from env var to property

Relaxed binding = Boot accepting several spellings of the same property. Environment variable names cannot contain dots or dashes, so Boot maps them: replace dots with underscores, remove dashes, uppercase:

spring.datasource.password               SPRING_DATASOURCE_PASSWORD
spring.datasource.hikari.maximum-pool-size   SPRING_DATASOURCE_HIKARI_MAXIMUMPOOLSIZE
server.port                              SERVER_PORT
spring.config.additional-location        SPRING_CONFIG_ADDITIONALLOCATION
my.list[0]                               MY_LIST_0_

The Notion question - where does SPRING_DATASOURCE_PASSWORD sit? - OS environment, position 8: it beats every properties/YAML file (inside or outside the jar) and loses to -D system properties and command-line arguments. In Kubernetes it usually comes from a Secret via valueFrom.secretKeyRef.

A common trap: SPRING_CONFIG_ADDITIONAL_LOCATION (with the underscore in the middle) is not spring.config.additional-location - the dash is removed, not turned into an underscore. The variable is silently ignored.

Where the files are looked for

Without any configuration, Boot searches:

optional:classpath:/            optional:classpath:/config/
optional:file:./                optional:file:./config/          optional:file:./config/*/

./ is the working directory of the process - WorkingDirectory=/opt/app for our services, not the directory the jar is in. Two properties change the search:

spring.config.location=...             REPLACES the defaults
spring.config.additional-location=...  ADDS to them (and wins over them)

Locations are comma-separated and can be files or directories (directories end in /). optional: means "fine if missing"; without it a missing location is a startup failure:

***************************
APPLICATION FAILED TO START
***************************

Description:

Config data resource 'file [/etc/orders/app.conf]' via location 'file:/etc/orders/app.conf' does not exist

Action:

Check that the value 'file:/etc/orders/app.conf' is correct, or prefix it with 'optional:'

The file extension decides the parser, so app.conf needs a hint:

file:/etc/orders/app.conf                  -> IllegalStateException: File extension of config file location
                                              'file:/etc/orders/app.conf' is not known to any PropertySourceLoader...
file:/etc/orders/app.conf[.properties]     -> read as a .properties file

The orders service on this box is started with SPRING_CONFIG_ADDITIONALLOCATION=optional:file:/etc/orders/app.conf[.properties] from a systemd drop-in - that is how /etc/orders/app.conf reaches it.

Placeholders

A placeholder ${name:default} inside a value means "the value of property name, or default if nothing sets it":

spring.datasource.hikari.maximum-pool-size=${db.pool.max:10}

The value of db.pool.max, or 10. Resolution happens across all sources, so an env var DB_POOL_MAX=30 changes the pool size even though nobody set the Hikari property directly. (HikariCP - "Hikari" - is the database connection pool Spring Boot uses: a fixed set of open database connections the app borrows and returns, lesson 21.15.)

Profiles

spring.profiles.active=prod           in a file, or SPRING_PROFILES_ACTIVE=prod
application-prod.properties           loaded on top of application.properties

Do not bake the environment into the image as a profile file per environment unless you have to - environment variables per Deployment are simpler to reason about.

Debugging "why does it have that value?"

  1. /actuator/env/<property> - every source that defines it, in precedence order, with an origin (file and line, env var name). The first one listed with a value wins. Values are masked; origins are not.
  2. systemctl show -p Environment <unit> or kubectl exec ... -- env - what the process actually got.
  3. jcmd PID VM.system_properties / VM.command_line - the -D and the arguments.
  4. For bound values: /actuator/configprops or the effective metric (hikaricp_connections_max is the pool size Hikari really uses, 21.11).

What you can now do

Why it helps

"Why does production use the wrong pool size?" is a ticket you will get. The answer is almost always precedence: an old -D in the entrypoint beating the env var you set, a profile file in the image, or an env var with the wrong name that Spring silently ignores. Knowing the order and relaxed binding rules lets you find it in minutes with /actuator/env.

It also shapes how you design config on the platform: one image, defaults inside it, and per-environment values injected as environment variables from ConfigMaps and Secrets. That is the pattern your Deployment templates will implement, and knowing where SPRING_DATASOURCE_PASSWORD sits in the order is a common interview question.

FAQ

How does an environment variable map to a Spring property?

Relaxed binding: replace dots with underscores, remove dashes, uppercase. spring.datasource.password becomes SPRING_DATASOURCE_PASSWORD, spring.datasource.hikari.maximum-pool-size becomes SPRING_DATASOURCE_HIKARI_MAXIMUMPOOLSIZE, and list index my.list[0] becomes MY_LIST_0_. The trap is dashes: SPRING_CONFIG_ADDITIONAL_LOCATION does not map to spring.config.additional-location, and Spring ignores it silently.

Do -D system properties beat environment variables?

Yes. Java system properties are higher in the order than OS environment variables, and command-line arguments like --server.port=9090 are higher still. This surprises people who set an env var in the Deployment and see no effect, because the image's entrypoint or JAVA_TOOL_OPTIONS contains a -D for the same property. jcmd PID VM.system_properties or /actuator/env/<name> shows which source is winning.

What does the optional: prefix do?

It makes a config location allowed to be missing. Without it, a location that does not exist is a startup failure: APPLICATION FAILED TO START with Config data resource ... does not exist. With optional:file:/etc/orders/app.conf, Boot starts without the file. The default search locations are all optional. Use it when a file is truly optional; leave it off when a missing file should stop the deployment.

Should I use profiles per environment?

Profiles work, but baking application-prod.properties and application-dev.properties into the image means config changes need a new image and every environment's settings live in one artifact. The simpler platform pattern is one image with sensible defaults, and per-environment values as environment variables from ConfigMaps and Secrets. Profiles remain useful for feature-style differences, like a kubernetes profile, rather than one per environment.

What is the difference between spring.config.location and additional-location?

spring.config.location replaces the default search locations entirely, so the application.properties next to the working directory or on the classpath may no longer be read the way you expect. spring.config.additional-location adds locations to the defaults, and those added locations win over the defaults. For injecting an extra file like /etc/orders/app.conf, additional-location is almost always what you want.

In an interview Mid

A setting you changed has no effect on the running Spring Boot app. How do you troubleshoot it?

Spring Boot reads each property from many property sources and the highest one that has it wins: command-line arguments > -D system properties > environment variables > config files outside the jar > files inside the jar (profile-specific beat plain).

  1. Which source won? curl localhost:8080/actuator/env/<property> lists every source that defines it, in precedence order, with its origin. Values are masked; origins are not.
  2. Did the process get my change? systemctl show -p Environment <unit> or kubectl exec ... -- env; jcmd PID VM.command_line / VM.system_properties for -D and arguments. Env and files are read at startup - was it restarted?
  3. Is the name right? Relaxed binding: dots to underscores, dashes removed, uppercase. SPRING_CONFIG_ADDITIONAL_LOCATION is silently ignored; it is SPRING_CONFIG_ADDITIONALLOCATION.
  4. What is really used? A placeholder like ${db.pool.max:10} may read a different key; /actuator/configprops or hikaricp_connections_max shows the bound value.

Also asked: How does Spring Boot decide a property's value when it is set in several places? · How do you override a property per environment without rebuilding the image? · What is a Spring profile and when would you use one?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.