OnCallReadyStart learning

Runtimes and data Lesson 28.17 · Kafka Operations · 12 min read

kafka-configs: dynamic, static and default settings

In plain words

Kafka settings are like house rules written in several places. The building has general rules (server.properties), the landlord can post a notice for every flat at once (the cluster-wide default), or for one flat only (a per-broker setting), and a tenant can have a special agreement for one room (a topic override). The most specific rule wins, and the others only apply where it says nothing.

Where a setting comes from

A Kafka setting can be set in five places. For a topic setting like retention.ms, the broker looks in this order and takes the first it finds:

  1. the topic override (kafka-configs --entity-type topics), stored in the cluster metadata;
  2. a dynamic per-broker setting (--entity-type brokers --entity-name 1), in the metadata;
  3. a dynamic cluster default (--entity-type brokers --entity-default), in the metadata;
  4. the static server.properties of the broker, read at start;
  5. Kafka's built-in default.

Topic settings have broker-side twins with other names: retention.ms <- log.retention.ms / log.retention.minutes / log.retention.hours, segment.bytes <- log.segment.bytes, cleanup.policy <- log.cleanup.policy, max.message.bytes <- message.max.bytes, min.insync.replicas <- itself. The broker's value is the default for every topic that has no override.

Dynamic settings (1-3) change without a restart and are the same on every broker that reads the metadata. Static ones (4) need an edit on each node and a (rolling) restart. Only a list of broker settings can be dynamic at all: the log and retention defaults, thread counts, throttles, listener and SSL settings. Everything else is read-only.

kafka-configs.sh --describe

Without --all, describe shows only what is set dynamically for that entity:

$ kafka-configs.sh --bootstrap-server kafka-1:9092 --describe --entity-type topics --entity-name orders
Dynamic configs for topic orders are:
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --describe --entity-type brokers --entity-default
Default configs for brokers in the cluster are:
  min.insync.replicas=2 sensitive=false synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:min.insync.replicas=2}

Each line is name=value, sensitive= (passwords print as null), and the synonyms: every place the value could come from that has one, in the order the broker resolves them - DYNAMIC_TOPIC_CONFIG, DYNAMIC_BROKER_CONFIG, DYNAMIC_DEFAULT_BROKER_CONFIG, STATIC_BROKER_CONFIG, DEFAULT_CONFIG. The first entry wins.

--all lists every setting with its effective value; filter it:

$ kafka-configs.sh --bootstrap-server kafka-1:9092 --describe --all --entity-type topics --entity-name orders | grep -E 'retention|segment.bytes|min.insync'
  delete.retention.ms=86400000 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.delete.retention.ms=86400000}
  local.retention.bytes=-2 sensitive=false synonyms={DEFAULT_CONFIG:log.local.retention.bytes=-2}
  local.retention.ms=-2 sensitive=false synonyms={DEFAULT_CONFIG:log.local.retention.ms=-2}
  min.insync.replicas=2 sensitive=false synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:min.insync.replicas=2, STATIC_BROKER_CONFIG:min.insync.replicas=2, DEFAULT_CONFIG:min.insync.replicas=1}
  retention.bytes=-1 sensitive=false synonyms={DEFAULT_CONFIG:log.retention.bytes=-1}
  retention.ms=604800000 sensitive=false synonyms={}
  segment.bytes=1073741824 sensitive=false synonyms={STATIC_BROKER_CONFIG:log.segment.bytes=1073741824, DEFAULT_CONFIG:log.segment.bytes=1073741824}

segment.bytes has two synonyms: STATIC_BROKER_CONFIG (the server.properties line) and the built-in default. retention.ms shows no synonyms although log.retention.hours=168 is set statically: the broker only lists the synonyms that start at the topic setting's main broker name (log.retention.ms), and that one is set nowhere. The value, 604800000 ms, is still the 168 hours.

The broker view of a single broker, with --all, shows static and default settings too:

$ kafka-configs.sh --bootstrap-server kafka-1:9092 --describe --all --entity-type brokers --entity-name 1 | grep -E 'log.retention|num.io.threads|auto.create'
  auto.create.topics.enable=false sensitive=false synonyms={STATIC_BROKER_CONFIG:auto.create.topics.enable=false, DEFAULT_CONFIG:auto.create.topics.enable=true}
  log.retention.bytes=-1 sensitive=false synonyms={DEFAULT_CONFIG:log.retention.bytes=-1}
  log.retention.check.interval.ms=60000 sensitive=false synonyms={STATIC_BROKER_CONFIG:log.retention.check.interval.ms=60000, DEFAULT_CONFIG:log.retention.check.interval.ms=300000}
  log.retention.hours=168 sensitive=false synonyms={STATIC_BROKER_CONFIG:log.retention.hours=168, DEFAULT_CONFIG:log.retention.hours=168}
  num.io.threads=8 sensitive=false synonyms={STATIC_BROKER_CONFIG:num.io.threads=8, DEFAULT_CONFIG:num.io.threads=8}

Changing settings

$ kafka-configs.sh --bootstrap-server kafka-1:9092 --alter --entity-type topics --entity-name payments --add-config retention.ms=2592000000,max.message.bytes=2097152
Completed updating config for topic payments.
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --describe --entity-type topics --entity-name payments
Dynamic configs for topic payments are:
  max.message.bytes=2097152 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:max.message.bytes=2097152, DEFAULT_CONFIG:message.max.bytes=1048588}
  retention.ms=2592000000 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:retention.ms=2592000000}
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --alter --entity-type topics --entity-name payments --delete-config max.message.bytes
Completed updating config for topic payments.

--add-config takes k=v,k=v; a value that contains commas goes in square brackets (follower.replication.throttled.replicas=[0:1,0:2]). --delete-config removes the override and the setting falls back to the next level. --topic orders is a shortcut for --entity-type topics --entity-name orders; --broker 1 and --broker-defaults work the same way.

Mistakes are caught by the controller:

$ kafka-configs.sh --bootstrap-server kafka-1:9092 --alter --topic payments --add-config retention.msx=1
Error while executing config command with args '--bootstrap-server kafka-1:9092 --alter --topic payments --add-config retention.msx=1'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic config name: retention.msx
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:173)
	at kafka.admin.ConfigCommand$.alterResourceConfig(ConfigCommand.scala:417)
	at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:195)
	at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:165)
	at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:86)
	at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic config name: retention.msx
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --alter --entity-type brokers --entity-default --add-config log.retention.check.interval.ms=60000
Error while executing config command with args '--bootstrap-server kafka-1:9092 --alter --entity-type brokers --entity-default --add-config log.retention.check.interval.ms=60000'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Cannot update these configs dynamically: [log.retention.check.interval.ms]
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:173)
	at kafka.admin.ConfigCommand$.alterResourceConfig(ConfigCommand.scala:417)
	at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:195)
	at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:165)
	at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:86)
	at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Cannot update these configs dynamically: [log.retention.check.interval.ms]

The second one: log.retention.check.interval.ms is not dynamic, so it can only change in server.properties with a restart. Cluster-wide defaults that are dynamic apply at once:

$ kafka-configs.sh --bootstrap-server kafka-1:9092 --alter --entity-type brokers --entity-default --add-config log.retention.ms=259200000
Completed updating default config for brokers in the cluster.
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --describe --all --topic notifications | grep 'retention.ms'
  delete.retention.ms=86400000 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.delete.retention.ms=86400000}
  local.retention.ms=-2 sensitive=false synonyms={DEFAULT_CONFIG:log.local.retention.ms=-2}
  retention.ms=259200000 sensitive=false synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.retention.ms=259200000, STATIC_BROKER_CONFIG:log.retention.hours=168, DEFAULT_CONFIG:log.retention.hours=168}

Every topic without its own retention.ms now keeps 3 days - including topics you did not think about. A cluster default is the widest change Kafka has; check what has no override first (kafka-topics.sh --describe --topics-with-overrides lists the topics that have some).

min.insync.replicas and ELR

With ELR enabled (lesson 4), min.insync.replicas is special: the cluster-level value is the one the controller uses, it cannot be removed, and it cannot be set per broker. A per-broker change is refused; change it per topic or as the cluster default. Changing it on a topic clears that topic's ELR state.

In an interview: "Topic override, then dynamic per-broker, then the dynamic cluster default, then server.properties, then Kafka's default. Dynamic configs apply without a restart; read-only ones need an edit on every broker and a rolling restart."

Why it helps

Most Kafka changes are config changes, and putting one at the wrong level either has no effect or changes far more than intended. kafka-configs shows where each value comes from through its synonyms, which is how you find a forgotten override. Knowing which configs are read-only saves you from trying to change log.dirs without a restart, and why a changed value seems to be ignored.

Commands in this lesson

kafka-configs.sh

FAQ

What is the difference between a static and a dynamic broker config?

A static config is in server.properties and changes need a restart of that broker. A dynamic config is set with kafka-configs, stored in the cluster metadata and applied without a restart, also to brokers that join later.

What does --entity-default do?

With --entity-type brokers it sets a cluster-wide default for every broker. It beats server.properties but loses to a per-broker value set with --entity-name N for that broker. It also applies to brokers that join the cluster later.

How do I find where a value comes from?

kafka-configs --describe --all lists every config with its source and synonyms: DYNAMIC_TOPIC_CONFIG, DYNAMIC_BROKER_CONFIG, DYNAMIC_DEFAULT_BROKER_CONFIG, STATIC_BROKER_CONFIG or DEFAULT_CONFIG. The first synonym in the list is the one in effect, and the others show what would apply without it.

Why are topic and broker config names different?

Many topic configs have a broker-level default under another name: max.message.bytes on a topic, message.max.bytes on the broker; retention.ms on a topic, log.retention.ms (or hours) on the broker. Use the topic name with --entity-type topics.

Why can I not change min.insync.replicas per broker?

With ELR enabled (the default for new clusters since 4.1), Kafka allows min.insync.replicas only as a cluster-wide default or a topic override; a per-broker value would make the eligible leader replicas inconsistent, so it is refused with an INVALID_CONFIG error.

In an interview Mid

How do you change a broker setting without restarting the broker?

With kafka-configs --alter --entity-type brokers, either --entity-default for every broker or --entity-name N for one, if the config is dynamically updatable; the value is stored in the metadata log and applies at once. Topic settings go on the topic with --entity-type topics. --describe --all shows each value's source; read-only configs like log.dirs need server.properties and a restart.

Also asked: What is the order of precedence for Kafka configs? · How do you remove a topic override? · What does a read-only broker config mean?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.