Where a setting comes from
A Kafka setting can be set in five places. For a topic setting like retention.ms, the broker looks in this order and takes the first it finds:
- the topic override (
kafka-configs --entity-type topics), stored in the cluster metadata; - a dynamic per-broker setting (
--entity-type brokers --entity-name 1), in the metadata; - a dynamic cluster default (
--entity-type brokers --entity-default), in the metadata; - the static
server.propertiesof the broker, read at start; - Kafka's built-in default.
Topic settings have broker-side twins with other names: retention.ms <- log.retention.ms / log.retention.minutes / log.retention.hours, segment.bytes <- log.segment.bytes, cleanup.policy <- log.cleanup.policy, max.message.bytes <- message.max.bytes, min.insync.replicas <- itself. The broker's value is the default for every topic that has no override.
Dynamic settings (1-3) change without a restart and are the same on every broker that reads the metadata. Static ones (4) need an edit on each node and a (rolling) restart. Only a list of broker settings can be dynamic at all: the log and retention defaults, thread counts, throttles, listener and SSL settings. Everything else is read-only.
kafka-configs.sh --describe
Without --all, describe shows only what is set dynamically for that entity:
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --describe --entity-type topics --entity-name orders
Dynamic configs for topic orders are:
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --describe --entity-type brokers --entity-default
Default configs for brokers in the cluster are:
min.insync.replicas=2 sensitive=false synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:min.insync.replicas=2}
Each line is name=value, sensitive= (passwords print as null), and the synonyms: every place the value could come from that has one, in the order the broker resolves them - DYNAMIC_TOPIC_CONFIG, DYNAMIC_BROKER_CONFIG, DYNAMIC_DEFAULT_BROKER_CONFIG, STATIC_BROKER_CONFIG, DEFAULT_CONFIG. The first entry wins.
--all lists every setting with its effective value; filter it:
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --describe --all --entity-type topics --entity-name orders | grep -E 'retention|segment.bytes|min.insync'
delete.retention.ms=86400000 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.delete.retention.ms=86400000}
local.retention.bytes=-2 sensitive=false synonyms={DEFAULT_CONFIG:log.local.retention.bytes=-2}
local.retention.ms=-2 sensitive=false synonyms={DEFAULT_CONFIG:log.local.retention.ms=-2}
min.insync.replicas=2 sensitive=false synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:min.insync.replicas=2, STATIC_BROKER_CONFIG:min.insync.replicas=2, DEFAULT_CONFIG:min.insync.replicas=1}
retention.bytes=-1 sensitive=false synonyms={DEFAULT_CONFIG:log.retention.bytes=-1}
retention.ms=604800000 sensitive=false synonyms={}
segment.bytes=1073741824 sensitive=false synonyms={STATIC_BROKER_CONFIG:log.segment.bytes=1073741824, DEFAULT_CONFIG:log.segment.bytes=1073741824}
segment.bytes has two synonyms: STATIC_BROKER_CONFIG (the server.properties line) and the built-in default. retention.ms shows no synonyms although log.retention.hours=168 is set statically: the broker only lists the synonyms that start at the topic setting's main broker name (log.retention.ms), and that one is set nowhere. The value, 604800000 ms, is still the 168 hours.
The broker view of a single broker, with --all, shows static and default settings too:
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --describe --all --entity-type brokers --entity-name 1 | grep -E 'log.retention|num.io.threads|auto.create'
auto.create.topics.enable=false sensitive=false synonyms={STATIC_BROKER_CONFIG:auto.create.topics.enable=false, DEFAULT_CONFIG:auto.create.topics.enable=true}
log.retention.bytes=-1 sensitive=false synonyms={DEFAULT_CONFIG:log.retention.bytes=-1}
log.retention.check.interval.ms=60000 sensitive=false synonyms={STATIC_BROKER_CONFIG:log.retention.check.interval.ms=60000, DEFAULT_CONFIG:log.retention.check.interval.ms=300000}
log.retention.hours=168 sensitive=false synonyms={STATIC_BROKER_CONFIG:log.retention.hours=168, DEFAULT_CONFIG:log.retention.hours=168}
num.io.threads=8 sensitive=false synonyms={STATIC_BROKER_CONFIG:num.io.threads=8, DEFAULT_CONFIG:num.io.threads=8}
Changing settings
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --alter --entity-type topics --entity-name payments --add-config retention.ms=2592000000,max.message.bytes=2097152
Completed updating config for topic payments.
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --describe --entity-type topics --entity-name payments
Dynamic configs for topic payments are:
max.message.bytes=2097152 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:max.message.bytes=2097152, DEFAULT_CONFIG:message.max.bytes=1048588}
retention.ms=2592000000 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:retention.ms=2592000000}
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --alter --entity-type topics --entity-name payments --delete-config max.message.bytes
Completed updating config for topic payments.
--add-config takes k=v,k=v; a value that contains commas goes in square brackets (follower.replication.throttled.replicas=[0:1,0:2]). --delete-config removes the override and the setting falls back to the next level. --topic orders is a shortcut for --entity-type topics --entity-name orders; --broker 1 and --broker-defaults work the same way.
Mistakes are caught by the controller:
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --alter --topic payments --add-config retention.msx=1
Error while executing config command with args '--bootstrap-server kafka-1:9092 --alter --topic payments --add-config retention.msx=1'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic config name: retention.msx
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:173)
at kafka.admin.ConfigCommand$.alterResourceConfig(ConfigCommand.scala:417)
at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:195)
at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:165)
at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:86)
at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic config name: retention.msx
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --alter --entity-type brokers --entity-default --add-config log.retention.check.interval.ms=60000
Error while executing config command with args '--bootstrap-server kafka-1:9092 --alter --entity-type brokers --entity-default --add-config log.retention.check.interval.ms=60000'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Cannot update these configs dynamically: [log.retention.check.interval.ms]
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:173)
at kafka.admin.ConfigCommand$.alterResourceConfig(ConfigCommand.scala:417)
at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:195)
at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:165)
at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:86)
at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Cannot update these configs dynamically: [log.retention.check.interval.ms]
The second one: log.retention.check.interval.ms is not dynamic, so it can only change in server.properties with a restart. Cluster-wide defaults that are dynamic apply at once:
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --alter --entity-type brokers --entity-default --add-config log.retention.ms=259200000
Completed updating default config for brokers in the cluster.
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --describe --all --topic notifications | grep 'retention.ms'
delete.retention.ms=86400000 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.delete.retention.ms=86400000}
local.retention.ms=-2 sensitive=false synonyms={DEFAULT_CONFIG:log.local.retention.ms=-2}
retention.ms=259200000 sensitive=false synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.retention.ms=259200000, STATIC_BROKER_CONFIG:log.retention.hours=168, DEFAULT_CONFIG:log.retention.hours=168}
Every topic without its own retention.ms now keeps 3 days - including topics you did not think about. A cluster default is the widest change Kafka has; check what has no override first (kafka-topics.sh --describe --topics-with-overrides lists the topics that have some).
min.insync.replicas and ELR
With ELR enabled (lesson 4), min.insync.replicas is special: the cluster-level value is the one the controller uses, it cannot be removed, and it cannot be set per broker. A per-broker change is refused; change it per topic or as the cluster default. Changing it on a topic clears that topic's ELR state.
In an interview: "Topic override, then dynamic per-broker, then the dynamic cluster default, then server.properties, then Kafka's default. Dynamic configs apply without a restart; read-only ones need an edit on every broker and a rolling restart."