OnCallReadyStart learning

Runtimes and data Lesson 28.5 · Kafka Operations · 16 min read

Topics, partitions and replicas: reading kafka-topics --describe

In plain words

A topic is like a big book split into chapters, and each chapter is copied onto several shelves. One copy is the one people write in (the leader); the others copy every new line from it (the followers). The list of copies that are fully up to date is the ISR. kafka-topics --describe is the library's index card that tells you, for each chapter, where every copy is and which are current.

Topics, partitions, replicas

A topic is a name; the work happens in its partitions. Each partition is an ordered log on disk, and Kafka only guarantees order inside one partition. Partitions are also the unit of parallelism: a consumer group can have at most one active consumer per partition, so a topic with 6 partitions can be read by at most 6 consumers of one group at the same time.

Each partition is replicated to replication factor brokers. One replica is the leader: all produce and fetch requests for that partition go to it. The others are followers: they fetch from the leader, exactly like a consumer, to stay in sync. Different partitions have their leaders on different brokers, so the load is spread.

Reading kafka-topics.sh --describe

$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --topic orders
Topic: orders	TopicId: hbZw5PYaJOpDrtsWAGGWFw	PartitionCount: 6	ReplicationFactor: 3	Configs: min.insync.replicas=2,segment.bytes=1073741824
	Topic: orders	Partition: 0	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1	Elr: 	LastKnownElr:
	Topic: orders	Partition: 1	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2	Elr: 	LastKnownElr:
	Topic: orders	Partition: 2	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3	Elr: 	LastKnownElr:
	Topic: orders	Partition: 3	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1	Elr: 	LastKnownElr:
	Topic: orders	Partition: 4	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2	Elr: 	LastKnownElr:
	Topic: orders	Partition: 5	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3	Elr: 	LastKnownElr:

The first line is the topic:

  • TopicId - a UUID given at creation. A topic deleted and re-created with the same name gets a new one; clients that cached the old id notice.
  • PartitionCount, ReplicationFactor.
  • Configs - only the settings that are not Kafka's built-in defaults. segment.bytes=1073741824 is there because log.segment.bytes is set in the brokers' server.properties (a static broker setting is not "default" even when it has the default value); min.insync.replicas=2 because the cluster has it as a dynamic default (lesson 4 and 9 explain both).

Then one line per partition:

  • Leader - the broker id of the leader, or none when no replica can lead: the partition is offline, nobody can write to it or read from it.
  • Replicas - the assigned brokers. The first one is the preferred leader: when everything is healthy, it should be the leader (lesson 10).
  • Isr - the in-sync replicas, the ones fully caught up with the leader. Healthy = same set as Replicas.
  • Elr / LastKnownElr - the eligible leader replicas (Kafka 4, lesson 4). Empty when healthy.

Creating a topic

$ kafka-topics.sh --bootstrap-server kafka-1:9092 --create --topic invoices --partitions 6 --replication-factor 3
Created topic invoices.
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --topic invoices
Topic: invoices	TopicId: 1QSzIzeUKaFWUhqj3GEJPg	PartitionCount: 6	ReplicationFactor: 3	Configs: min.insync.replicas=2,segment.bytes=1073741824
	Topic: invoices	Partition: 0	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3	Elr: 	LastKnownElr:
	Topic: invoices	Partition: 1	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1	Elr: 	LastKnownElr:
	Topic: invoices	Partition: 2	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2	Elr: 	LastKnownElr:
	Topic: invoices	Partition: 3	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3	Elr: 	LastKnownElr:
	Topic: invoices	Partition: 4	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1	Elr: 	LastKnownElr:
	Topic: invoices	Partition: 5	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2	Elr: 	LastKnownElr:

The controller spreads the replicas round-robin over the brokers, starting at a different broker for each topic, so leaders end up balanced. Without --partitions and --replication-factor the topic gets the cluster's num.partitions and default.replication.factor - both 1 out of the box. In KRaft the controller creates the topic and fills them in, so the lab sets them (3 and 3) in the controllers' server.properties; a broker that has them in its file logs a WARN at start that 5.0 will ignore them there. A replication factor larger than the number of brokers is refused:

$ kafka-topics.sh --bootstrap-server kafka-1:9092 --create --topic big --replication-factor 4
Error while executing topic command : Unable to replicate the partition 4 time(s): The target replication factor of 4 cannot be reached because only 3 broker(s) are registered or some brokers have all their log directories cordoned.
[2026-09-22 20:00:12,800] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Unable to replicate the partition 4 time(s): The target replication factor of 4 cannot be reached because only 3 broker(s) are registered or some brokers have all their log directories cordoned.
 (org.apache.kafka.tools.TopicCommand)

Note what goes where: the "Error while executing topic command" line goes to standard output, the ERROR log line with the exception to standard error. That ERROR line is the tool's log4j2 output (config/tools-log4j2.yaml, WARN and above, on stderr). Exceptions that come back from the brokers carry no stack trace, which is why the closing (org.apache.kafka.tools.TopicCommand) sits alone on the next line.

A name with . or _ gets a warning, because both turn into the same character in metric names:

$ kafka-topics.sh --bootstrap-server kafka-1:9092 --create --topic payment_events.v2 --partitions 3
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic payment_events.v2.

Changing a topic

You can add partitions, never remove them:

$ kafka-topics.sh --bootstrap-server kafka-1:9092 --alter --topic invoices --partitions 8
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --alter --topic invoices --partitions 4
Error while executing topic command : The topic invoices currently has 8 partition(s); 4 would not be an increase.
[2026-09-22 20:00:15,800] ERROR org.apache.kafka.common.errors.InvalidPartitionsException: The topic invoices currently has 8 partition(s); 4 would not be an increase.
 (org.apache.kafka.tools.TopicCommand)

Adding partitions has a cost the help text spells out: records with a key go to murmur2(key) % number-of-partitions, so after the change the same key lands in a different partition, and per-key ordering across the change is gone. Plan the partition count up front. Topic settings (retention and the like) are not changed here but with kafka-configs.sh (lesson 9); --alter --config is refused with a message that says so.

Deleting is instant and prints nothing on success:

$ kafka-topics.sh --bootstrap-server kafka-1:9092 --delete --topic payment_events.v2
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --list
__consumer_offsets
invoices
notifications
orders
payments

There is no undo. In production, auto.create.topics.enable=false (this lab sets it) stops a typo in a producer's config from creating a new topic silently - with the default true, the first write to ordres creates ordres with one partition and one replica.

Everything at once

--describe without --topic describes every topic, including the 50 partitions of __consumer_offsets. --exclude-internal leaves the internal topics out, and four filters only print the partitions that need attention - you will use them in the next lesson:

FlagPrints partitions where
--under-replicated-partitionsISR smaller than the replica set
--under-min-isr-partitionsISR smaller than min.insync.replicas (or no leader)
--at-min-isr-partitionsISR exactly min.insync.replicas: one more failure and writes stop
--unavailable-partitionsno leader, or the leader is not alive
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --under-replicated-partitions

Empty output is the good answer.

In an interview: "Ordering is per partition, parallelism is per partition, and you can only add partitions - which moves keys. RF 3 puts every partition on three brokers; the first replica in the list is the preferred leader."

Why it helps

Nearly every Kafka incident starts with kafka-topics --describe. Reading Leader, Replicas, Isr and Elr correctly tells you whether a broker is missing, whether writes are at risk and whether data could be lost. Creating topics with the right partitions and replication factor avoids problems that are hard to fix later, such as single-copy topics or too few partitions, and it is the first command of most incidents.

Commands in this lesson

kafka-topics.sh

FAQ

What is the preferred leader?

The first broker in a partition's Replicas list. Kafka tries to make it the leader, because the assignment spreads first replicas evenly over the brokers, so leadership and client traffic are balanced. After restarts, leaders can sit elsewhere until a preferred election moves them back.

Why do errors from kafka-topics appear twice?

The tool prints "Error while executing topic command" on standard output and also logs the same error with a timestamp through its logger on standard error. Scripts that only capture one stream see one of them; the exit code is 1 in both cases.

Can I reduce the number of partitions?

No. Partitions can only be added, never removed, and adding them changes which partition a key goes to. The usual way to get fewer partitions is a new topic and a migration of producers and consumers.

What do the four report flags show?

--under-replicated-partitions lists partitions whose ISR is smaller than the replica list, --under-min-isr-partitions those below min.insync.replicas, --at-min-isr-partitions those exactly at it, and --unavailable-partitions those without a leader. No output means none, which is the answer you hope for on a page.

Why is auto.create.topics.enable off in the lab?

With it on, a typo in a producer's topic name silently creates a new topic with default settings. Turning it off makes the producer wait and then fail with "not present in metadata", which is a clear error instead of data going to the wrong place.

In an interview Mid

Read me a line of kafka-topics --describe and tell me if anything is wrong.

Each partition line has Leader (the broker serving it), Replicas (the assignment, first one preferred) and Isr (replicas in sync). If Isr is shorter than Replicas the partition is under-replicated: a broker is down or behind. If the ISR is below min.insync.replicas, acks=all writes fail. Elr lists replicas that may still take over cleanly. --under-replicated-partitions shows only the problems.

Also asked: What is the difference between Replicas and Isr? · Why can you not remove partitions? · What does auto.create.topics.enable do?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.