OnCallReadyStart learning

Runtimes and data Lesson 28.19 · Kafka Operations · 16 min read

Moving partitions: reassignment, throttles and preferred leaders

In plain words

Adding a new shelf to a library does not move any books onto it. Somebody has to plan which books go where, copy them across while the library stays open - slowly enough that readers are not disturbed - and only then take the old copies away. That plan and the careful copying is a partition reassignment, and the plan you keep to undo it is the rollback file.

A new broker gets nothing

When a broker joins the cluster, Kafka does not move any existing partition to it. New topics will use it; old ones stay where they are until you move them. The same goes for draining a broker before you retire it. Moving replicas is a partition reassignment, and the tool is kafka-reassign-partitions.sh.

This lab now has a fourth broker, kafka-4:

$ kafka-cluster.sh list-endpoints --bootstrap-server kafka-1:9092
ID         HOST    PORT       RACK       STATE      ENDPOINT_TYPE
1          kafka-1 9092       null       unfenced   broker
2          kafka-2 9092       null       unfenced   broker
3          kafka-3 9092       null       unfenced   broker
4          kafka-4 9092       null       unfenced   broker
$ kafka-log-dirs.sh --bootstrap-server kafka-1:9092 --describe --broker-list 4
Querying brokers for log directories information
Received log directory information from brokers 4
{"brokers":[{"broker":4,"logDirs":[{"partitions":[],"error":null,"logDir":"/var/lib/kafka/data"}]}],"version":1}

Registered, unfenced - and empty.

1. Generate a plan

--generate takes a JSON list of topics and the target brokers, and proposes an even assignment:

$ echo '{"version":1,"topics":[{"topic":"orders"}]}' > /tmp/move.json
$ kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 --topics-to-move-json-file /tmp/move.json --broker-list 1,2,3,4 --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"orders","partition":0,"replicas":[2,3,1],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":1,"replicas":[3,1,2],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":2,"replicas":[1,2,3],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":3,"replicas":[2,3,1],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":4,"replicas":[3,1,2],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":5,"replicas":[1,2,3],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"orders","partition":0,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"orders","partition":1,"replicas":[3,2,4],"log_dirs":["any","any","any"]},{"topic":"orders","partition":2,"replicas":[4,3,1],"log_dirs":["any","any","any"]},{"topic":"orders","partition":3,"replicas":[1,4,2],"log_dirs":["any","any","any"]},{"topic":"orders","partition":4,"replicas":[2,3,4],"log_dirs":["any","any","any"]},{"topic":"orders","partition":5,"replicas":[3,4,1],"log_dirs":["any","any","any"]}]}

It prints the current assignment and a proposal. Nothing has changed yet. The proposal spreads replicas round-robin from a random starting broker, so it can move more replicas than needed; for a few partitions, writing the JSON by hand is often better. Save both parts:

$ kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 --topics-to-move-json-file /tmp/move.json --broker-list 1,2,3,4 --generate | sed -n 5p > /tmp/plan.json
$ cat /tmp/plan.json
{"version":1,"partitions":[{"topic":"orders","partition":0,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"orders","partition":1,"replicas":[3,2,4],"log_dirs":["any","any","any"]},{"topic":"orders","partition":2,"replicas":[4,3,1],"log_dirs":["any","any","any"]},{"topic":"orders","partition":3,"replicas":[1,4,2],"log_dirs":["any","any","any"]},{"topic":"orders","partition":4,"replicas":[2,3,4],"log_dirs":["any","any","any"]},{"topic":"orders","partition":5,"replicas":[3,4,1],"log_dirs":["any","any","any"]}]}

2. Execute, throttled

Moving a replica means copying the whole partition over the network, on brokers that are serving production traffic. --throttle (bytes per second) caps the replication traffic of the move:

$ kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 --reassignment-json-file /tmp/plan.json --execute --throttle 20000000
Current partition replica assignment

{"version":1,"partitions":[{"topic":"orders","partition":0,"replicas":[2,3,1],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":1,"replicas":[3,1,2],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":2,"replicas":[1,2,3],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":3,"replicas":[2,3,1],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":4,"replicas":[3,1,2],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":5,"replicas":[1,2,3],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]}]}

Save this to use as the --reassignment-json-file option during rollback
Warning: You must run --verify periodically, until the reassignment completes, to ensure the throttle is removed.
The inter-broker throttle limit was set to 20000000 B/s
Successfully started partition reassignments for orders-0,orders-1,orders-2,orders-3,orders-4,orders-5

The tool prints the current assignment first: copy it, it is your rollback plan. Then it sets the throttle - two broker settings on every broker involved, leader.replication.throttled.rate and follower.replication.throttled.rate, plus two topic settings listing which replicas are throttled - and starts the moves.

While a move runs, the partition has more replicas than its target: the new ones are adding, the old ones removing, and the describe shows both:

$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --topic orders
Topic: orders	TopicId: hbZw5PYaJOpDrtsWAGGWFw	PartitionCount: 6	ReplicationFactor: 3	Configs: leader.replication.throttled.replicas=0:1,0:2,0:3,1:1,1:2,1:3,2:1,2:2,2:3,3:1,3:2,3:3,4:1,4:2,4:3,5:1,5:2,5:3,min.insync.replicas=2,follower.replication.throttled.replicas=1:4,2:4,3:4,4:4,5:4,segment.bytes=1073741824
	Topic: orders	Partition: 0	Leader: 2	Replicas: 2,1,3	Isr: 2,3,1	Elr: 	LastKnownElr:
	Topic: orders	Partition: 1	Leader: 3	Replicas: 3,2,4,1	Isr: 3,1,2	Adding Replicas: 4	Removing Replicas: 1	Elr: 	LastKnownElr:
	Topic: orders	Partition: 2	Leader: 1	Replicas: 4,3,1,2	Isr: 1,2,3	Adding Replicas: 4	Removing Replicas: 2	Elr: 	LastKnownElr:
	Topic: orders	Partition: 3	Leader: 2	Replicas: 1,4,2,3	Isr: 2,3,1	Adding Replicas: 4	Removing Replicas: 3	Elr: 	LastKnownElr:
	Topic: orders	Partition: 4	Leader: 3	Replicas: 2,3,4,1	Isr: 3,1,2	Adding Replicas: 4	Removing Replicas: 1	Elr: 	LastKnownElr:
	Topic: orders	Partition: 5	Leader: 1	Replicas: 3,4,1,2	Isr: 1,2,3	Adding Replicas: 4	Removing Replicas: 2	Elr: 	LastKnownElr:
$ kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 --list
Current partition reassignments:
orders-1: replicas: 3,2,4,1. adding: 4. removing: 1.
orders-2: replicas: 4,3,1,2. adding: 4. removing: 2.
orders-3: replicas: 1,4,2,3. adding: 4. removing: 3.
orders-4: replicas: 2,3,4,1. adding: 4. removing: 1.
orders-5: replicas: 3,4,1,2. adding: 4. removing: 2.

A new replica joins the ISR when it has caught up; then the controller switches the replica set to the target, drops the removed replicas (their directories are renamed ...-delete and deleted a minute later) and, if the leader was removed, elects a new one.

3. Verify - and remove the throttle

$ kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 --reassignment-json-file /tmp/plan.json --verify
Status of partition reassignment:
Reassignment of partition orders-0 is completed.
Reassignment of partition orders-1 is still in progress.
Reassignment of partition orders-2 is still in progress.
Reassignment of partition orders-3 is still in progress.
Reassignment of partition orders-4 is still in progress.
Reassignment of partition orders-5 is still in progress.

Still in progress: kafka-4 copies five partitions at once, and the 20 MB/s throttle is shared by all of them (a throttle rate is per broker, not per partition). About 3 GB at 20 MB/s is a few minutes. Later:

$ kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 --reassignment-json-file /tmp/plan.json --verify
Status of partition reassignment:
Reassignment of partition orders-0 is completed.
Reassignment of partition orders-1 is completed.
Reassignment of partition orders-2 is completed.
Reassignment of partition orders-3 is completed.
Reassignment of partition orders-4 is completed.
Reassignment of partition orders-5 is completed.

Clearing broker-level throttles on brokers 1,2,3,4
Clearing topic-level throttles on topic orders

--verify is not optional: when everything is complete, it removes the throttles. Forget it and the throttle stays on, and some day a broker that has to catch up after a restart does so at 20 MB/s. Check:

$ kafka-configs.sh --bootstrap-server kafka-1:9092 --describe --entity-type brokers --entity-name 4
Dynamic configs for broker 4 are:

To change the throttle of a running move, re-run --execute with the same file, a new --throttle and --additional. --cancel with the same file stops the moves that have not finished and restores the original replica sets.

Preferred leaders

The first replica in a partition's list is its preferred leader. After failovers, restarts and moves, leadership drifts: one broker leads too many partitions and carries most of the traffic. The controller corrects it on its own: every leader.imbalance.check.interval.seconds (300) it checks each broker, and when more than leader.imbalance.per.broker.percentage (10 %) of the partitions it prefers are led by someone else, it moves leadership back (auto.leader.rebalance.enable=true). Some teams turn that off and do it by hand, at a quiet moment:

$ kafka-leader-election.sh --bootstrap-server kafka-1:9092 --election-type preferred --all-topic-partitions
Successfully completed leader election (PREFERRED) for partitions orders-2, orders-3, orders-4, orders-5
$ kafka-leader-election.sh --bootstrap-server kafka-1:9092 --election-type preferred --topic orders --partition 0
Valid replica already elected for partitions orders-0

With --all-topic-partitions, only the partitions that changed are listed; with one partition named, a leader that is already right prints "Valid replica already elected". A preferred replica that is not in the ISR cannot be elected (PreferredLeaderNotAvailableException): wait until it has caught up.

In an interview: "New brokers get no data until you reassign. Generate or write the plan, keep the rollback JSON, execute with a throttle, and run --verify until it is complete, because --verify removes the throttle. Preferred leader election moves leadership back to the first replica."

Why it helps

Adding brokers, replacing disks and balancing load all need reassignments, and an unthrottled one can starve production traffic. Forgetting --verify leaves throttles behind that slow down later recoveries. Preferred leader elections fix the uneven load after restarts. These are routine maintenance tasks that go wrong in predictable ways, so a fixed procedure for them saves a lot of trouble and late nights.

Commands in this lesson

kafka-cluster.sh kafka-log-dirs.sh echo kafka-reassign-partitions.sh cat kafka-topics.sh kafka-configs.sh kafka-leader-election.sh

FAQ

Why is a new broker empty?

Kafka never moves existing partitions by itself. A new broker only receives replicas of topics created after it joined, or of partitions you reassign to it with kafka-reassign-partitions. Until then it is registered and healthy but serves nothing.

What does --generate do?

It proposes a new assignment for the topics in a JSON file over the brokers you list, and prints the current assignment as well. It changes nothing; you save the current assignment as the rollback file and the proposal as the plan.

How does --throttle work?

It sets leader and follower throttled-replica configs on the moving partitions and a rate on the brokers involved. The rate is per broker: all throttled replicas copying on one broker share it, so a move of many partitions takes longer than one.

Why is --verify required?

--verify reports whether each partition is done and, once all are complete, removes the throttle configs. If nobody runs it, the throttle stays and a later catch-up after a broker restart is slowed to the same rate.

What does kafka-leader-election --election-type preferred do?

It moves leadership of each partition to its preferred replica, the first in the replica list, when that replica is in the ISR. It answers "Valid replica already elected" for partitions that are already led by it.

In an interview Mid

How do you move data onto a new Kafka broker without hurting production?

kafka-reassign-partitions --generate with the topics and the new broker list gives a proposal and the current assignment; save the current one as rollback JSON. --execute with --throttle starts the copy at a bounded rate, --describe shows Adding Replicas and Removing Replicas, and --verify until every partition is completed, which also removes the throttles. Then check URP is 0 and run a preferred leader election if needed.

Also asked: Why does a new broker get no data? · What happens if you never run --verify? · What is the preferred leader?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.