A new broker gets nothing
When a broker joins the cluster, Kafka does not move any existing partition to it. New topics will use it; old ones stay where they are until you move them. The same goes for draining a broker before you retire it. Moving replicas is a partition reassignment, and the tool is kafka-reassign-partitions.sh.
This lab now has a fourth broker, kafka-4:
$ kafka-cluster.sh list-endpoints --bootstrap-server kafka-1:9092
ID HOST PORT RACK STATE ENDPOINT_TYPE
1 kafka-1 9092 null unfenced broker
2 kafka-2 9092 null unfenced broker
3 kafka-3 9092 null unfenced broker
4 kafka-4 9092 null unfenced broker
$ kafka-log-dirs.sh --bootstrap-server kafka-1:9092 --describe --broker-list 4
Querying brokers for log directories information
Received log directory information from brokers 4
{"brokers":[{"broker":4,"logDirs":[{"partitions":[],"error":null,"logDir":"/var/lib/kafka/data"}]}],"version":1}
Registered, unfenced - and empty.
1. Generate a plan
--generate takes a JSON list of topics and the target brokers, and proposes an even assignment:
$ echo '{"version":1,"topics":[{"topic":"orders"}]}' > /tmp/move.json
$ kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 --topics-to-move-json-file /tmp/move.json --broker-list 1,2,3,4 --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"orders","partition":0,"replicas":[2,3,1],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":1,"replicas":[3,1,2],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":2,"replicas":[1,2,3],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":3,"replicas":[2,3,1],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":4,"replicas":[3,1,2],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":5,"replicas":[1,2,3],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"orders","partition":0,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"orders","partition":1,"replicas":[3,2,4],"log_dirs":["any","any","any"]},{"topic":"orders","partition":2,"replicas":[4,3,1],"log_dirs":["any","any","any"]},{"topic":"orders","partition":3,"replicas":[1,4,2],"log_dirs":["any","any","any"]},{"topic":"orders","partition":4,"replicas":[2,3,4],"log_dirs":["any","any","any"]},{"topic":"orders","partition":5,"replicas":[3,4,1],"log_dirs":["any","any","any"]}]}
It prints the current assignment and a proposal. Nothing has changed yet. The proposal spreads replicas round-robin from a random starting broker, so it can move more replicas than needed; for a few partitions, writing the JSON by hand is often better. Save both parts:
$ kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 --topics-to-move-json-file /tmp/move.json --broker-list 1,2,3,4 --generate | sed -n 5p > /tmp/plan.json
$ cat /tmp/plan.json
{"version":1,"partitions":[{"topic":"orders","partition":0,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"orders","partition":1,"replicas":[3,2,4],"log_dirs":["any","any","any"]},{"topic":"orders","partition":2,"replicas":[4,3,1],"log_dirs":["any","any","any"]},{"topic":"orders","partition":3,"replicas":[1,4,2],"log_dirs":["any","any","any"]},{"topic":"orders","partition":4,"replicas":[2,3,4],"log_dirs":["any","any","any"]},{"topic":"orders","partition":5,"replicas":[3,4,1],"log_dirs":["any","any","any"]}]}
2. Execute, throttled
Moving a replica means copying the whole partition over the network, on brokers that are serving production traffic. --throttle (bytes per second) caps the replication traffic of the move:
$ kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 --reassignment-json-file /tmp/plan.json --execute --throttle 20000000
Current partition replica assignment
{"version":1,"partitions":[{"topic":"orders","partition":0,"replicas":[2,3,1],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":1,"replicas":[3,1,2],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":2,"replicas":[1,2,3],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":3,"replicas":[2,3,1],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":4,"replicas":[3,1,2],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]},{"topic":"orders","partition":5,"replicas":[1,2,3],"log_dirs":["/var/lib/kafka/data","/var/lib/kafka/data","/var/lib/kafka/data"]}]}
Save this to use as the --reassignment-json-file option during rollback
Warning: You must run --verify periodically, until the reassignment completes, to ensure the throttle is removed.
The inter-broker throttle limit was set to 20000000 B/s
Successfully started partition reassignments for orders-0,orders-1,orders-2,orders-3,orders-4,orders-5
The tool prints the current assignment first: copy it, it is your rollback plan. Then it sets the throttle - two broker settings on every broker involved, leader.replication.throttled.rate and follower.replication.throttled.rate, plus two topic settings listing which replicas are throttled - and starts the moves.
While a move runs, the partition has more replicas than its target: the new ones are adding, the old ones removing, and the describe shows both:
$ kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --topic orders
Topic: orders TopicId: hbZw5PYaJOpDrtsWAGGWFw PartitionCount: 6 ReplicationFactor: 3 Configs: leader.replication.throttled.replicas=0:1,0:2,0:3,1:1,1:2,1:3,2:1,2:2,2:3,3:1,3:2,3:3,4:1,4:2,4:3,5:1,5:2,5:3,min.insync.replicas=2,follower.replication.throttled.replicas=1:4,2:4,3:4,4:4,5:4,segment.bytes=1073741824
Topic: orders Partition: 0 Leader: 2 Replicas: 2,1,3 Isr: 2,3,1 Elr: LastKnownElr:
Topic: orders Partition: 1 Leader: 3 Replicas: 3,2,4,1 Isr: 3,1,2 Adding Replicas: 4 Removing Replicas: 1 Elr: LastKnownElr:
Topic: orders Partition: 2 Leader: 1 Replicas: 4,3,1,2 Isr: 1,2,3 Adding Replicas: 4 Removing Replicas: 2 Elr: LastKnownElr:
Topic: orders Partition: 3 Leader: 2 Replicas: 1,4,2,3 Isr: 2,3,1 Adding Replicas: 4 Removing Replicas: 3 Elr: LastKnownElr:
Topic: orders Partition: 4 Leader: 3 Replicas: 2,3,4,1 Isr: 3,1,2 Adding Replicas: 4 Removing Replicas: 1 Elr: LastKnownElr:
Topic: orders Partition: 5 Leader: 1 Replicas: 3,4,1,2 Isr: 1,2,3 Adding Replicas: 4 Removing Replicas: 2 Elr: LastKnownElr:
$ kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 --list
Current partition reassignments:
orders-1: replicas: 3,2,4,1. adding: 4. removing: 1.
orders-2: replicas: 4,3,1,2. adding: 4. removing: 2.
orders-3: replicas: 1,4,2,3. adding: 4. removing: 3.
orders-4: replicas: 2,3,4,1. adding: 4. removing: 1.
orders-5: replicas: 3,4,1,2. adding: 4. removing: 2.
A new replica joins the ISR when it has caught up; then the controller switches the replica set to the target, drops the removed replicas (their directories are renamed ...-delete and deleted a minute later) and, if the leader was removed, elects a new one.
3. Verify - and remove the throttle
$ kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 --reassignment-json-file /tmp/plan.json --verify
Status of partition reassignment:
Reassignment of partition orders-0 is completed.
Reassignment of partition orders-1 is still in progress.
Reassignment of partition orders-2 is still in progress.
Reassignment of partition orders-3 is still in progress.
Reassignment of partition orders-4 is still in progress.
Reassignment of partition orders-5 is still in progress.
Still in progress: kafka-4 copies five partitions at once, and the 20 MB/s throttle is shared by all of them (a throttle rate is per broker, not per partition). About 3 GB at 20 MB/s is a few minutes. Later:
$ kafka-reassign-partitions.sh --bootstrap-server kafka-1:9092 --reassignment-json-file /tmp/plan.json --verify
Status of partition reassignment:
Reassignment of partition orders-0 is completed.
Reassignment of partition orders-1 is completed.
Reassignment of partition orders-2 is completed.
Reassignment of partition orders-3 is completed.
Reassignment of partition orders-4 is completed.
Reassignment of partition orders-5 is completed.
Clearing broker-level throttles on brokers 1,2,3,4
Clearing topic-level throttles on topic orders
--verify is not optional: when everything is complete, it removes the throttles. Forget it and the throttle stays on, and some day a broker that has to catch up after a restart does so at 20 MB/s. Check:
$ kafka-configs.sh --bootstrap-server kafka-1:9092 --describe --entity-type brokers --entity-name 4
Dynamic configs for broker 4 are:
To change the throttle of a running move, re-run --execute with the same file, a new --throttle and --additional. --cancel with the same file stops the moves that have not finished and restores the original replica sets.
Preferred leaders
The first replica in a partition's list is its preferred leader. After failovers, restarts and moves, leadership drifts: one broker leads too many partitions and carries most of the traffic. The controller corrects it on its own: every leader.imbalance.check.interval.seconds (300) it checks each broker, and when more than leader.imbalance.per.broker.percentage (10 %) of the partitions it prefers are led by someone else, it moves leadership back (auto.leader.rebalance.enable=true). Some teams turn that off and do it by hand, at a quiet moment:
$ kafka-leader-election.sh --bootstrap-server kafka-1:9092 --election-type preferred --all-topic-partitions
Successfully completed leader election (PREFERRED) for partitions orders-2, orders-3, orders-4, orders-5
$ kafka-leader-election.sh --bootstrap-server kafka-1:9092 --election-type preferred --topic orders --partition 0
Valid replica already elected for partitions orders-0
With --all-topic-partitions, only the partitions that changed are listed; with one partition named, a leader that is already right prints "Valid replica already elected". A preferred replica that is not in the ISR cannot be elected (PreferredLeaderNotAvailableException): wait until it has caught up.
In an interview: "New brokers get no data until you reassign. Generate or write the plan, keep the rollback JSON, execute with a throttle, and run --verify until it is complete, because --verify removes the throttle. Preferred leader election moves leadership back to the first replica."