Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
Resizing a Kafka cluster via bash
#!/bin/bash -xe
# Parse to get Zookeeper hosts
ZK_HOST=$(grep 'zookeeper.connect=' /etc/kafka/ | cut -d'=' -f 2)
# Find the most newly created directory of Kafka in the /opt directory
KAFKA_PATH=$(find /opt/ -maxdepth 1 -name kafka-2* -print0 | xargs -0 ls -d -1 -t | head -n1)
# Manually configured list of brokers to rebalance across (zero-indexed)
# record all topics
$KAFKA_PATH/bin/ --zookeeper $ZK_HOST --list > ~/repartition/all_topics
# generate json for listing which topics to reassign
grep -v deletion ~/repartition/all_topics | tr '\n' ' ' | head --bytes -1 | jq -R 'split(" ") | reduce .[] as $topic ([]; . + [{"topic": $topic }]) | {"topics": . , "version": 1}' > ~/repartition/all_topics.json
# generate the json for reassigning topics
$KAFKA_PATH/bin/ --zookeeper $ZK_HOST --topics-to-move-json-file ~/repartition/all_topics.json --broker-list "$BROKER_LIST" --generate | tail -n 1 > ~/repartition/reassignment.json
# actually do the reassignment
$KAFKA_PATH/bin/ --zookeeper $ZK_HOST --reassignment-json-file ~/repartition/reassignment.json --execute
# watch to see when reassignment is completed
watch -n 5 "$KAFKA_PATH/bin/ --zookeeper $ZK_HOST --reassignment-json-file ~/repartition/reassignment.json --verify | grep progress"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment