Skip to content

Instantly share code, notes, and snippets.

View lazystone's full-sized avatar
💭
¯\_(ツ)_/¯

Andrej Urvantsev lazystone

💭
¯\_(ツ)_/¯
View GitHub Profile
@lazystone
lazystone / gist:f18d459a9905fa4045b4726c950ca620
Last active November 9, 2017 15:48
Resizing a Kafka cluster
ZK_HOST=$(/usr/bin/consul watch -http-addr localhost:8500 -type service -service zookeeper-ms | jq -r "(.[]|(.Node.Address +\":\"+(.Service.Port|tostring)))" | xargs -i echo "{}" | paste -sd "," -)

kafka/bin/kafka-topics.sh --zookeeper $ZK_HOST --list > all_topics

grep -v deletion all_topics | tr '\n' ' ' | head --bytes -1 | jq -R 'split(" ") | reduce .[] as $topic ([]; . + [{"topic": $topic }]) | {"topics": . , "version": 1}' > all_topics.json

kafka/bin/kafka-reassign-partitions.sh --zookeeper $ZK_HOST --topics-to-move-json-file all_topics.json --broker-list "5062,51221,52218" --generate | tail -n 1 > reassignment.json

# Edit the reassignment.json file manually