Resizing a Kafka cluster via bash

gistfile1.txt

#!/bin/bash -xe

# requires jq and 3 brokers to start
which jq || echo 'no jq found, bye!' && exit 1
ZK_HOST=zkhost1:2181/kafka

# record all topics
/usr/local/kafka/bin/kafka-topics.sh --zookeeper $ZK_HOST --list > all_topics

# generate json for listing which topics to reassign
grep -v deletion all_topics | tr '\n' ' ' | head --bytes -1 | jq -R 'split(" ") | reduce .[] as $topic ([]; . + [{"topic": $topic }]) | {"topics": . , "version": 1}' > all_topics.json

# generate the json for reassigning topics, the tool outputs current state then proposed state so take only the last line.  For a real cluster resize, you'd include all the brokers, but in this case I'm retiring brokers 0, 1 and 2.
/usr/local/kafka/bin/kafka-reassign-partitions.sh --zookeeper $ZK_HOST --topics-to-move-json-file all_topics.json --broker-list "4,5,6" --generate | tail -n 1 > reassignment.json

# actually do the reassignment
/usr/local/kafka/bin/kafka-reassign-partitions.sh --zookeeper $ZK_HOST --topics-to-move-json-file all_topics.json --reassignment-json-file reassignment.json --execute

# watch to see when reassignment is completed
watch -n 5 /usr/local/kafka/bin/kafka-reassign-partitions.sh --zookeeper $ZK_HOST --topics-to-move-json-file all_topics.json --reassignment-json-file reassignment.json --verify

Thanks.
Tip: Use the -s option with jq for large lists. It prevents in from splitting in between a line when a buffer fills. Ex

    ...
    {
      "topic": "this.is.actually."
    }
  ],
  "version": 1
}
{
  "topics": [
    {
      "topic": "one.topic"
    },
    ...

in line 17 you don't need this part --topics-to-move-json-file all_topics.json

in line 20 , we don't need this part --topics-to-move-json-file all_topics.json