The Kafka topics partitions replicas reassignment with replica increase

kafka-topics-partitions-replicas-reassignment.md

The Kafka topics partitions replicas reassignment with replica increase

The script gets a csv file with no header which contains kafka topics and the number of each topic's partitions and produces the JSON file to reassign and replicate. The replica broker numbers defined in the BROKERS array.

kafka-topics-partitions-replicas-reassignment.sh

#!/usr/bin/env bash

# TITLE: kafka-topics-partitions-replicas-reassignment

# DESCRIPTION:
# The script gets a `csv` file with no header which contains **kafka** topics 
# and the number of each topic's partitions and produces the JSON file to 
# reassign and replicate. The replica broker numbers defined in the BROKERS 
# array.

# INPUT TEMPLATE:
# Afile named "topics.csv" with two columns like below:
# topic1,11
# topic2,12
# topic3,13

# VERSION: 0.1.0 at Tue, 24 Nov 2020 14:02:47 +0330
# AUTHOR: debman @ GitHub

set -euo pipefail

INPUT_FILE="topics.csv"
OUTPUT_FILE="output.json"
BROKERS=(2 3 5 7 8 9)
TOTAL=$(wc -l < $INPUT_FILE)

echo -n "{\"version\":1,
  \"partitions\":[
" > $OUTPUT_FILE
while read -r LINE ; do
  TOPIC=$(echo $LINE | cut -f1 -d\,)
  PARTITIONS=$(echo $LINE | cut -f2 -d\,)
  for PARTITION in $(seq 0 $((PARTITIONS - 1))); do
    ARRAY=( $(shuf -n 3 -e ${BROKERS[@]}) )
    echo '    {"topic":"'$TOPIC'","partition":'$PARTITION',"replicas":['\
	${ARRAY[0]}','${ARRAY[1]}','${ARRAY[2]}']},' >> $OUTPUT_FILE
  done
done < $INPUT_FILE
echo ']}' >> $OUTPUT_FILE
LINES=$(wc -l < $OUTPUT_FILE)
sed -ri $(($LINES - 1))'s/(.*),/\1 /' $OUTPUT_FILE

Inspired from:

https://stackoverflow.com/questions/37960767/how-to-change-the-number-of-replicas-of-a-kafka-topic