MSK-ClusterExpansion.txt

Lab-2: Cluster Expansion
------------------------

1. Go to Cloud9 console and open your environment IDE
2. In a Cloud 9 terminal use the following command to ssh into Kafka EC2 instance

    Note: change the IP address with Kafka EC2 instance private IP address running in your AWS account
    
    ssh -i msk-workshop-pem.pem ec2-user@10.0.1.124


--------run all the steps in Kafka EC2 Instance terminal/shell---------------

3. set the following environment variables

Note: change the Cluster ARN with your MSK cluster ARN
export CLUSTER_ARN=<your msk cluster arn>
export BS=$(aws kafka get-bootstrap-brokers --cluster-arn $CLUSTER_ARN --output text) 
export ZK=$(aws kafka describe-cluster --cluster-arn $CLUSTER_ARN --output json | jq ".ClusterInfo.ZookeeperConnectString" | tr -d \")


4. Create a new topic with 10 partitions and replication factor of 3

cd kafka 

(From kafka directory i.e. /home/ec2-user/kafka)

./bin/kafka-topics.sh --create --bootstrap-server $BS \
    --topic test10 \
    --partitions 10 \
    --replication-factor 3

5. Add additional brokers to your cluster, go to MSK console, select your MSK cluster

    Under "Broker details", click "Edit Number of Brokers"

    Number of brokers per Zone = 2

    "Save changes"

    Note: Cluster resize takes time.

6. Get current partition allocation

./bin/kafka-topics.sh --bootstrap-server $BS \
  --describe --topic test10
  
7. Create our list of topics to move

change directory to user home directory i.e. /home/ec2-user

create a file "topics-to-move.json" in the user home directory i.e. /home/ec2-user with the following content and save

{ "topics": [ { "topic" : "test10"}], "version":1}


8. Generate repartition plan

cd kafka

(curent directory = /home/ec2-user/kafka)

./bin/kafka-reassign-partitions.sh --bootstrap-server $BS \
    --topics-to-move-json-file ../topics-to-move.json \
    --broker-list "1,2,3,4,5,6" \
    --generate

9. save the output that's under "Current partition replica assignment" (only JSON text) to a 
   new file "current-assignment.json" in the user home directory i.e. /home/ec2-user
   
10. save the output that's under "Proposed partition reassignment configuration" (only JSON text) to a 
   new file "new-assignment.json" in the user home directory i.e. /home/ec2-user

11. Run partition reassignment 

./bin/kafka-reassign-partitions.sh --bootstrap-server $BS \
    --reassignment-json-file ../new-assignment.json \
    --execute

12. Monitor the progress

./bin/kafka-reassign-partitions.sh --bootstrap-server $BS \
    --reassignment-json-file ../new-assignment.json \
    --verify
    
13. Verify partition reassignment
 
 ./bin/kafka-topics.sh --bootstrap-server $BS \
    --topic test10 \
    --describe
 
14. Rollback to the previous assignment. Use current-assignment.json
 
 ./bin/kafka-reassign-partitions.sh --bootstrap-server $BS \
    --reassignment-json-file ../current-assignment.json \
    --execute
    
15. Monitor partition rollback
 
 ./bin/kafka-reassign-partitions.sh --bootstrap-server $BS \
    --reassignment-json-file ../current-assignment.json \
    --verify
 
16. Verify partition reassignment rollback
 
 ./bin/kafka-topics.sh --bootstrap-server $BS \
    --topic test10 \
    --describe