kafka-workshop.md

Welcome to Kafka from scratch workshops!

The below insructions will work on the assumption that

• You are using OSX
• You use Kafka 2.0 version

Exercise 1 (Setup)

1. Install Kafka on your machine - brew install kafka
2. Start Kafka together with ZooKeeper - zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties
3. Make sure auto.create.topics.enable property is set to true
4. Please check what are the ports on which ZooKeeper and Kafka are listening

Exercise 2 (Producer - Consumer)

1. In new terminal tab/window go to /usr/local/Cellar/kafka/<version>/libexec/bin
2. Check list of available topics by using ./kafka-topics.sh --list --zookeeper localhost:2181. It should be empty.
3. Create a topic - ./kafka-topics.sh --create --topic example-topic --zookeeper localhost:2181 --partitions 1 --replication-factor 1 and check if command for listing topics will return it
4. Create a consumer for the topic with the following command - ./kafka-console-consumer.sh --topic example-topic --from-beginning --bootstrap-server localhost:9092
5. Create a producer that will publish infinite stream of messages in the following way - ./kafka-verifiable-producer --topic example-topic --broker-list localhost:9092 --throughput 1

Exercise 3 (Rebalancing)

1. Crete a new topic with 3 paritions - ./kafka-topics.sh --create --topic rebalancing-example-topic --zookeeper localhost:2181 --partitions 3 --replication-factor 1
2. In two separate tabs/windows create two consumers within the same consumer group ./kafka-console-consumer.sh --topic rebalancing-example-topic --bootstrap-server localhost:9092 --group workshop-rebalance --enable-systest-events --from-beginning
3. In third tab/window create a producer that will be publishing messages to the topic - ./kafka-verifiable-producer --topic rebalancing-example-topic --broker-list localhost:9092 --throughput 1
4. Try to find out how partitions are assigned. After you do that kill one consumer and observe what will happen. After a while you can bring back consumer in order to trigger another rebalancing.

Exercise 4 (Application integration with Kafka)

1. Write a Java application that will be using native Kafka producer/consumer API and simulating the program we had in Exercise 2. Producer will be producing integers and consumers will be storing them in shared Map which will simulate database. While writing producer and consumer think about how to optimaize your configuration for
   1. performance
   2. ensuring you will not loose any message Think about different scenarios under which you could loose messages.