Skip to content

Instantly share code, notes, and snippets.

@simkimsia

simkimsia/kafka-commands.md

Last active April 16, 2018 11:26
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save simkimsia/2999f6de7f82a0c9eddba6883daf3246 to your computer and use it in GitHub Desktop.
Save simkimsia/2999f6de7f82a0c9eddba6883daf3246 to your computer and use it in GitHub Desktop.
Download ZIP
Kafka commands
Raw
kafka-commands.md

Commands

How to get started

  1. start docker

  2. run this command in 1 terminal

docker run --rm -it \
           -p 2181:2181 -p 3030:3030 -p 8081:8081 \
           -p 8082:8082 -p 8083:8083 -p 9092:9092 \
           -e ADV_HOST=127.0.0.1 \
           landoop/fast-data-dev
  1. because the kafka is long running process, have to keep the terminal opened
  2. u can go to 127.0.0.1:3030 to see the ui for the virtual machine on browser
  3. open a new terminal and paste docker run --rm -it --net=host landoop/fast-data-dev bash

How to CRUD topics

  1. kafka-topics --zookeeper 127.0.0.1:2181 press TAB to see what flags are available
  2. To create a topic kafka-topics --zookeeper 127.0.0.1:2181 --create --topic first_topic --partitions 3 --replication-factor 1
  3. To list topics kafka-topics --zookeeper 127.0.0.1:2181 --list
  4. To describe a topic kafka-topics --zookeeper 127.0.0.1:2181 --describe --topic first_topic
  5. To delete a topic kafka-topics --zookeeper 127.0.0.1:2181 --delete --topic first_topic

Send messages

  1. kafka-console-producer --broker-list 127.0.0.1:9092 --topic first_topic then type and hit enter each time
  2. check the ui topics to see the messages
  3. messages will be randomly assigned to different partitions across topic
  4. use ctrl+C to end

Consume messages

  1. must open in yet another terminal because the consumer consumes live so need to send message at same time
  2. kafka-console-consumer must use bootstrap-server as option
  3. kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic then type and hit enter each time
  4. kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic --from-beginning to get all historical messages
  5. kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic --from-beginning --partition 0 to get all historical messages from just partition 0
  6. but so far none of the above consumer commands will commit any offsets to the consumer offsets topic
  7. kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic --from-beginning --consumer-property group.id=mygroup1 run once to get all messages but second time u will get nothing because of consumer offsets
  8. use ctrl+C to end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment