Kafka tools handy commands

kafka-commands.md

Kafka Broker creation with Docker Compose

Using from https://github.com/wurstmeister/kafka-docker.git and docker-compose.yml is:

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
    volumes:
      - ~/workon/kafka/vol-zk/data:/var/lib/zookeeper/data
      - ~/workon/kafka/vol-zk/log:/var/lib/zookeeper/log
  kafka:
    build: .
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ~/workon/kafka/vol-kafka/data:/var/lib/kafka/data

Run docker-compose up -d to start it up

Topics

List topics

kafka-topics.sh --zookeeper zookeeper:2181 --list

Describe topic

kafka-topics.sh --zookeeper zookeeper:2181 --describe --topic <topic-name>

Create topic with one node broker (hence --replication-factor 1)

kafka-topics.sh --zookeeper zookeeper:2181 --create --topic <topic-name> --partitions 3 --replication-factor 1

Delete topic

kafka-topics.sh --zookeeper zookeeper:2181 --delete --topic <topic-name>

Purging a topic (by lowering retention time) Lower the retention time

kafka-topics --zookeeper zookeeper:2181 --alter --topic <topic-name> --config retention.ms=1000

Then waiting until kafka does cleanup...

Interesting offset counting...

Last message offsets, using JVM-based tools (slow)

kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic <topic-name> --time -1

First (smallest) offsets for topic partitions, using JVM-based tools (slow)

kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic <topic-name> --time -2

Difference between last and first (last-first) gives the number of messages in the topic.

Consumer

Starting a consumer from terminal. If no --group a default group will be assigned. It will be consuming and outputting messages until Ctrl+z

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic <topic-name> --group <group-name>

Consumer groups

Getting info on consumer groups currently consuming from topics

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group <group-name>