Docker compose Kafka, Zookeeper and Kafka manager

README.md

Docker compose Kafka, Zookeeper and Kafka manager

Gist inspired by this one intending to be an updated version of it : https://gist.github.com/17twenty/d619f922ab209f750630824f7c6836e3

Install

• Create the environment variable that contains our host name (IP address) :

$ export EXPOSED_HOSTNAME=$(ip addr | grep 'state UP' -A2 | tail -n1 | awk '{print $2}' | cut -f1  -d'/')

• Create the folders :

$ mkdir -p zookeeper/{data,logs}

Run

• Execute the docker compose :

$ docker-compose -f docker-compose.yml up -d

• Access Kafka Manager : http://localhost:9000/.

• Add a new cluster.

• Name it as you like (Localtest for example).

• IP : localhost:2181

• Kafka version (corresponding here to the Kafka version inside the image : wurstmeister/kafka:1.0.0). But as I write this Gist, the latest Kafka version available in Kafka Manager is 0.11.0.0 so I select this one, but it's sufficiently compatible with the 1.0.0 version of Kafka according to this topic: yahoo/CMAK#451

• Tick Enable JMX Polling in order to see the metrics of the topics

• Tick Poll consumer information to know the consumer of a topic (it may not work)

• Tick Enable Active OffsetCache to see the offsets

And then save it.

✨ This is done ! ✨

Uninstall

• Stop the containers and remove them :

$ docker-compose -f docker-compose.yml stop && docker-compose -f docker-compose.yml rm -vf

Tips

• Remove content of Docker volumes (we need to do this trick to remove hidden and non-hidden files) :

$ rm -rf zookeeper/{data,logs} && mkdir -p zookeeper/{data,logs}

docker-compose.yml

version: '3.1'

services:
  zookeeper:
    container_name: zookeeper
    image: zookeeper:3.4
    restart: on-failure
    volumes:
      - "./zookeeper/data:/data"
      - "./zookeeper/logs:/datalog"
    ports:
      - "2181:2181"
    network_mode: "host"

  kafka:
    container_name: kafka
    image: wurstmeister/kafka:1.0.0
    restart: on-failure
    depends_on:
      - zookeeper
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - KAFKA_ZOOKEEPER_CONNECT=${EXPOSED_HOSTNAME}:2181
      - KAFKA_ADVERTISED_HOST_NAME=${EXPOSED_HOSTNAME}
      - JMX_PORT=9093
      - KAFKA_ADVERTISED_PORT=9092
      - KAFKA_DELETE_TOPIC_ENABLE=true
      - KAFKA_LOG_RETENTION_HOURS=1
      - KAFKA_MESSAGE_MAX_BYTES=10000000
      - KAFKA_REPLICA_FETCH_MAX_BYTES=10000000
      - KAFKA_GROUP_MAX_SESSION_TIMEOUT_MS=60000
      - KAFKA_NUM_PARTITIONS=2
      - KAFKA_DELETE_RETENTION_MS=1000
    ports:
      - "9092:9092"
      - "9093:9093"
    network_mode: "host"

  kafka-manager:
    container_name: kafka-manager
    image: hlebalbau/kafka-manager:1.3.3.16
    restart: on-failure
    depends_on:
      - kafka
      - zookeeper
    command: -Dconfig.file=/kafka-manager/conf/application.conf -Dapplication.home=/kafkamanager
    environment:
      - ZK_HOSTS=${EXPOSED_HOSTNAME}
      - APPLICATION_SECRET=letmein
    ports:
      - "9000:9000"
    network_mode: "host"

I'm having an issue when I try to run a consumer within the Kafka container:
bin/kafka-console-consumer.sh --topic mqtt-sensor-1 --from-beginning --bootstrap-server ${EXPOSED_HOSTNAME}:9092

Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 9093; nested exception is: 
        java.net.BindException: Address already in use (Bind failed)

Resolved: Just unset JMX_PORT after you login in the Kafka container