Confluent - Kafka setup

grafana-kafka-dashboard.jpg

Setup-Steps.md

Environments

Docker host name: localhost
Docker host ip: 192.168.0.98 - (macOS: `ifconfig en0 inet`; Linux: `/sbin/ifconfig  eth0`)

Zookeeper address: 192.168.0.98:2181
Kafka address: 192.168.0.98:9092
Grafana UI: http://192.168.0.98:3000
Prometheus UI: http://192.168.0.98:9090/graph
Kafka metrics: http://192.168.0.98:8080

Confluent address: .....

1. setup monitoring

git clone https://github.com/Lax/kafka-prometheus-monitoring -b lax-patch-1
cd kafka-prometheus-monitoring

Changes: https://github.com/Lax/kafka-prometheus-monitoring/pull/1/files

docker-compose create
docker-compose up

check if all services running, open another terminal

docker-compose ps
netstat -ntlp

2. setup confluent [skip]

3. setup streaming

git clone https://github.com/Lax/kafka-confluent-examples.git -b patch-1
cd kafka-confluent-examples

Changes: https://github.com/Lax/kafka-confluent-examples/pull/1/files

mvn clean package
java -cp target/streams-examples-3.0.0-standalone.jar io.confluent.examples.streams.WordCountLambdaExample

4. confirm data

# in kafka-prometheus-monitoring/
docker-compose exec kafka /bin/sh -c 'unset JMX_PORT; /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper:2181 --list'

# in kafka-prometheus-monitoring/
docker-compose exec kafka /bin/sh -c 'unset JMX_PORT; /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic wordcount-avro-lambda-example'

Attachments

jenkins -> [1] -> kafka -> [2] -> pushgateway -> prometheus -> grafana

jenkins -> [1] -> kafka -> [streaming] -> kafka -> [2] -> pushgateway -> prometheus -> grafana