Confluent - Kafka setup

grafana-kafka-dashboard.jpg

Setup-Steps.md

Environments

Docker host name: localhost
Docker host ip: 192.168.0.98 - (macOS: `ifconfig en0 inet`; Linux: `/sbin/ifconfig  eth0`)

Zookeeper address: 192.168.0.98:2181
Kafka address: 192.168.0.98:9092
Grafana UI: http://192.168.0.98:3000
Prometheus UI: http://192.168.0.98:9090/graph
Kafka metrics: http://192.168.0.98:8080

Confluent address: .....

1. setup monitoring

git clone https://github.com/Lax/kafka-prometheus-monitoring -b lax-patch-1
cd kafka-prometheus-monitoring

Changes: https://github.com/Lax/kafka-prometheus-monitoring/pull/1/files

docker-compose create
docker-compose up

check if all services running, open another terminal

docker-compose ps
netstat -ntlp

2. setup confluent [skip]

3. setup streaming

git clone https://github.com/Lax/kafka-confluent-examples.git -b patch-1
cd kafka-confluent-examples

Changes: https://github.com/Lax/kafka-confluent-examples/pull/1/files

mvn clean package
java -cp target/streams-examples-3.0.0-standalone.jar io.confluent.examples.streams.WordCountLambdaExample

4. confirm data

# in kafka-prometheus-monitoring/
docker-compose exec kafka /bin/sh -c 'unset JMX_PORT; /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper:2181 --list'

# in kafka-prometheus-monitoring/
docker-compose exec kafka /bin/sh -c 'unset JMX_PORT; /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic wordcount-avro-lambda-example'

Attachments

check if kafka is working

with a test topic my-test-topic:

# in kafka-prometheus-monitoring/
# Consumer
docker-compose exec kafka /bin/sh -c 'unset JMX_PORT; /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic my-test-topic'

# open another terminal
# Producer
docker-compose exec kafka /bin/sh -c 'unset JMX_PORT; /opt/kafka/bin/kafka-console-producer.sh --broker-list kafka:9092 --topic my-test-topic'

# then type some random strings.
# Switch to previous consumer terminal, see the output

 jenkins -> kafka -> [streaming] -> kafka -> [exporter] -> prometheus -> grafana

Step 1: jenkins -> kafka

This is a program(written in java?), which

1. periodically run
2. read data from jenkins api
3. write data direct to kafka (as producer)

Step 2: kafka -> [streaming] -> kafka

This program uses kafka realtime streaming, which

1. always run
2. processing data

Step 3: kafka -> [exporter] -> prometheus

1. always run
2. export data from kafak to prometheus

Step 4: grafana

 jenkins -> kafka -> [streaming] -> kafka -> [pushgateway] -> prometheus -> grafana

Step 1: jenkins -> kafka

This is a program(written in java?), which

1. periodically run
2. read data from jenkins api
3. write data direct to kafka (as producer)

Step 2: kafka -> [streaming] -> kafka

This program uses kafka realtime streaming, which

1. always run
2. processing data

Step 3: kafka -> [pushgateway] -> prometheus

1. always run
2. export data from kafak, push to prometheus pushgateway

Step 4: grafana

jenkins -> [1] -> kafka -> [2] -> pushgateway -> prometheus -> grafana

jenkins -> [1] -> kafka -> [streaming] -> kafka -> [2] -> pushgateway -> prometheus -> grafana