Cluster Kafka c/ 3 Brokers

kafka.md

Recursos

• Java (JDK) 1.8
• 4GB memória ram
• 10GB espaço em disco
• Kafka 2.4.0: http://bit.ly/kafka_2_4_0

Kafka Home

export KAFKA_HOME="/path/to/kafka_2.12-2.4.0"

Kafka Bin

export PATH=$PATH:"$KAFKA_HOME/kafka_2.12-2.4.0/bin"

Zookeeper

zookeeper-server-start.sh \
  "$KAFKA_HOME/config/zookeeper.properties"

Broker #1

kafka-server-start.sh "$KAFKA_HOME/config/server.properties" \
  --override broker.id=1 \
  --override log.dirs=/tmp/broker1-logs \
  --override listeners=PLAINTEXT://:9091 \
  --override zookeeper.connect=localhost:2181 \
  --override zookeeper.connection.timeout.ms=10000

Broker #2

kafka-server-start.sh "$KAFKA_HOME/config/server.properties" \
  --override broker.id=2 \
  --override log.dirs=/tmp/broker2-logs \
  --override listeners=PLAINTEXT://:9092 \
  --override zookeeper.connect=localhost:2181 \
  --override zookeeper.connection.timeout.ms=10000

Broker #3

kafka-server-start.sh "$KAFKA_HOME/config/server.properties" \
  --override broker.id=3 \
  --override log.dirs=/tmp/broker3-logs \
  --override listeners=PLAINTEXT://:9093 \
  --override zookeeper.connect=localhost:2181 \
  --override zookeeper.connection.timeout.ms=10000

meu-topico (replication-factor 3)

Criar

kafka-topics.sh --create \
    --bootstrap-server localhost:9092 \
    --replication-factor 3 \
    --partitions 7 \
    --topic meu-topico

Detalhar

kafka-topics.sh --describe \
    --bootstrap-server localhost:9092 \
    --topic meu-topico

topico-a (duas partições)

kafka-topics.sh --create \
    --bootstrap-server localhost:9092 \
    --replication-factor 3 \
    --partitions 2 \
    --topic topico-a

consumer-ex

Exemplo de consumidor kafka escrito e Java.

• Clone o projeto: https://github.com/fabiojose/consumer-ex
• Execute-o com o comando ./gradlew run
• Inicie dois consumidores

Produzir registros

kafka-producer-perf-test.sh \
  --topic topico-a \
  --num-records 10 \
  --record-size 50 \
  --throughput -1 \
  --producer-props \
      acks=1 \
      bootstrap.servers=localhost:9092 \
      batch.size=0