Skip to content

Instantly share code, notes, and snippets.

@rmoff
Last active June 2, 2024 17:14
Show Gist options
  • Save rmoff/fb7c39cc189fc6082a5fbd390ec92b3d to your computer and use it in GitHub Desktop.
Save rmoff/fb7c39cc189fc6082a5fbd390ec92b3d to your computer and use it in GitHub Desktop.
Docker-Compose for Kafka and Zookeeper with internal and external listeners
---
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka:
# "`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-
# An important note about accessing Kafka from clients on other machines:
# -----------------------------------------------------------------------
#
# The config used here exposes port 29092 for _external_ connections to the broker
# i.e. those from _outside_ the docker network. This could be from the host machine
# running docker, or maybe further afield if you've got a more complicated setup.
# If the latter is true, you will need to change the value 'localhost' in
# KAFKA_ADVERTISED_LISTENERS to one that is resolvable to the docker host from those
# remote clients
#
# For connections _internal_ to the docker network, such as from other services
# and components, use kafka:9092.
#
# See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details
# "`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-
#
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- 29092:29092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
@shani530
Copy link

@rmoff
I am not able to send the messages from the local machine for the kafka container created by the above docker compose file.
also added 2 more environment properties to get the topics created automatically:
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
KAFKA_CREATE_TOPICS: order-topic:3:1

Command for producing messages :kafka-console-producer.bat --broker-list localhost:29092 --topic order-topic
Error Received E:\Kafka Learning Udemy\OrderManagementAndFulfilmentApp\scripts>kafka-console-producer.bat --broker-list localhost:29092 --topic order-topic

[2020-04-20 14:31:50,879] WARN [Producer clientId=console-producer] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-20 14:31:51,932] WARN [Producer clientId=console-producer] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

Please help me in understanding what I am doing wrong here.
Regards,
Shani Jaiswal

@rmoff
Copy link
Author

rmoff commented Apr 20, 2020

KAFKA_AUTO_CREATE_TOPICS_ENABLE is not a property supported by this Docker image.

Regarding the connection error see https://rmoff.net/2018/08/02/kafka-listeners-explained/

For any more questions, the best place to ask this is on:
→ Mailing list: https://groups.google.com/forum/#!forum/confluent-platform
or
→ Slack group: http://cnfl.io/slack

@shani530
Copy link

Issue is resolved by making change here :
KAFKA_ADVERTISED_LISTENERS: 'SSL://:9093,PLAINTEXT://kafka:9092,PLAINTEXT_HOST://192.168.99.100:29092'.

Need to change localhost to 192.168.99.100(IP address)

@netrebel
Copy link

netrebel commented Feb 3, 2021

shouldn't port 9092 be exposed in order for PLAINTEXT://kafka:9092 to work?
https://gist.github.com/rmoff/fb7c39cc189fc6082a5fbd390ec92b3d#file-docker-compose-yml-L36

Thanks for sharing! 🙏

@rmoff
Copy link
Author

rmoff commented Feb 3, 2021

shouldn't port 9092 be exposed in order for PLAINTEXT://kafka:9092 to work?
https://gist.github.com/rmoff/fb7c39cc189fc6082a5fbd390ec92b3d#file-docker-compose-yml-L36

No, because 9092 is for use within the Docker network (and thus doesn't need to be exposed). 29092 is for access from the host and thus is exposed.

See https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/

@CodeMiner84
Copy link

Just for additional information how to create, send and consume messages on that simple setup.

  1. docker-compose up -d to setup project
    Next commands should be executed on the kafka container, so first log in into the container by typing:
    docker-compose exec kafka bash to enter kafka`
  2. /bin/kafka-topics --create --topic topic-name --bootstrap-server localhost:9092 - will create topic
  3. /bin/kafka-console-consumer --topic topic-name --from-beginning --bootstrap-server localhost:9092 <-- will execute consumer on topic-name topic
  4. In new terminal on kafka container type /bin/kafka-console-producer --topic topic-name --bootstrap-server localhost:9092 <--- and now You can type messages that will be listening in You consumer

image

@ImAbhishekTomar
Copy link

Thanks!! Its working.....

@Ishang22
Copy link

i am getting this org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

@franbusleiman
Copy link

i am getting this org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

Here the same!

@snbeynon
Copy link

snbeynon commented Oct 5, 2021

I was also getting the "broker not available" error - took me a few mins to discover that the internal port is also 29092.. magic.

@elaademiir
Copy link

@Ishang22 @franbusleiman could you solve this issue "connection to node -1 could not be established" ?

@CodeMiner84
Copy link

@elanurozlem whcih command returns this error ? I've check this and it works ... only two new parameters needs to be added for example to create topic:
/bin/kafktopics --create --topic topic-name --partitions 1 --replication-factor 1ng --bootstrap-server localhost:9092

@aaronphilip
Copy link

If you're on a mac and running into Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available., replace localhost with host.docker.internal. Now KAFKA_ADVERTISED_LISTENERS should look like the following,

      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://host.docker.internal:29092

Note that if you're connecting to kafka from another container, you'll need to replace localhost in the client's bootstrap servers with host.docker.internal as well.

@Oleh-Stefanovych
Copy link

Oleh-Stefanovych commented Feb 10, 2023

If you are getting Broker may not be available. then make sure you are connecting to the container using docker-compose exec kafka bash, not docker exec -it <CONTAINER_ID> bash

@dan0nchik
Copy link

Thank you very much! It was a life-saver!

@AashishNandakumar
Copy link

I have a doubt, why are we port mapping 29092:29092, shouldn't it be 29092:9092, because kafka provides services at port no: 9092

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment