-
-
Save rmoff/fb7c39cc189fc6082a5fbd390ec92b3d to your computer and use it in GitHub Desktop.
--- | |
version: '2' | |
services: | |
zookeeper: | |
image: confluentinc/cp-zookeeper:latest | |
environment: | |
ZOOKEEPER_CLIENT_PORT: 2181 | |
ZOOKEEPER_TICK_TIME: 2000 | |
kafka: | |
# "`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,- | |
# An important note about accessing Kafka from clients on other machines: | |
# ----------------------------------------------------------------------- | |
# | |
# The config used here exposes port 29092 for _external_ connections to the broker | |
# i.e. those from _outside_ the docker network. This could be from the host machine | |
# running docker, or maybe further afield if you've got a more complicated setup. | |
# If the latter is true, you will need to change the value 'localhost' in | |
# KAFKA_ADVERTISED_LISTENERS to one that is resolvable to the docker host from those | |
# remote clients | |
# | |
# For connections _internal_ to the docker network, such as from other services | |
# and components, use kafka:9092. | |
# | |
# See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details | |
# "`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,- | |
# | |
image: confluentinc/cp-kafka:latest | |
depends_on: | |
- zookeeper | |
ports: | |
- 29092:29092 | |
environment: | |
KAFKA_BROKER_ID: 1 | |
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 | |
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092 | |
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT | |
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT | |
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 |
KAFKA_AUTO_CREATE_TOPICS_ENABLE
is not a property supported by this Docker image.
Regarding the connection error see https://rmoff.net/2018/08/02/kafka-listeners-explained/
For any more questions, the best place to ask this is on:
→ Mailing list: https://groups.google.com/forum/#!forum/confluent-platform
or
→ Slack group: http://cnfl.io/slack
Issue is resolved by making change here :
KAFKA_ADVERTISED_LISTENERS: 'SSL://:9093,PLAINTEXT://kafka:9092,PLAINTEXT_HOST://192.168.99.100:29092'.
Need to change localhost to 192.168.99.100(IP address)
shouldn't port 9092
be exposed in order for PLAINTEXT://kafka:9092
to work?
https://gist.github.com/rmoff/fb7c39cc189fc6082a5fbd390ec92b3d#file-docker-compose-yml-L36
Thanks for sharing! 🙏
shouldn't port
9092
be exposed in order forPLAINTEXT://kafka:9092
to work?
https://gist.github.com/rmoff/fb7c39cc189fc6082a5fbd390ec92b3d#file-docker-compose-yml-L36
No, because 9092 is for use within the Docker network (and thus doesn't need to be exposed). 29092 is for access from the host and thus is exposed.
See https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
Just for additional information how to create, send and consume messages on that simple setup.
docker-compose up -d
to setup project
Next commands should be executed on the kafka container, so first log in into the container by typing:
docker-compose exec kafka bash
to enter kafka`/bin/kafka-topics --create --topic topic-name --bootstrap-server localhost:9092
- will create topic/bin/kafka-console-consumer --topic topic-name --from-beginning --bootstrap-server localhost:9092
<-- will execute consumer on topic-name topic- In new terminal on kafka container type
/bin/kafka-console-producer --topic topic-name --bootstrap-server localhost:9092
<--- and now You can type messages that will be listening in You consumer
Thanks!! Its working.....
i am getting this org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
i am getting this org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
Here the same!
I was also getting the "broker not available" error - took me a few mins to discover that the internal port is also 29092.. magic.
@Ishang22 @franbusleiman could you solve this issue "connection to node -1 could not be established" ?
@elanurozlem whcih command returns this error ? I've check this and it works ... only two new parameters needs to be added for example to create topic:
/bin/kafktopics --create --topic topic-name --partitions 1 --replication-factor 1ng --bootstrap-server localhost:9092
If you're on a mac and running into Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available.
, replace localhost
with host.docker.internal
. Now KAFKA_ADVERTISED_LISTENERS
should look like the following,
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://host.docker.internal:29092
Note that if you're connecting to kafka from another container, you'll need to replace localhost
in the client's bootstrap servers with host.docker.internal
as well.
If you are getting Broker may not be available.
then make sure you are connecting to the container using docker-compose exec kafka bash
, not docker exec -it <CONTAINER_ID> bash
Thank you very much! It was a life-saver!
I have a doubt, why are we port mapping 29092:29092
, shouldn't it be 29092:9092
, because kafka provides services at port no: 9092
@rmoff
I am not able to send the messages from the local machine for the kafka container created by the above docker compose file.
also added 2 more environment properties to get the topics created automatically:
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
KAFKA_CREATE_TOPICS: order-topic:3:1
Command for producing messages :kafka-console-producer.bat --broker-list localhost:29092 --topic order-topic
Error Received E:\Kafka Learning Udemy\OrderManagementAndFulfilmentApp\scripts>kafka-console-producer.bat --broker-list localhost:29092 --topic order-topic
Please help me in understanding what I am doing wrong here.
Regards,
Shani Jaiswal