Skip to content

Instantly share code, notes, and snippets.

@rmoff
Last active July 22, 2023 16:29
  • Star 31 You must be signed in to star a gist
  • Fork 9 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save rmoff/fb7c39cc189fc6082a5fbd390ec92b3d to your computer and use it in GitHub Desktop.
Docker-Compose for Kafka and Zookeeper with internal and external listeners
---
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka:
# "`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-
# An important note about accessing Kafka from clients on other machines:
# -----------------------------------------------------------------------
#
# The config used here exposes port 29092 for _external_ connections to the broker
# i.e. those from _outside_ the docker network. This could be from the host machine
# running docker, or maybe further afield if you've got a more complicated setup.
# If the latter is true, you will need to change the value 'localhost' in
# KAFKA_ADVERTISED_LISTENERS to one that is resolvable to the docker host from those
# remote clients
#
# For connections _internal_ to the docker network, such as from other services
# and components, use kafka:9092.
#
# See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details
# "`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-
#
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- 29092:29092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
@snbeynon
Copy link

snbeynon commented Oct 5, 2021

I was also getting the "broker not available" error - took me a few mins to discover that the internal port is also 29092.. magic.

@elaademiir
Copy link

@Ishang22 @franbusleiman could you solve this issue "connection to node -1 could not be established" ?

@CodeMiner84
Copy link

@elanurozlem whcih command returns this error ? I've check this and it works ... only two new parameters needs to be added for example to create topic:
/bin/kafktopics --create --topic topic-name --partitions 1 --replication-factor 1ng --bootstrap-server localhost:9092

@aaronphilip
Copy link

If you're on a mac and running into Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available., replace localhost with host.docker.internal. Now KAFKA_ADVERTISED_LISTENERS should look like the following,

      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://host.docker.internal:29092

Note that if you're connecting to kafka from another container, you'll need to replace localhost in the client's bootstrap servers with host.docker.internal as well.

@Oleh-Stefanovych
Copy link

Oleh-Stefanovych commented Feb 10, 2023

If you are getting Broker may not be available. then make sure you are connecting to the container using docker-compose exec kafka bash, not docker exec -it <CONTAINER_ID> bash

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment