Docker-Compose for Kafka and Zookeeper with internal and external listeners

docker-compose.yml

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    # "`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-
    # An important note about accessing Kafka from clients on other machines: 
    # -----------------------------------------------------------------------
    #
    # The config used here exposes port 29092 for _external_ connections to the broker
    # i.e. those from _outside_ the docker network. This could be from the host machine
    # running docker, or maybe further afield if you've got a more complicated setup. 
    # If the latter is true, you will need to change the value 'localhost' in 
    # KAFKA_ADVERTISED_LISTENERS to one that is resolvable to the docker host from those 
    # remote clients
    #
    # For connections _internal_ to the docker network, such as from other services
    # and components, use kafka:9092.
    #
    # See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details
    # "`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-
    #
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Just for additional information how to create, send and consume messages on that simple setup.

1. docker-compose up -d to setup project
   Next commands should be executed on the kafka container, so first log in into the container by typing:
   docker-compose exec kafka bash to enter kafka`
2. /bin/kafka-topics --create --topic topic-name --bootstrap-server localhost:9092 - will create topic
3. /bin/kafka-console-consumer --topic topic-name --from-beginning --bootstrap-server localhost:9092 <-- will execute consumer on topic-name topic
4. In new terminal on kafka container type /bin/kafka-console-producer --topic topic-name --bootstrap-server localhost:9092 <--- and now You can type messages that will be listening in You consumer

Thanks!! Its working.....

i am getting this org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

i am getting this org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

Here the same!

I was also getting the "broker not available" error - took me a few mins to discover that the internal port is also 29092.. magic.

@Ishang22 @franbusleiman could you solve this issue "connection to node -1 could not be established" ?

@elanurozlem whcih command returns this error ? I've check this and it works ... only two new parameters needs to be added for example to create topic:
/bin/kafktopics --create --topic topic-name --partitions 1 --replication-factor 1ng --bootstrap-server localhost:9092

If you're on a mac and running into Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available., replace localhost with host.docker.internal. Now KAFKA_ADVERTISED_LISTENERS should look like the following,

      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://host.docker.internal:29092

Note that if you're connecting to kafka from another container, you'll need to replace localhost in the client's bootstrap servers with host.docker.internal as well.

If you are getting Broker may not be available. then make sure you are connecting to the container using docker-compose exec kafka bash, not docker exec -it <CONTAINER_ID> bash