Docker Compose Kafka Setup for Local Development

docker-compose.yaml

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
    ports:
      - 2181:2181

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
      - 9094:9094
    volumes:
      - ./data/kafka1/data:/var/lib/kafka/data
      - ./data/kafka1/etc:/etc/kafka
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_LISTENERS: INTERNAL://:9092,OUTSIDE://:9094
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,OUTSIDE://host.docker.internal:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
    extra_hosts:
      - "host.docker.internal:172.17.0.1"

  kafka-topics-generator:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - kafka
    command: >
      bash -c
        "sleep 5s &&
        kafka-topics --create --topic=services --if-not-exists --bootstrap-server=kafka:9092"
        
  control-center:
    image: confluentinc/cp-enterprise-control-center:latest
    hostname: control-center
    depends_on:
      - kafka
    ports:
      - "9021:9021"
    environment:
      CONTROL_CENTER_BOOTSTRAP_SERVERS: 'kafka:9092'
      CONTROL_CENTER_REPLICATION_FACTOR: 1
      PORT: 9021

not working

not working

May I know what issue you are having? Sometimes the data folder can get corrupt. Try running ‘docker-compose start’ again in case Kafka is stopped.

not working

May I know what issue you are having? Sometimes the data folder can get corrupt. Try running ‘docker-compose start’ again in case Kafka is stopped.

Thanks for the prompt reply. There are a couple of problems. 1st is the internal network. And 2nd is its could not find the own /etc etc. folders. I found a similar example with Networking it worked however this time localhost:9021 not responding.

not working

May I know what issue you are having? Sometimes the data folder can get corrupt. Try running ‘docker-compose start’ again in case Kafka is stopped.

Thanks for the prompt reply. There are a couple of problems. 1st is the internal network. And 2nd is its could not find the own /etc etc. folders. I found a similar example with Networking it worked however this time localhost:9021 not responding.

@denizumutdereli which operating system are you using? I'm running it on OSX Big Sur and it's working fine. Make sure your docker-compose file is on a folder that is sharable on Docker, otherwise it won't be able to create the folders ./data/kafka1/data and ./data/kafka1/etc for mountings.

@thiagotigaz
i am using this in local env
getting error:-

WARN [AdminClient clientId=adminclient-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

thanx in advance