Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Run Kafka Container

Start Kafka service

The following commands will start a container with Kafka and Zookeeper running on mapped ports 2181 (Zookeeper) and 9092 (Kafka).

docker pull spotify/kafka
docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=kafka --env ADVERTISED_PORT=9092 --name kafka spotify/kafka

Why Spotify?

ADVERTISTED_HOST was set to kafka, which will allow other containers to be able to run Producers and Consumers.

Setting ADVERTISED_HOST to localhost, 127.0.0.1, or 0.0.0.0 will work great only if Producers and Consumers are started within the kafka container itself, or if you are using DockerForMac (like me) and you want to run Producers and Consumers from OSX. These are far less interesting use cases though, so we'll start Producers and Consumers from other containers.

We need to use an IP address or hostname in order for the kafka service to be reachable from another container. IP address is not known before the container is started, so we have to choose a hostname, and I chose kafka in this example.

Create a topic

docker exec kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

output:

Created topic "test".

List topics

docker exec kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh --list --zookeeper localhost:2181

output:

test

Start a producer (in a new terminal window)

This command will run an unnamed instance of spotify/kafka linked to the kafka service, start a producer, and wait for newline-delimited input until you quit (which destroys the container):

docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-producer.sh --broker-list kafka:9092 --topic test

Start a consumer (in a new terminal window)

This command will start an unnamed instance of spotify/kafka linked to the kafka service, start a consumer, display existing messages from the test topic, and wait for new messages until you quit (which destroys the container):

docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test --from-beginning

Send messages

Send some newline-delimited messages in the Producer terminal window. The messages appear in the Consumer terminal window.

@abacaphiliac

This comment has been minimized.

Copy link
Owner Author

commented Jan 17, 2017

thanks to @kpankonen for helping me understand how ADVERTISED_HOST affects other containers.

@MarcinMilewski

This comment has been minimized.

Copy link

commented Apr 27, 2017

Thank You

@gdurastanti

This comment has been minimized.

Copy link

commented Jul 19, 2017

Thank you for this. Much simpler than the original doc.

@danswater

This comment has been minimized.

Copy link

commented Aug 8, 2017

Thanks for sharing this one it really helps a lot.
One thing I noticed is when running consumer, my terminal window was flood with this message
WARN Error while fetching metadata with correlation id 6 : {testing=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

and typing something in producer window also emit this kind of message.

I'm able to resolve the issue by adding --hostname when starting the kafka service. [stackoverflow]

@rahulgupta-jsr

This comment has been minimized.

Copy link

commented Dec 21, 2017

First of all Thanks! This got me started quickly.

I was getting lot of error like this:
[2017-12-21 04:46:22,224] WARN Error while fetching metadata with correlation id 95 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

This was solved by adding "-h kafka" when starting the container.

docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=kafka --env ADVERTISED_PORT=9092 --name kafka -h kafka spotify/kafka

@Rohit2387

This comment has been minimized.

Copy link

commented Jan 18, 2018

I am geting below error.
I tried kafka also and localhost also but same result.
docker@default:~$ docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

abcddd
[2018-01-18 11:48:19,683] ERROR Error when sending message to topic test with key: null, value: 0 by
tes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

Can anybody please help to resolve this issue.

@ErnestK

This comment has been minimized.

Copy link

commented Feb 22, 2018

@Rohit2387
Hey,

I had this error also,
in my case this error occurs if you run docker container in composer in network,
you should run in terminal command docker inspect kafka | grep IPAddress, find ip below part network
and run consumer/producer like this
docker run -it --rm --net my_net --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-producer.sh --broker-list ip_from_inspect:9092 --topic test

@gopaltirupur

This comment has been minimized.

Copy link

commented Mar 5, 2018

Works fine, thanks a lot

@jarvis89

This comment has been minimized.

Copy link

commented Mar 29, 2018

Hi,

Thanks for pointing to this simple kafka implementation, it is really helpful to kafka beginners like me. Can you please point me some sample application of .net client using which I can produce and consume kafka messages from console application.

@shrutigoyal2098

This comment has been minimized.

Copy link

commented Apr 23, 2018

@Rohit2387

i found the issue, 9092 was not my port. For me listening port was 6667 (you can check this at kafka.properties file)
Add the following
listeners=PLAINTXT://0.0.0.0:9092
listeners=PLAINTEXT://localhost.localdomain:9092

@chj3737

This comment has been minimized.

Copy link

commented Jun 30, 2018

@danswater

I had the same issue, and it works for me by using the commands like this (ie. using ip address instead of "kafka")
docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=192.168.99.100 --env ADVERTISED_PORT=9092 --name kafka spotify/kafka

docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-producer.sh --broker-list 192.168.99.100:9092 --topic test

docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-consumer.sh --bootstrap-server 192.168.99.100:9092 --topic test --from-beginning

@jesusalvb

This comment has been minimized.

Copy link

commented Sep 19, 2018

Using you commands i have this output in producer and consumer

[2018-09-19 16:57:11,688] WARN Error while fetching metadata with correlation id 245 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-09-19 16:57:11,793] WARN Error while fetching metadata with correlation id 246 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

@jesusalvb

This comment has been minimized.

Copy link

commented Sep 19, 2018

Using you commands i have this output in producer and consumer

[2018-09-19 16:57:11,688] WARN Error while fetching metadata with correlation id 245 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-09-19 16:57:11,793] WARN Error while fetching metadata with correlation id 246 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=kafka --env ADVERTISED_PORT=9092 --hostname kafka --name kafka spotify/kafka

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.