Kafka CLI example using Apache Kafka on Heroku

kafka-from-cli.md

Assumed conditions:

• Apache Kafka on Heroku cluster provisioned
• Topic has been created named cli-test (but you can change the name used below)
• Heroku CLI
• Heroku CLI Kafka plugin
• Homebrew, sed

Install C/C++ Kafka Library (librdkafka) and Kafka CLI tool (kafkacat)

For OS X, use the Homebrew command below. For others, see installation instructions at the links above.

brew install librdkafka kafkacat

Note: Requires librdkafka >= 0.9.1 (SSL support)

Export local environment variables

If you are running the consumer and producer at the same time from two command line windows, you'll need to export these variables on each command line. Replace my-app-with-kafka with your Heroku app name that has the Kafka add-on.

export KAFKA_TRUSTED_CERT="$(heroku config:get KAFKA_TRUSTED_CERT -a my-app-with-kafka)"
export KAFKA_CLIENT_CERT="$(heroku config:get KAFKA_TRUSTED_CERT -a my-app-with-kafka)"
export KAFKA_CLIENT_CERT_KEY="$(heroku config:get KAFKA_TRUSTED_CERT -a my-app-with-kafka)"
export KAFKA_URL="$(heroku config:get KAFKA_URL -a my-app-with-kafka | sed 's/kafka\+ssl\:\/\///g')"
export TOPIC=cli-test

---

Run the following two commands each in its own command line window and watch messages fly from the Consumer through the Kafka cluster to the Producer!

Consumer

This command will start listening and print to stdout any messages it receives from the Kafka cluster. Ctrl-C to stop.

kafkacat -C -t $TOPIC -b $KAFKA_URL \
-X security.protocol=ssl \
-X ssl.key.location=<(echo $KAFKA_CLIENT_CERT_KEY) \
-X ssl.certificate.location=<(echo $KAFKA_CLIENT_CERT) \
-X ssl.ca.location=<(echo $KAFKA_TRUSTED_CERT)

Producer

This command will read input from stdin and send it to the Kafka cluster. Type a message followed by enter. Ctrl-D to stop.

kafkacat -P -t $TOPIC -b $KAFKA_URL \
-X security.protocol=ssl \
-X ssl.key.location=<(echo $KAFKA_CLIENT_CERT_KEY) \
-X ssl.certificate.location=<(echo $KAFKA_CLIENT_CERT) \
-X ssl.ca.location=<(echo $KAFKA_TRUSTED_CERT)

Typo on the export commands: all retrieve the same config variable KAFKA_TRUSTED_CERT