Skip to content

Instantly share code, notes, and snippets.

@crcastle
Last active August 26, 2021 13:02
Show Gist options
  • Save crcastle/cb21c2148fc57ad89753bf28b561d910 to your computer and use it in GitHub Desktop.
Save crcastle/cb21c2148fc57ad89753bf28b561d910 to your computer and use it in GitHub Desktop.
Kafka CLI example using Apache Kafka on Heroku

Assumed conditions:

  • Apache Kafka on Heroku cluster provisioned
  • Topic has been created named cli-test (but you can change the name used below)
  • Heroku CLI
  • Heroku CLI Kafka plugin
  • Homebrew, sed

For OS X, use the Homebrew command below. For others, see installation instructions at the links above.

brew install librdkafka kafkacat

Note: Requires librdkafka >= 0.9.1 (SSL support)

Export local environment variables

If you are running the consumer and producer at the same time from two command line windows, you'll need to export these variables on each command line. Replace my-app-with-kafka with your Heroku app name that has the Kafka add-on.

export KAFKA_TRUSTED_CERT="$(heroku config:get KAFKA_TRUSTED_CERT -a my-app-with-kafka)"
export KAFKA_CLIENT_CERT="$(heroku config:get KAFKA_TRUSTED_CERT -a my-app-with-kafka)"
export KAFKA_CLIENT_CERT_KEY="$(heroku config:get KAFKA_TRUSTED_CERT -a my-app-with-kafka)"
export KAFKA_URL="$(heroku config:get KAFKA_URL -a my-app-with-kafka | sed 's/kafka\+ssl\:\/\///g')"
export TOPIC=cli-test

Run the following two commands each in its own command line window and watch messages fly from the Consumer through the Kafka cluster to the Producer!

Consumer

This command will start listening and print to stdout any messages it receives from the Kafka cluster. Ctrl-C to stop.

kafkacat -C -t $TOPIC -b $KAFKA_URL \
-X security.protocol=ssl \
-X ssl.key.location=<(echo $KAFKA_CLIENT_CERT_KEY) \
-X ssl.certificate.location=<(echo $KAFKA_CLIENT_CERT) \
-X ssl.ca.location=<(echo $KAFKA_TRUSTED_CERT)

Producer

This command will read input from stdin and send it to the Kafka cluster. Type a message followed by enter. Ctrl-D to stop.

kafkacat -P -t $TOPIC -b $KAFKA_URL \
-X security.protocol=ssl \
-X ssl.key.location=<(echo $KAFKA_CLIENT_CERT_KEY) \
-X ssl.certificate.location=<(echo $KAFKA_CLIENT_CERT) \
-X ssl.ca.location=<(echo $KAFKA_TRUSTED_CERT)
@brianlow
Copy link

Typo on the export commands: all retrieve the same config variable KAFKA_TRUSTED_CERT

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment