Connect Kafka

README.md

connect-kafka

Connect Segment to Kafka in 5 minutes. Pipe Segment's data sources into your Kafka cluster.

Release: ALPHA

Features

connect-kafka is a simple server that you deploy in your infrastructure. It listens for Segment events and forwards them to the Kafka topic of your choice.

• Easily forward web, mobile, server analytics events to your Kafka instance
• Deploys in your infrastructure
• Supports any Kafka cluster
• Built with Heroku Kafka support in mind (with public/private space support)
• Deploys in 5 minutes
• Allows SSL (or not) connections
• Supports all Segment standard methods (identify, track, page, screen, group)

Quickstart

1. Connect to Kafka - connect the connect-kafka to your Kafka instance.
2. Setup Webbook - Enter connect-kafka's listen address into your Segment webhook menu.

Connect to Kafka

Download connect-kafka using curl:

curl -s http://connect.segment.com/install-connect-kafka.sh | sh

If you just want the binary and install it yourself:

http://connect.segment.com/connect-kafka-darwin-amd64

You can also use Docker:

make docker
docker run segment/connect-kafka [...]

You can connect to any internal Kafka deployment.

$ connect-kafka -h

Usage:
  connect-kafka
    [--debug]
    --topic=<topic>
    --broker=<url>...
    [--listen=<addr>]
    [--trusted-cert=<path> --client-cert=<path> --client-cert-key=<path>]
  connect-kafka -h | --help
  connect-kafka --version

Options:
  -h --help                   Show this screen
  --version                   Show version
  --topic=<topic>             Kafka topic name
  --listen=<addr>             Address to listen on [default: localhost:3000]
  --broker=<url>              Kafka broker URL

Heroku Kafka

Below is an example to connect to a Heroku Kafka in a public space (via SSL):

go get -u github.com/segment-integrations/connect-kafka
heroku config:get KAFKA_URL -a kafka-integration-demo  # copy the kafka broker urls into command below
heroku config:get KAFKA_TRUSTED_CERT -a kafka-integration-demo > kafka_trusted_cert.cer
heroku config:get KAFKA_CLIENT_CERT -a kafka-integration-demo > kafka_client_cert.cer
heroku config:get KAFKA_CLIENT_CERT_KEY -a kafka-integration-demo > kafka_client_key_cert.cer
connect-kafka \
 --debug \
 --topic=segment \
 --broker=kafka+ssl://ec2-51-16-10-109.compute-1.amazonaws.com:9096 \
 --broker=kafka+ssl://ec2-62-7-61-181.compute-1.amazonaws.com:9096 \
 --broker=kafka+ssl://ec2-33-20-240-35.compute-1.amazonaws.com:9096 \
 --trusted-cert=kafka_trusted_cert.cer \
 --client-cert=kafka_client_cert.cer \
 --client-cert-key=kafka_client_key_cert.cer

Setup Webhook

1. Go to the Segment.com and select the source you want to connect to Kafka
2. Add your connect-kafka server's address to the webhook integration's settings.

Testing

via localtunnel

You can open up a localtunnel on your local machine while you're testing:

npm install -g localtunnel
lt --port 3000

Enter the resulting localtunnel url as the Segment webhook with /listen appended, like: https://aqjujyhnck.localtunnel.me/listen

License

MIT