How to develop/test Minio event notifications to Kafka

minio-kafka-notification-dev.org

How to develop/test Minio event notifications to Kafka

It is most convenient to use Docker to do this.

A Kafka setup is required (and this requires a Zookeeper setup). A Kafka consumer is needed to check the events sent to Kafka. I used kafkacat.

Kafka Setup in Docker

Use the Kafka+Zookeeper bundled image created at Spotify - https://hub.docker.com/r/spotify/kafka/:

$ # Pull image
$ docker pull spotify/kafka

$ # Run Kafka (with Zookeeper). The ADVERTISED_HOST environment variable
$ # needs to be set to the IP of the docker interface on your host
$ # machine. See the docker repository page for this image for more
$ # details.
$ docker run -p 2181:2181 -p 9092:9092 \
      --name kafka --rm \
      --env ADVERTISED_HOST=172.17.0.1 \
      --env ADVERTISED_PORT=9092 \
      spotify/kafka

This starts a Kafka server, listening on your machine at port 9092.

Kafka Consumer Setup

I used the kafkacat tool from https://github.com/edenhill/kafkacat

It is available in the Ubuntu 16.04 repo.

So to get started:

$ sudo apt install kafkacat
$ # Run kafkacat. The -t option sets the topic name to listen on.
$ kafkacat -b localhost:9092 -t myobjectevents

Configuring Minio Server

Edit the Kafka section with content as shown below.

"kafka": {
        "1": {
                "enable": true,
                "brokers": ["localhost:9092"],
                "topic": "myminioobjects"
        }
}

Using mc to configure events

$ mc events add myminio/bucket arn:minio:sqs:us-east-1:1:kafka
$ mc events list myminio/bucket
arn:minio:sqs:us-east-1:1:kafka   s3:ObjectCreated:*,s3:ObjectRemoved:*   Filter: 

Seeing it in action

$ # From a Terminal 1
$ mc cp main.go myminio/bucket

$ # In the terminal on which kafkacat is running, you can see the
$ # output
$ kafkacat -b localhost:9092 -t myminioobjects 
% Auto-selecting Consumer mode (use -P or -C to override)
{"EventType":"s3:ObjectCreated:Put","Key":"bucket/main.go","Records":[{"eventVersion":"2.0","eventSource":"aws:s3","awsRegion":"us-east-1","eventTime":"2016-12-13T13:39:12Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minio"},"requestParameters":{"sourceIPAddress":"127.0.0.1:57356"},"responseElements":{},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"bucket","ownerIdentity":{"principalId":"minio"},"arn":"arn:aws:s3:::bucket"},"object":{"key":"main.go","size":960,"eTag":"4db901b40817713ec7244abbf707dde8","sequencer":"148FD469ACBD347D"}}}],"level":"info","msg":"","time":"2016-12-13T19:09:12+05:30"}