notifications_test_kafka.md

start cluster:

MON=1 OSD=1 MDS=0 MGR=0 RGW=1 ../src/vstart.sh -n -d 

download Kafka from here, unzip, and go into the kafka directory.

start Zookeeper (in a separate terminal):

bin/zookeeper-server-start.sh config/zookeeper.properties

start Kafka broker (in a separate terminal):

bin/kafka-server-start.sh config/server.properties

configure the RGW:

aws --region=default --endpoint-url http://localhost:8000 sns create-topic --name=fishtopic \
  --attributes='{"push-endpoint": "kafka://localhost"}'
aws --endpoint-url http://localhost:8000 s3 mb s3://fish
aws --region=default --endpoint-url http://localhost:8000 s3api put-bucket-notification-configuration  --bucket fish \
  --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:default::fishtopic", "Events": []}]}'

run a consumer for the topic:

bin/kafka-console-consumer.sh --topic fishtopic --bootstrap-server localhost:9092

create a file, and upload it:

head -c 512 </dev/urandom > myfile
aws --endpoint-url http://localhost:8000 s3 cp myfile s3://fish

to test the kafka notifications under higher load, use hsbench.

first we need to create the bucket:

hsbench -a 0555b35654ad1656d804 -s h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q== \ 
  -u http://localhost:8000 -bp bk -m i

then, create the notification on the bucket (reuse the same topic from before):

aws --region=default --endpoint-url http://localhost:8000 s3api put-bucket-notification-configuration \
  --bucket bk000000000000 --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:default::fishtopic", "Events": []}]}'

and last, put and delete objects:

hsbench -a 0555b35654ad1656d804 -s h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q== \ 
  -u http://localhost:8000 -bp bk -m pd -t $(nproc) -z 1K -l 100