gsoc21-nats-plan.md

GOING NATS

Goal

Bucket notifications are a mechanism to send notification regarding these operations to external systems. Please look at the following for usecases on how and when bucket notifications could be used:

• Analyzing X-rays based on Ceph and Knative
• Data Pipelines on based on a different storage solution called minio and argo
• Pulling notifications from CEDA

Currently, we support sending notifications via Kafka and AMQP0.9.1 message brokers, as well as to an HTTP endpoint.

In this project, the goal is to add NATS to the list of bucket notifications endpoints. NATS is a cloud native messaging system designed for Kubernetes.

Since Ceph itself can now run inside Kubernetes (and Openshift) using the Rook project, integration with a cloud native messaging systems is very appealing.

However, unlike Kafka and AMQP, NATS is a young project, therefore we want to avoid (for now...) adding a NATS client library to our code base.

So, the approach here would be to do this integration using our Lua scripting feature, together with a NATS Lua client.

RGW Plan Pointers

The Code

• notifications are sent whenever an S3 client is performing an operation on the RADOS Gateway (RGW).
• Lua/C++ API is defined in the following files:
  • src/rgw/rgw_lua_request.cc/h for binding the RGW "request" context to lua
  • src/rgw/rgw_lua.cc/h for lua script management
  • src/rgw/rgw_lua_utils.cc/h for more generic unitilies used for the binding
• the code for sending the notifications to the NATS server should be done in a Lua script (executed in the RGW "request" context)
• installation of dependencies and libraries should be via luarocks
• we should optimize the connection process (avoid connect/disconnect) per notification (this may require enhancement to our Lua/C++ API)

Coroutines

• we use coroutine based asynchrony in our frontend (which is based on boost beast)
• one of the goals would be to allow for the sending of the notifications to be asynchronous via coroutines (this may require enhancement to our Lua/C++ API)

Testing

• we should be able to tests against a real broker
• end2end tests should be automated in bash/pyhton. please make sure you open source any tooling you create for that (could be in your own repo)
• as a stretch goal we would add them to the exiting, python based, test automation system (called "teuthology")

Documentation

• any enhancements to our Lua infrastructure API should be documented here: doc/radosgw/lua-scripting.rst
• the lua script itself (together with any documentation) should be added to the ceph examples directory