gsoc21-amqp-plan.md

NEXT LEVEL AMQP

Goal

Bucket notifications are a mechanism to send notification regarding these operations to external systems. Please look at the following for usecases on how and when bucket notifications could be used:

• Analyzing X-rays based on Ceph and Knative
• Data Pipelines on based on a different storage solution called minio and argo
• Pulling notifications from CEDA

Currently, we support sending notifications via Kafka and AMQP0.9.1 message brokers (mostly rabbirmq, as well as an HTTP endpoint. In this project, the goal is to add support AMQP1.0 (mostly Qpid and ActiveMQ). Though the names are similar, these protocols are different, and will require different client libraries. Even tough AMQP0.9.1 is more dominant in the market of brokers, AMQP1.0 has its place (e.g. this comparisson).

RGW Plan Pointers

The Code

• Notifications are sent whenever an S3 client is performing an operation on the RADOS Gateway (RGW).
• Will be implemented in the RGW code base which is part of the Ceph code base
• The interface for notifications is defined in: src/rgwrgw_notify.h
• These calls are invoked from the "operations" context (see: src/rgw/rgw_op.cc)
• Depending with the type of endpoint, the notification is sent via the relevant module (HTTP, Kafka, AMQP)
• The code that select the correct endpoint could be found in: src/rgw/rgw_pubsub_push.cc/h
• Currently we support kafka (see: src/rgw/rgw_kafka.cc/h), amqp0.9.1 (see: src/rgw/rgw_amqp.cc/h) and cURL based http client (see: src/rgw/rgw_http_client.cc/h and src/rgw/rgw_http_client_curl.cc/h)
• As part of this project we should add: src/rgw/rgw_amqp_1.cc/h to expose what we need from the
• Note that the configuration of the topics is done in: src/rgw/rgw_rest_pubsub.cc
• For AMQP0.9.1 we use librabbitmq, however, it does not support AMQP1.0, so we will have to use a different library
• As client code we should probably use the Qpid Proton library. Some things to investigate:
  • Is the library prepackaged for the linux distros relevant for ceph (centos8, rhel8, fedora, ubuntu18/20, opensuse15)?
  • Are there any significant discrepencies netween the libarary versions packaged for the different distros?
  • Should we use the C or C++ versions of the lib?
  • What are the security features that the library provide?

Threading

• We use coroutine based asynchrony in our frontend (which is based on boost beast)
• In the "operations" context we have an optional_yield parameter which enable asynchrony in our calls to the endpoint
• We also have an option to use "persistent notifications" where the notifications will be sent from a different thread context (using a similar coroutine mechanism)
• Both in kafka and amqp we use a lockfree queue from the "operations" context into the endpiint module, and a callback mechanism to release the caller when the actual ack/nack is received from the endpint
• As part of the project we should investigate how the Qpid Proton library treats asynchrony and wrap that with the solution that fits our system (this is mostly what the rgw_amqp_1.cc file should implement)

Security

• This is often a complex are where we will have to scope down what we plan on supporting
• Basic requirement would be to allow for AMQP1.0 over SSL support (as well as non secure transport)

Testing

• we should be able to tests against a real broker: ideally both Qpid and ActiveMQ
• end2end tests should be automated in bash/pyhton. please make sure you open source any tooling you create for that (could be in your own repo)
• as a stretch goal we would add them to the exiting, python based, test automation system (called "teuthology")
• integration features are hard to unit test, therefore unit tests are expected only for the rgw_amqp_1 code (see: src/test/rgw/test_rgw_amqp.cc), and will probably require mocking of the library

Documentation

• is expected to be done in our markdown system (see: doc/radosgw/notifications.rst)