Skip to content

Instantly share code, notes, and snippets.

The More the Merrier

Background

Persistent bucket notifications are a very useful and powerful feature. To learn more about it, you can look at this tech talk and usecase example.

Persistent notifications are usually better that synchronous notification, due to several reasons:

  • the queue they are using is, in fact, a RADOS object. This gives the queue the reliability level of RADOS
  • they do not add the delay of sending the notification to the broker to the client request round trip time
  • they allow for temporary disconnects with the broker or broker restarts without affecting the service
  • they have a time and attempts retry mechanism
################################################################################################################
# Define the settings for the rook-ceph cluster with common settings for a small test cluster.
# All nodes with available raw devices will be used for the Ceph cluster. One node is sufficient
# in this example.

Test

this test assumes ceph cluster with RGW is deployed via vstart

  • create the "log' bucket:
aws --endpoint-url http://localhost:8000 s3 mb s3://all-logs

Standard Mode

  • create a bucket for standard logging:

Phase 0

  • draft PR
  • initial PR
  • initial test PR

Phase 1

  • add "flush" REST API call to fix the issue of lazy-commit. use POST /<bucket name>/?logging as the command
  • add admin command to get bucket logging info: radosgw-admin bucket logging get
  • handle copy correctly:
  • in "Journal" mode, we should just see the "PUT" of the new object (existing behavior)

install:

sudo dnf -y install epel-release
curl -s https://packagecloud.io/install/repositories/rabbitmq/erlang/script.rpm.sh | sudo bash
sudo dnf -y install erlang         
curl -s https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.rpm.sh | sudo bash
sudo dnf -y install rabbitmq-server  

add conf to allow guest when server is not on localhost:

existing functionality

  • frontend requests tracing on the RGW
  • OSD traces
  • jaeger orchestration via cephadm
  • multipart upload tracing when the process is done across multiple RGWs
  • end2end (RGW<->OSD) tracing of PUT object operations
  • conditional tracing on the RGW using Lua scripting

work in progress

start a single zone cluster with a realm:

OSD=1 MON=1 MDS=0 MGR=0 ../src/test/rgw/test-rgw-multisite.sh 1

create a persistent topic with a bucket and notifications:

aws --region=zg1 --endpoint-url http://localhost:8001 sns create-topic --name=fishtopic --attributes='{"push-endpoint": "kafka://localhost", "persistent": "true"}'
aws --region=zg1 --endpoint-url http://localhost:8001 s3 mb s3://fish
aws --region=zg1 --endpoint-url http://localhost:8001 s3api put-bucket-notification-configuration --bucket fish --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:default::fishtopic", "Events": []}]}'

based on the data from here:

sudo dnf install http://rpmfind.net/linux/fedora/linux/releases/40/Everything/x86_64/os/Packages/j/jericho-html-3.3-30.fc40.noarch.rpm
sudo dnf install http://rpmfind.net/linux/fedora/linux/releases/40/Everything/x86_64/os/Packages/b/batik-util-1.14-13.fc40.noarch.rpm
sudo dnf install http://rpmfind.net/linux/fedora/linux/releases/40/Everything/x86_64/os/Packages/x/xmlgraphics-commons-2.9-3.fc40.noarch.rpm
sudo dnf install http://rpmfind.net/linux/fedora/linux/releases/40/Everything/x86_64/os/Packages/b/batik-css-1.14-13.fc40.noarch.rpm
sudo dnf install http://rpmfind.net/linux/fedora/linux/releases/40/Everything/x86_64/os/Packages/b/batik-1.14-13.fc40.noarch.rpm
sudo dnf install http://rpmfind.net/linux/fedora/linux/updates/testing/40/Everything/x86_64/Packages/d/ditaa-0.10-24.fc40.noarch.rpm

pacific

get tag range:

git tag -l "v16*"

list of related commits (needs some cleanup):

git log  --oneline v16.0.0...v16.2.15 --grep="amqp|kafka|pubsub|notif" --no-merges --extended-regexp

quincy

# bucket notification parser according to the "pacific" formatting:
# https://github.com/ceph/ceph/blob/pacific/src/rgw/rgw_pubsub.h
# Note: JSON formatting does not match the one define at:
# https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html
# but includes all data except: metadata, tags and opaque data
import struct
import sys
def prase_string(buff, offset):