yuvalif

prerequisites

sudo dnf install glib2-devel
sudo dnf install libgcrypt-devel
sudo dnf install qt-devel
sudo dnf install qt6-qtbase-devel
sudo dnf install qt6-linguist-devel
sudo dnf install qt6-qttools-devel
sudo dnf install qt6-qt5compat-devel

yuvalif

Instructions for fedora 36

Java

install java. curently gradle does not work with jdk higher than 11, so we would need to:

sudo dnf install java-11-openjdk-devel.x86_64

if other version is already installed, use:

sudo alternatives --config java

yuvalif

Background

• initial project description
• code
• presentation

TODOs

Fixes

• use env variables for broker and S3 endpoint
• test with local mosquitto broker
• add SSL as parameter

yuvalif

from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
import threading
import time
import sys 


class RequestHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        print("active threads: ", str(threading.active_count()))
        self.send_response(200)

yuvalif

// extensions to: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/

type RGWNotificationConfigurationFilter struct {

    // A container for object key name prefix and suffix filtering rules.
    Key      *KeyFilter `locationName:"S3Key" type:"structure"`
    Metadata *RGWMapFilter `locationName:"RGWMetadata" type:"structure"`
    Tag      *RGWMapFilter `locationName:"RGWTag" type:"structure"`
    // contains filtered or unexported fields
}

yuvalif

-- calculate entropy of an object
function object_entropy(full_name)
                                                           
  local byte_hist = {}                                                                                                                                                                                                                        
  local byte_hist_size = 256 
  for i = 1,byte_hist_size do                                                                                          
    byte_hist[i] = 0                                                                                                   
  end                                                                                                                  
  local total = 0 
                                                                                                                       

yuvalif

# assuming we are running a vstart cluster
import subprocess
import time
import sys

def system(cmd):
  output = subprocess.check_output(cmd, shell=True).decode(sys.stdout.encoding).strip()
  return output

prev_rgw_sum = 0

yuvalif

Goal

The goal of this setup is to overload a single RGW so that adding another one would increase the throughput without overloaifng the OSDs.

Setup

• machine with multiple nvme drives and enough CPU/RAM to run both Ceph and the clients. e.g.

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 893.8G  0 disk 
└─sda1        8:1    0 893.8G  0 part /

yuvalif

start cluster:

MON=1 OSD=1 MDS=0 MGR=0 RGW=1 ../src/vstart.sh -n -d 

start HTTP endpoint:

wget https://gist.githubusercontent.com/mdonkers/63e115cc0c79b4f6b8b3a6b797e485c7/raw/a6a1d090ac8549dac8f2bd607bd64925de997d40/server.py
python server.py 10900

yuvalif

Delivery Guarantee

Due to the batching, there is a situation where messages reside inside the converter. The MQTT protocol does not allow for end 2 end acknowledgments, meaning that once the messages arrive at the converter, they are considered as “delivered”. Therefore, if the converter fails, the messages that were not yet uploaded into the S3 object are going to be lost. To make sure that delivery is guaranteed, we would need a mechanism that makes sure that the messages are not lost if the converter crashes while waiting for a batch to fill.

One option fo that would be to write every message to persistent media (e.g. disk) as it arrives. If a process restarts, it would read that file and send the data in it. However, this would have 2 main drawbacks:

• there will be a significant performance cost
• since mesages are automatically acked when received, everything in th disk buffer or that was not written yet will be lost on crash