Skip to content

Instantly share code, notes, and snippets.

View build-wireshark-fedora36.md

prerequisites

sudo dnf install glib2-devel
sudo dnf install libgcrypt-devel
sudo dnf install qt-devel
sudo dnf install qt6-qtbase-devel
sudo dnf install qt6-linguist-devel
sudo dnf install qt6-qttools-devel
sudo dnf install qt6-qt5compat-devel
View build-apache-kafka.md

Instructions for fedora 36

Java

install java. curently gradle does not work with jdk higher than 11, so we would need to:

sudo dnf install java-11-openjdk-devel.x86_64

if other version is already installed, use:

sudo alternatives --config java
View mqtt-converter-todo.md
View delayed_http_server.py
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
import threading
import time
import sys
class RequestHandler(BaseHTTPRequestHandler):
def do_GET(self):
print("active threads: ", str(threading.active_count()))
self.send_response(200)
View rgw_notification_extensions.go
// extensions to: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/
type RGWNotificationConfigurationFilter struct {
// A container for object key name prefix and suffix filtering rules.
Key *KeyFilter `locationName:"S3Key" type:"structure"`
Metadata *RGWMapFilter `locationName:"RGWMetadata" type:"structure"`
Tag *RGWMapFilter `locationName:"RGWTag" type:"structure"`
// contains filtered or unexported fields
}
View ransomware.lua
-- calculate entropy of an object
function object_entropy(full_name)
local byte_hist = {}
local byte_hist_size = 256
for i = 1,byte_hist_size do
byte_hist[i] = 0
end
local total = 0
View measure-rgw-latency.py
# assuming we are running a vstart cluster
import subprocess
import time
import sys
def system(cmd):
output = subprocess.check_output(cmd, shell=True).decode(sys.stdout.encoding).strip()
return output
prev_rgw_sum = 0
View manual-rgw-scaling.md

Goal

The goal of this setup is to overload a single RGW so that adding another one would increase the throughput without overloaifng the OSDs.

Setup

  • machine with multiple nvme drives and enough CPU/RAM to run both Ceph and the clients. e.g.
$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 893.8G  0 disk 
└─sda1        8:1    0 893.8G  0 part /
View notifications_test_http.md

start cluster:

MON=1 OSD=1 MDS=0 MGR=0 RGW=1 ../src/vstart.sh -n -d 

start HTTP endpoint:

wget https://gist.githubusercontent.com/mdonkers/63e115cc0c79b4f6b8b3a6b797e485c7/raw/a6a1d090ac8549dac8f2bd607bd64925de997d40/server.py
python server.py 10900
View delivery-guarantee.md

Delivery Guarantee

Due to the batching, there is a situation where messages reside inside the converter. The MQTT protocol does not allow for end 2 end acknowledgments, meaning that once the messages arrive at the converter, they are considered as “delivered”. Therefore, if the converter fails, the messages that were not yet uploaded into the S3 object are going to be lost. To make sure that delivery is guaranteed, we would need a mechanism that makes sure that the messages are not lost if the converter crashes while waiting for a batch to fill.

One option fo that would be to write every message to persistent media (e.g. disk) as it arrives. If a process restarts, it would read that file and send the data in it. However, this would have 2 main drawbacks:

  • there will be a significant performance cost
  • since mesages are automatically acked when received, everything in th disk buffer or that was not written yet will be lost on crash