harshavardhana

Previous

    Outbound
    ┌───────────────────┐             ┌────────────────┐               ┌───────────────┐                      ┌────────────────┐
    │                   │   Parr.     │                │  (http body)  │               │                      │                │
    │ Bitrot Hash       │     Write   │      Pipe      │      Read     │  HTTP buffer  │    Write (syscall)   │  TCP Buffer    │
    │ Erasure Shard     │ ──────────► │  (unbuffered)  │ ────────────► │   (64K Max)   │ ───────────────────► │    (4MB)       │
    │                   │             │                │               │  (io.Copy)    │                      │                │

harshavardhana

version: '3.7'

# Settings and configurations that are common for all containers
x-minio-common: &minio-common
  image: quay.io/minio/minio:${RELEASE}
  command: server http://site1-minio{1...4}/data{1...2}
  environment:
    - MINIO_PROMETHEUS_AUTH_TYPE=public
    - CI=true

harshavardhana

"""
FTP benchmark.

Usage:
    ftpbench --help
    ftpbench -h <host> -u <user> -p <password> [options] login
    ftpbench -h <host> -u <user> -p <password> [options] upload <workdir> [-s <size>]
    ftpbench -h <host> -u <user> -p <password> [options] download <workdir> [-s <size>] [--files <count>]

Connection options:

harshavardhana

#!/bin/bash                                                                                                                                                                                                       \
                                                                                                                                                                                                                   
i=1

sudo rm -f /tmp/fstab
for disk in $(lsblk -i -p -n -o NAME | grep -v 'loop\|nvme0\|nvme1\|nvme2\|nvme3\|ubuntu\|md\|sda' | sort); do
    sudo mkdir -p /disk${i}
    sudo mkfs.xfs -f -L disk${i} $disk
    echo "LABEL=disk${i} /disk${i} xfs defaults,noatime 0 0" | sudo tee -a /tmp/fstab
    i=$(( $i + 1 ))

harshavardhana

Spark-shell with S3A based checkpointing

Spark context Web UI available at http://nirisvara:4040
Spark context available as 'sc' (master = local[*], app id = local-1677271782301).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
 _\ \/ _ \/ _ `/ __/ '_/

harshavardhana

Kernel Tuning for MinIO Production Deployment on Linux Servers

List of kernel tuning recommendations for MinIO server deployments, a copy of this script is available here that can be applied on the servers.

NOTE: These are common recommendations on any Linux servers, users must be careful on any premature optimization. These settings are not mandatory, these settings do not fix any hardware issues and should not be considered as an alternative to boost performance to mask underlying hardware problems. Under all circumstances it is recommended to perform this tuning only after performing benchmarks for the hardware with expected baseline results.

harshavardhana

Google Public NTP [AS15169]:

time.google.com

time1.google.com

time2.google.com

time3.google.com

harshavardhana

package deadlineconn

// DeadlineConn - is a generic stream-oriented network connection supporting buffered reader and read/write timeout.
type DeadlineConn struct {
	net.Conn
	readTimeout            time.Duration // sets the read timeout in the connection.
	writeTimeout           time.Duration // sets the write timeout in the connection.
}

// Sets read timeout

harshavardhana

[pvs]
# hostname local to this system on k8s
node=host1
# capacity for all devices this is common for all devices.
capacity=400Gi
# list of devices to format in command separated list.
devices=/dev/sdb

[minio]
release=RELEASE.2022-07-15T03-44-22Z

harshavardhana

package main

import (
	"crypto/tls"
	"fmt"
	"net"
	"net/http"
	"os"
	"strconv"
	"time"