Spark context Web UI available at http://nirisvara:4040
Spark context available as 'sc' (master = local[*], app id = local-1677271782301).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
version: '3.7' | |
# Settings and configurations that are common for all containers | |
x-minio-common: &minio-common | |
image: quay.io/minio/minio:${RELEASE} | |
command: server http://site1-minio{1...4}/data{1...2} | |
environment: | |
- MINIO_PROMETHEUS_AUTH_TYPE=public | |
- CI=true |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
""" | |
FTP benchmark. | |
Usage: | |
ftpbench --help | |
ftpbench -h <host> -u <user> -p <password> [options] login | |
ftpbench -h <host> -u <user> -p <password> [options] upload <workdir> [-s <size>] | |
ftpbench -h <host> -u <user> -p <password> [options] download <workdir> [-s <size>] [--files <count>] | |
Connection options: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash \ | |
i=1 | |
sudo rm -f /tmp/fstab | |
for disk in $(lsblk -i -p -n -o NAME | grep -v 'loop\|nvme0\|nvme1\|nvme2\|nvme3\|ubuntu\|md\|sda' | sort); do | |
sudo mkdir -p /disk${i} | |
sudo mkfs.xfs -f -L disk${i} $disk | |
echo "LABEL=disk${i} /disk${i} xfs defaults,noatime 0 0" | sudo tee -a /tmp/fstab | |
i=$(( $i + 1 )) |
List of kernel tuning recommendations for MinIO server deployments, a copy of this script is available here that can be applied on the servers.
NOTE: These are common recommendations on any Linux servers, users must be careful on any premature optimization. These settings are not mandatory, these settings do not fix any hardware issues and should not be considered as an alternative to boost performance to mask underlying hardware problems. Under all circumstances it is recommended to perform this tuning only after performing benchmarks for the hardware with expected baseline results.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package deadlineconn | |
// DeadlineConn - is a generic stream-oriented network connection supporting buffered reader and read/write timeout. | |
type DeadlineConn struct { | |
net.Conn | |
readTimeout time.Duration // sets the read timeout in the connection. | |
writeTimeout time.Duration // sets the write timeout in the connection. | |
} | |
// Sets read timeout |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[pvs] | |
# hostname local to this system on k8s | |
node=host1 | |
# capacity for all devices this is common for all devices. | |
capacity=400Gi | |
# list of devices to format in command separated list. | |
devices=/dev/sdb | |
[minio] | |
release=RELEASE.2022-07-15T03-44-22Z |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package main | |
import ( | |
"crypto/tls" | |
"fmt" | |
"net" | |
"net/http" | |
"os" | |
"strconv" | |
"time" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
ENGINE=sync ## can be changed to psync as well | |
RESULT_FILE=/tmp/result_$$ | |
rm -rf $HOME/tmp-fio | |
mkdir -p $HOME/tmp-fio && cd $HOME/tmp-fio | |
echo "Starting storage tests" | |
echo "Results will be saved in $RESULT_FILE" |
NewerOlder