Skip to content

Instantly share code, notes, and snippets.

@kvaps

kvaps/test_big_files.sh

Last active May 22, 2020 22:05
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kvaps/c93ab87f751781a69f990570aee8c262 to your computer and use it in GitHub Desktop.
Save kvaps/c93ab87f751781a69f990570aee8c262 to your computer and use it in GitHub Desktop.
Download ZIP
minio ec:2 vs raid6 (16 drives / 2 parity / 20GB LACP / ST6000NM0115-1YZ110)
Raw
test_big_files.sh
# s3bench -accessKey=minioadmin -accessSecret=minioadmin -bucket=bench -endpoint=http://minio:9001/ -numSamples=1000 -numClients=100
Test parameters Test parameters
endpoint(s): [minio-ec2] | endpoint(s): [minio-raid6]
bucket: bench bucket: bench
objectNamePrefix: loadgen_test_ objectNamePrefix: loadgen_test_
objectSize: 80.0000 MB objectSize: 80.0000 MB
numClients: 100 numClients: 100
numSamples: 1000 numSamples: 1000
Results Summary for Write Operation(s) Results Summary for Write Operation(s)
Total Transferred: 80000.000 MB Total Transferred: 80000.000 MB
Total Throughput: 828.71 MB/s | Total Throughput: 751.34 MB/s
Total Duration: 96.536 s | Total Duration: 106.477 s
Number of Errors: 0 Number of Errors: 0
------------------------------------ ------------------------------------
Write times Max: 12.154 s | Write times Max: 13.469 s
Write times 99th %ile: 11.771 s | Write times 99th %ile: 12.962 s
Write times 90th %ile: 11.308 s | Write times 90th %ile: 12.162 s
Write times 75th %ile: 11.011 s | Write times 75th %ile: 11.627 s
Write times 50th %ile: 10.742 s | Write times 50th %ile: 10.855 s
Write times 25th %ile: 7.896 s | Write times 25th %ile: 8.827 s
Write times Min: 1.237 s | Write times Min: 1.328 s
Results Summary for Read Operation(s) Results Summary for Read Operation(s)
Total Transferred: 80000.000 MB Total Transferred: 80000.000 MB
Total Throughput: 918.79 MB/s | Total Throughput: 635.28 MB/s
Total Duration: 87.071 s | Total Duration: 125.930 s
Number of Errors: 0 Number of Errors: 0
------------------------------------ ------------------------------------
Read times Max: 13.957 s | Read times Max: 28.049 s
Read times 99th %ile: 12.441 s | Read times 99th %ile: 26.429 s
Read times 90th %ile: 10.735 s | Read times 90th %ile: 23.405 s
Read times 75th %ile: 9.711 s | Read times 75th %ile: 19.747 s
Read times 50th %ile: 8.486 s | Read times 50th %ile: 8.867 s
Read times 25th %ile: 7.300 s | Read times 25th %ile: 7.883 s
Read times Min: 4.246 s | Read times Min: 4.124 s
Raw
test_small_files.sh
# s3bench -accessKey=minioadmin -accessSecret=minioadmin -bucket=bench -endpoint=http://minio:9001/ -numSamples=100000 -numClients=100 -objectSize=1048576
Test parameters Test parameters
endpoint(s): [minio-ec2] | endpoint(s): [minio-raid6]
bucket: bench bucket: bench
objectNamePrefix: loadgen_test_ objectNamePrefix: loadgen_test_
objectSize: 1.0000 MB objectSize: 1.0000 MB
numClients: 100 numClients: 100
numSamples: 100000 numSamples: 100000
Results Summary for Write Operation(s) Results Summary for Write Operation(s)
Total Transferred: 100000.000 MB Total Transferred: 100000.000 MB
Total Throughput: 119.53 MB/s | Total Throughput: 345.84 MB/s
Total Duration: 836.644 s | Total Duration: 289.155 s
Number of Errors: 0 Number of Errors: 0
------------------------------------ ------------------------------------
Write times Max: 4.702 s | Write times Max: 6.427 s
Write times 99th %ile: 2.047 s | Write times 99th %ile: 1.766 s
Write times 90th %ile: 1.314 s | Write times 90th %ile: 0.580 s
Write times 75th %ile: 1.013 s | Write times 75th %ile: 0.343 s
Write times 50th %ile: 0.742 s | Write times 50th %ile: 0.171 s
Write times 25th %ile: 0.564 s | Write times 25th %ile: 0.129 s
Write times Min: 0.170 s | Write times Min: 0.055 s
Results Summary for Read Operation(s) Results Summary for Read Operation(s)
Total Transferred: 100000.000 MB Total Transferred: 100000.000 MB
Total Throughput: 633.27 MB/s | Total Throughput: 915.16 MB/s
Total Duration: 157.911 s | Total Duration: 109.270 s
Number of Errors: 0 Number of Errors: 0
------------------------------------ ------------------------------------
Read times Max: 8.625 s | Read times Max: 7.819 s
Read times 99th %ile: 0.506 s | Read times 99th %ile: 0.431 s
Read times 90th %ile: 0.300 s | Read times 90th %ile: 0.253 s
Read times 75th %ile: 0.194 s | Read times 75th %ile: 0.115 s
Read times 50th %ile: 0.126 s | Read times 50th %ile: 0.072 s
Read times 25th %ile: 0.088 s | Read times 25th %ile: 0.052 s
Read times Min: 0.008 s | Read times Min: 0.006 s
Raw
test_small_files_ec2_zfs.sh
# s3bench -accessKey=minioadmin -accessSecret=minioadmin -bucket=bench -endpoint=http://minio:9001/ -numSamples=100000 -numClients=100 -objectSize=1048576
Test parameters
endpoint(s): [minio-ec2-zfs]
bucket: bench
objectNamePrefix: loadgen_test_
objectSize: 1.0000 MB
numClients: 100
numSamples: 100000
Results Summary for Write Operation(s)
Total Transferred: 100000.000 MB
Total Throughput: 162.74 MB/s
Total Duration: 614.466 s
Number of Errors: 0
------------------------------------
Write times Max: 2.721 s
Write times 99th %ile: 1.555 s
Write times 90th %ile: 0.911 s
Write times 75th %ile: 0.697 s
Write times 50th %ile: 0.555 s
Write times 25th %ile: 0.462 s
Write times Min: 0.115 s
Results Summary for Read Operation(s)
Total Transferred: 100000.000 MB
Total Throughput: 159.08 MB/s
Total Duration: 628.627 s
Number of Errors: 0
------------------------------------
Read times Max: 3.651 s
Read times 99th %ile: 0.960 s
Read times 90th %ile: 0.765 s
Read times 75th %ile: 0.686 s
Read times 50th %ile: 0.614 s
Read times 25th %ile: 0.551 s
Read times Min: 0.213 s
Raw
test_small_files_ec2_zfs_tune.sh
Test parameters
endpoint(s): [minio-ec2-zfs-tune]
bucket: bench
objectNamePrefix: loadgen_test_
objectSize: 1.0000 MB
numClients: 100
numSamples: 100000
Results Summary for Write Operation(s)
Total Transferred: 100000.000 MB
Total Throughput: 247.12 MB/s
Total Duration: 404.669 s
Number of Errors: 0
------------------------------------
Write times Max: 4.945 s
Write times 99th %ile: 1.245 s
Write times 90th %ile: 0.523 s
Write times 75th %ile: 0.426 s
Write times 50th %ile: 0.367 s
Write times 25th %ile: 0.320 s
Write times Min: 0.076 s
Results Summary for Read Operation(s)
Total Transferred: 100000.000 MB
Total Throughput: 393.50 MB/s
Total Duration: 254.130 s
Number of Errors: 0
------------------------------------
Read times Max: 1.313 s
Read times 99th %ile: 0.470 s
Read times 90th %ile: 0.344 s
Read times 75th %ile: 0.292 s
Read times 50th %ile: 0.245 s
Read times 25th %ile: 0.204 s
Read times Min: 0.053 s
Raw
test_small_files_less_clients.sh
# s3bench -accessKey=minioadmin -accessSecret=minioadmin -bucket=bench -endpoint=http://minio:9001/ -numSamples=100000 -numClients=8 -objectSize=1048576
Test parameters Test parameters
endpoint(s): [minio-ec2] | endpoint(s): [minio-raid6]
bucket: bench bucket: bench
objectNamePrefix: loadgen_test_ objectNamePrefix: loadgen_test_
objectSize: 1.0000 MB objectSize: 1.0000 MB
numClients: 8 numClients: 8
numSamples: 100000 numSamples: 100000
Results Summary for Write Operation(s) Results Summary for Write Operation(s)
Total Transferred: 100000.000 MB Total Transferred: 100000.000 MB
Total Throughput: 34.07 MB/s | Total Throughput: 88.08 MB/s
Total Duration: 2934.805 s | Total Duration: 1135.352 s
Number of Errors: 0 Number of Errors: 0
------------------------------------ ------------------------------------
Write times Max: 1.301 s | Write times Max: 1.673 s
Write times 99th %ile: 0.602 s | Write times 99th %ile: 0.477 s
Write times 90th %ile: 0.347 s | Write times 90th %ile: 0.112 s
Write times 75th %ile: 0.250 s | Write times 75th %ile: 0.086 s
Write times 50th %ile: 0.198 s | Write times 50th %ile: 0.075 s
Write times 25th %ile: 0.180 s | Write times 25th %ile: 0.067 s
Write times Min: 0.069 s | Write times Min: 0.034 s
Results Summary for Read Operation(s) Results Summary for Read Operation(s)
Total Transferred: 100000.000 MB Total Transferred: 100000.000 MB
Total Throughput: 289.12 MB/s | Total Throughput: 696.93 MB/s
Total Duration: 345.873 s | Total Duration: 143.486 s
Number of Errors: 0 Number of Errors: 0
------------------------------------ ------------------------------------
Read times Max: 1.576 s | Read times Max: 0.462 s
Read times 99th %ile: 0.076 s | Read times 99th %ile: 0.059 s
Read times 90th %ile: 0.043 s | Read times 90th %ile: 0.019 s
Read times 75th %ile: 0.034 s | Read times 75th %ile: 0.012 s
Read times 50th %ile: 0.025 s | Read times 50th %ile: 0.008 s
Read times 25th %ile: 0.017 s | Read times 25th %ile: 0.006 s
Read times Min: 0.004 s | Read times Min: 0.003 s
Raw
test_small_files_raidz2_zfs_vs_ext4-on-zvol.sh
# s3bench -accessKey=minioadmin -accessSecret=minioadmin -bucket=bench -endpoint=http://minio:9001/ -numSamples=100000 -numClients=100 -objectSize=1048576
Test parameters Test parameters
endpoint(s): [raidz2-minio] | endpoint(s): [raidz2-zvol-minio]
bucket: bench bucket: bench
objectNamePrefix: loadgen_test_ objectNamePrefix: loadgen_test_
objectSize: 1.0000 MB objectSize: 1.0000 MB
numClients: 100 numClients: 100
numSamples: 100000 numSamples: 100000
Results Summary for Write Operation(s) Results Summary for Write Operation(s)
Total Transferred: 100000.000 MB Total Transferred: 100000.000 MB
Total Throughput: 225.56 MB/s | Total Throughput: 117.12 MB/s
Total Duration: 443.331 s | Total Duration: 853.802 s
Number of Errors: 0 Number of Errors: 0
------------------------------------ ------------------------------------
Write times Max: 1.490 s | Write times Max: 51.051 s
Write times 99th %ile: 0.709 s | Write times 99th %ile: 19.007 s
Write times 90th %ile: 0.540 s | Write times 90th %ile: 0.577 s
Write times 75th %ile: 0.482 s | Write times 75th %ile: 0.433 s
Write times 50th %ile: 0.431 s | Write times 50th %ile: 0.303 s
Write times 25th %ile: 0.389 s | Write times 25th %ile: 0.195 s
Write times Min: 0.075 s | Write times Min: 0.042 s
Results Summary for Read Operation(s) Results Summary for Read Operation(s)
Total Transferred: 100000.000 MB Total Transferred: 100000.000 MB
Total Throughput: 510.01 MB/s | Total Throughput: 96.04 MB/s
Total Duration: 196.073 s | Total Duration: 1041.196 s
Number of Errors: 0 Number of Errors: 0
------------------------------------ ------------------------------------
Read times Max: 6.194 s | Read times Max: 24.683 s
Read times 99th %ile: 0.469 s | Read times 99th %ile: 1.924 s
Read times 90th %ile: 0.308 s | Read times 90th %ile: 1.243 s
Read times 75th %ile: 0.244 s | Read times 75th %ile: 1.092 s
Read times 50th %ile: 0.185 s | Read times 50th %ile: 0.975 s
Read times 25th %ile: 0.131 s | Read times 25th %ile: 0.885 s
Read times Min: 0.010 s | Read times Min: 0.355 s
Raw
test_tiny_files.sh
# s3bench -accessKey=minioadmin -accessSecret=minioadmin -bucket=bench -endpoint=http://minio:9001/ -numSamples=100000 -numClients=100 -objectSize=1024
Test parameters Test parameters
endpoint(s): [minio-ec2] | endpoint(s): [minio-raid6]
bucket: bench bucket: bench
objectNamePrefix: loadgen_test_ objectNamePrefix: loadgen_test_
objectSize: 0.0010 MB objectSize: 0.0010 MB
numClients: 100 numClients: 100
numSamples: 100000 numSamples: 100000
Results Summary for Write Operation(s) Results Summary for Write Operation(s)
Total Transferred: 97.656 MB Total Transferred: 97.656 MB
Total Throughput: 0.15 MB/s | Total Throughput: 0.39 MB/s
Total Duration: 643.246 s | Total Duration: 251.628 s
Number of Errors: 0 Number of Errors: 0
------------------------------------ ------------------------------------
Write times Max: 3.240 s | Write times Max: 3.277 s
Write times 99th %ile: 1.674 s | Write times 99th %ile: 1.917 s
Write times 90th %ile: 1.059 s | Write times 90th %ile: 0.354 s
Write times 75th %ile: 0.758 s | Write times 75th %ile: 0.236 s
Write times 50th %ile: 0.566 s | Write times 50th %ile: 0.162 s
Write times 25th %ile: 0.433 s | Write times 25th %ile: 0.119 s
Write times Min: 0.155 s | Write times Min: 0.040 s
Results Summary for Read Operation(s) Results Summary for Read Operation(s)
Total Transferred: 97.656 MB | Total Transferred: 97.231 MB
Total Throughput: 1.58 MB/s | Total Throughput: 1.82 MB/s
Total Duration: 61.834 s | Total Duration: 53.540 s
Number of Errors: 0 | Number of Errors: 435
------------------------------------ ------------------------------------
Read times Max: 1.046 s | Read times Max: 1.830 s
Read times 99th %ile: 0.361 s | Read times 99th %ile: 0.370 s
Read times 90th %ile: 0.129 s | Read times 90th %ile: 0.107 s
Read times 75th %ile: 0.068 s | Read times 75th %ile: 0.046 s
Read times 50th %ile: 0.044 s | Read times 50th %ile: 0.028 s
Read times 25th %ile: 0.022 s | Read times 25th %ile: 0.018 s
Read times Min: 0.003 s | Read times Min: 0.001 s
Raw
zz_server.sh
mkfs.ext4 -m0 /dev/sdu -F
mkfs.ext4 -m0 /dev/sdv -F
mkfs.ext4 -m0 /dev/sdw -F
mkfs.ext4 -m0 /dev/sdx -F
mkfs.ext4 -m0 /dev/sdy -F
mkfs.ext4 -m0 /dev/sdz -F
mkfs.ext4 -m0 /dev/sdaa -F
mkfs.ext4 -m0 /dev/sdab -F
mkfs.ext4 -m0 /dev/sdac -F
mkfs.ext4 -m0 /dev/sdad -F
mkfs.ext4 -m0 /dev/sdae -F
mkfs.ext4 -m0 /dev/sdaf -F
mkfs.ext4 -m0 /dev/sdag -F
mkfs.ext4 -m0 /dev/sdah -F
mkfs.ext4 -m0 /dev/sdai -F
mkfs.ext4 -m0 /dev/sdaj -F
mkfs.xfs /dev/sdu -f
mkfs.xfs /dev/sdv -f
mkfs.xfs /dev/sdw -f
mkfs.xfs /dev/sdx -f
mkfs.xfs /dev/sdy -f
mkfs.xfs /dev/sdz -f
mkfs.xfs /dev/sdaa -f
mkfs.xfs /dev/sdab -f
mkfs.xfs /dev/sdac -f
mkfs.xfs /dev/sdad -f
mkfs.xfs /dev/sdae -f
mkfs.xfs /dev/sdaf -f
mkfs.xfs /dev/sdag -f
mkfs.xfs /dev/sdah -f
mkfs.xfs /dev/sdai -f
mkfs.xfs /dev/sdaj -f
mkdir -p /mnt/sdu
mkdir -p /mnt/sdv
mkdir -p /mnt/sdw
mkdir -p /mnt/sdx
mkdir -p /mnt/sdy
mkdir -p /mnt/sdz
mkdir -p /mnt/sdaa
mkdir -p /mnt/sdab
mkdir -p /mnt/sdac
mkdir -p /mnt/sdad
mkdir -p /mnt/sdae
mkdir -p /mnt/sdaf
mkdir -p /mnt/sdag
mkdir -p /mnt/sdah
mkdir -p /mnt/sdai
mkdir -p /mnt/sdaj
mount /dev/sdu /mnt/sdu
mount /dev/sdv /mnt/sdv
mount /dev/sdw /mnt/sdw
mount /dev/sdx /mnt/sdx
mount /dev/sdy /mnt/sdy
mount /dev/sdz /mnt/sdz
mount /dev/sdaa /mnt/sdaa
mount /dev/sdab /mnt/sdab
mount /dev/sdac /mnt/sdac
mount /dev/sdad /mnt/sdad
mount /dev/sdae /mnt/sdae
mount /dev/sdaf /mnt/sdaf
mount /dev/sdag /mnt/sdag
mount /dev/sdah /mnt/sdah
mount /dev/sdai /mnt/sdai
mount /dev/sdaj /mnt/sdaj
minio server --address :9001 \
/mnt/sdu /mnt/sdv /mnt/sdw /mnt/sdx /mnt/sdy /mnt/sdz /mnt/sdaa /mnt/sdab \
/mnt/sdac /mnt/sdad /mnt/sdae /mnt/sdaf /mnt/sdag /mnt/sdah /mnt/sdai /mnt/sdaj
umount /mnt/sdu
umount /mnt/sdv
umount /mnt/sdw
umount /mnt/sdx
umount /mnt/sdy
umount /mnt/sdz
umount /mnt/sdaa
umount /mnt/sdab
umount /mnt/sdac
umount /mnt/sdad
umount /mnt/sdae
umount /mnt/sdaf
umount /mnt/sdag
umount /mnt/sdah
umount /mnt/sdai
umount /mnt/sdaj
mdadm --create --assume-clean --verbose /dev/md6 --level=6 --raid-devices=16 \
/dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz /dev/sdaa /dev/sdab \
/dev/sdac /dev/sdad /dev/sdae /dev/sdaf /dev/sdag /dev/sdah /dev/sdai /dev/sdaj
mkfs.ext4 -m0 /dev/md6 -F
mkfs.xfs /dev/md6 -f
mkdir /mnt/md6
mount /dev/md6 /mnt/md6
minio server --address :9001 /mnt/md6
umount /mnt/md6
mdadm --stop /dev/md6
mdadm --zero-superblock /dev/sdu
mdadm --zero-superblock /dev/sdv
mdadm --zero-superblock /dev/sdw
mdadm --zero-superblock /dev/sdx
mdadm --zero-superblock /dev/sdy
mdadm --zero-superblock /dev/sdz
mdadm --zero-superblock /dev/sdaa
mdadm --zero-superblock /dev/sdab
mdadm --zero-superblock /dev/sdac
mdadm --zero-superblock /dev/sdad
mdadm --zero-superblock /dev/sdae
mdadm --zero-superblock /dev/sdaf
mdadm --zero-superblock /dev/sdag
mdadm --zero-superblock /dev/sdah
mdadm --zero-superblock /dev/sdai
mdadm --zero-superblock /dev/sdaj
zpool create -f tank raidz2 \
/dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz /dev/sdaa /dev/sdab \
/dev/sdac /dev/sdad /dev/sdae /dev/sdaf /dev/sdag /dev/sdah /dev/sdai /dev/sdaj
minio server --address :9001 /tank
zfs create -V 60T tank/test
mkfs.ext4 -m0 /dev/zvol/tank/test -F
mkdir /mnt/zvol
mount /dev/zvol/tank/test /mnt/zvol/
minio server --address :9001 /mnt/zvol
umount /mnt/zvol/
zfs destroy tank/test
zpool destroy tank
zpool create -f sdu -m /mnt/sdu /dev/sdu
zpool create -f sdv -m /mnt/sdv /dev/sdv
zpool create -f sdw -m /mnt/sdw /dev/sdw
zpool create -f sdx -m /mnt/sdx /dev/sdx
zpool create -f sdy -m /mnt/sdy /dev/sdy
zpool create -f sdz -m /mnt/sdz /dev/sdz
zpool create -f sdaa -m /mnt/sdaa /dev/sdaa
zpool create -f sdab -m /mnt/sdab /dev/sdab
zpool create -f sdac -m /mnt/sdac /dev/sdac
zpool create -f sdad -m /mnt/sdad /dev/sdad
zpool create -f sdae -m /mnt/sdae /dev/sdae
zpool create -f sdaf -m /mnt/sdaf /dev/sdaf
zpool create -f sdag -m /mnt/sdag /dev/sdag
zpool create -f sdah -m /mnt/sdah /dev/sdah
zpool create -f sdai -m /mnt/sdai /dev/sdai
zpool create -f sdaj -m /mnt/sdaj /dev/sdaj
minio server --address :9001 \
/mnt/sdu /mnt/sdv /mnt/sdw /mnt/sdx /mnt/sdy /mnt/sdz /mnt/sdaa /mnt/sdab \
/mnt/sdac /mnt/sdad /mnt/sdae /mnt/sdaf /mnt/sdag /mnt/sdah /mnt/sdai /mnt/sdaj
zpool destroy -f sdu
zpool destroy -f sdv
zpool destroy -f sdw
zpool destroy -f sdx
zpool destroy -f sdy
zpool destroy -f sdz
zpool destroy -f sdaa
zpool destroy -f sdab
zpool destroy -f sdac
zpool destroy -f sdad
zpool destroy -f sdae
zpool destroy -f sdaf
zpool destroy -f sdag
zpool destroy -f sdah
zpool destroy -f sdai
zpool destroy -f sdaj
zfs set recordsize=1M xattr=sa atime=off sdu
zfs set recordsize=1M xattr=sa atime=off sdv
zfs set recordsize=1M xattr=sa atime=off sdw
zfs set recordsize=1M xattr=sa atime=off sdx
zfs set recordsize=1M xattr=sa atime=off sdy
zfs set recordsize=1M xattr=sa atime=off sdz
zfs set recordsize=1M xattr=sa atime=off sdaa
zfs set recordsize=1M xattr=sa atime=off sdab
zfs set recordsize=1M xattr=sa atime=off sdac
zfs set recordsize=1M xattr=sa atime=off sdad
zfs set recordsize=1M xattr=sa atime=off sdae
zfs set recordsize=1M xattr=sa atime=off sdaf
zfs set recordsize=1M xattr=sa atime=off sdag
zfs set recordsize=1M xattr=sa atime=off sdah
zfs set recordsize=1M xattr=sa atime=off sdai
zfs set recordsize=1M xattr=sa atime=off sdaj
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment