Prior to running minio, the following preparations have to be made:
- format the drives
- mount the drives, it is suggested to use a "standardized" way of naming the mounts, eg. /mnt/drive1, /mnt/drive2, /mnt/drive3, etc.
- setup networking (see below)
Before running minio, it makes sense to verify the server to client networking performance, here are some pointers:
Set the MTU size to 9000 (jumbo frames):
$ ifconfig ens12345 mtu 9000
Be aware to use iperf2 (as opposed to iperf3), and use multiple threads (eg. 8 or 16).
On the server side:
$ iperf -s -P8
On the client side (expected output, for single 100 Gbit connection):
$ iperf -c 12.12.12.6 -P8
------------------------------------------------------------
Client connecting to 12.12.12.6, TCP port 5001
TCP window size: 22.5 KByte (default)
------------------------------------------------------------
[ 10] local 12.12.12.5 port 56143 connected with 12.12.12.6 port 5001
[ 6] local 12.12.12.5 port 56139 connected with 12.12.12.6 port 5001
......
[ 7] local 12.12.12.5 port 56141 connected with 12.12.12.6 port 5001
[ 9] local 12.12.12.5 port 56142 connected with 12.12.12.6 port 5001
[ ID] Interval Transfer Bandwidth
[ 10] 0.0-10.0 sec 15.4 GBytes 13.3 Gbits/sec
[ 6] 0.0-10.0 sec 15.3 GBytes 13.1 Gbits/sec
......
[ 7] 0.0-10.0 sec 15.4 GBytes 13.2 Gbits/sec
[ 9] 0.0-10.0 sec 15.4 GBytes 13.2 Gbits/sec
[SUM] 0.0-10.0 sec 109 GBytes 94.1 Gbits/sec
Depending on how the network is setup (eg. bonding or not), we may need to run multiple iperf jobs in parallel.
$ wget https://s3-us-west-2.amazonaws.com/minio-ppc64le/ppc64le/minio
$ chmod +x minio
$ MINIO_ACCESS_KEY=minio MINIO_SECRET_KEY=minio123 MINIO_STORAGE_CLASS_STANDARD=EC:2 ./minio server /mnt/drive{1...16}
Expected result:
Status: 16 Online, 0 Offline.
Endpoint: http://192.168.1.7:9000 http://127.0.0.1:9000
AccessKey: U09DX0NB71K0HBYRQGVD
SecretKey: PsqdAhZD9tYkZ4+eQL0lOywKMoWnGTDpQNeoF47K
Region: us-east-1
Browser Access:
http://192.168.1.7:9000 http://127.0.0.1:9000
Command-line Access:
$ mc config host add myminio http://192.168.1.7:9000 U09DX0NB71K0HBYRQGVD PsqdAhZD9tYkZ4+eQL0lOywKMoWnGTDpQNeoF47K