ioping -WWW -c 200 -i 0 -L -s 4K /mnt/
...
--- /mnt/ (xfs /dev/nvme0n1) ioping statistics ---
199 requests completed in 14.1 ms, 796 KiB written, 14.1 k iops, 55.3 MiB/s
generated 200 requests in 16.1 ms, 800 KiB, 12.5 k iops, 48.7 MiB/s
min/avg/max/mdev = 61.4 us / 70.7 us / 273.9 us / 16.9 us
ioping -GGG -c 200 -i 0 -L -s 4K /mnt/
...
--- /mnt/ (xfs /dev/nvme0n1) ioping statistics ---
199 requests completed in 12.4 ms, 796 KiB, 16.1 k iops, 62.8 MiB/s
generated 200 requests in 14.6 ms, 800 KiB, 13.7 k iops, 53.6 MiB/s
min/avg/max/mdev = 44.5 us / 62.2 us / 118.1 us / 13.3 us
# cat test_random_read_iops.fio
[global]
bs=4K
iodepth=256
direct=1
ioengine=libaio
group_reporting
time_based
numjobs=2
runtime=120
size=15g
name=raw-randread
rw=randread
[job1]
filename=nvme0n1
# fio test_random_read_iops.fio
job1: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [r(2)][100.0%][r=704MiB/s,w=0KiB/s][r=180k,w=0 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=2): err= 0: pid=5146: Mon Mar 8 11:42:41 2021
read: IOPS=180k, BW=703MiB/s (737MB/s)(82.4GiB/120004msec)
slat (nsec): min=1379, max=486739, avg=3358.70, stdev=2627.55
clat (usec): min=107, max=20476, avg=2840.08, stdev=481.73
lat (usec): min=126, max=20486, avg=2843.52, stdev=481.82
clat percentiles (usec):
| 1.00th=[ 832], 5.00th=[ 2114], 10.00th=[ 2245], 20.00th=[ 2606],
| 30.00th=[ 2737], 40.00th=[ 2868], 50.00th=[ 2966], 60.00th=[ 2999],
| 70.00th=[ 3032], 80.00th=[ 3097], 90.00th=[ 3195], 95.00th=[ 3326],
| 99.00th=[ 4080], 99.50th=[ 4293], 99.90th=[ 5145], 99.95th=[ 5342],
| 99.99th=[ 6718]
bw ( KiB/s): min=351560, max=368584, per=50.00%, avg=360034.40, stdev=2419.14, samples=480
iops : min=87890, max=92146, avg=90008.61, stdev=604.79, samples=480
lat (usec) : 250=0.12%, 500=0.43%, 750=0.34%, 1000=0.29%
lat (msec) : 2=1.03%, 4=96.35%, 10=1.42%, 20=0.01%, 50=0.01%
cpu : usr=12.22%, sys=31.89%, ctx=845386, majf=0, minf=534
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=21603661,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=703MiB/s (737MB/s), 703MiB/s-703MiB/s (737MB/s-737MB/s), io=82.4GiB (88.5GB), run=120004-120004msec
Disk stats (read/write):
nvme0n1: ios=21582609/0, merge=0/0, ticks=58774845/0, in_queue=45678796, util=100.00%
Test file random read/write # cat test_random_read_write_iops.fio
[global]
bs=4K
iodepth=256
direct=1
ioengine=libaio
group_reporting
time_based
numjobs=2
runtime=120
size=15g
name=raw-randrw
rw=randrw
[job1]
filename=nvme0n1
# fio test_random_read_write_iops.fio
job1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [m(2)][100.0%][r=251MiB/s,w=252MiB/s][r=64.2k,w=64.4k IOPS][eta 00m:00s]
job1: (groupid=0, jobs=2): err= 0: pid=5509: Mon Mar 8 11:46:47 2021
read: IOPS=64.3k, BW=251MiB/s (263MB/s)(29.4GiB/120007msec)
slat (nsec): min=1389, max=343016, avg=4127.89, stdev=3008.80
clat (usec): min=43, max=22151, avg=3990.37, stdev=3621.65
lat (usec): min=46, max=22161, avg=3994.61, stdev=3621.71
clat percentiles (usec):
| 1.00th=[ 169], 5.00th=[ 231], 10.00th=[ 289], 20.00th=[ 433],
| 30.00th=[ 906], 40.00th=[ 1156], 50.00th=[ 2999], 60.00th=[ 5080],
| 70.00th=[ 6915], 80.00th=[ 7242], 90.00th=[ 8717], 95.00th=[10683],
| 99.00th=[13304], 99.50th=[14091], 99.90th=[15270], 99.95th=[15664],
| 99.99th=[16188]
bw ( KiB/s): min=114056, max=140384, per=50.00%, avg=128573.02, stdev=4412.63, samples=480
iops : min=28514, max=35096, avg=32143.23, stdev=1103.14, samples=480
write: IOPS=64.3k, BW=251MiB/s (263MB/s)(29.4GiB/120007msec)
slat (nsec): min=1527, max=304382, avg=4594.19, stdev=3167.48
clat (usec): min=45, max=61038, avg=3963.79, stdev=3612.98
lat (usec): min=51, max=61040, avg=3968.49, stdev=3613.05
clat percentiles (usec):
| 1.00th=[ 133], 5.00th=[ 243], 10.00th=[ 310], 20.00th=[ 523],
| 30.00th=[ 840], 40.00th=[ 1074], 50.00th=[ 2933], 60.00th=[ 5014],
| 70.00th=[ 6783], 80.00th=[ 7177], 90.00th=[ 8586], 95.00th=[10683],
| 99.00th=[13304], 99.50th=[13960], 99.90th=[15270], 99.95th=[15795],
| 99.99th=[17695]
bw ( KiB/s): min=114992, max=143416, per=50.00%, avg=128556.69, stdev=4329.16, samples=480
iops : min=28748, max=35854, avg=32139.16, stdev=1082.29, samples=480
lat (usec) : 50=0.01%, 100=0.21%, 250=5.77%, 500=14.87%, 750=5.97%
lat (usec) : 1000=8.88%
lat (msec) : 2=9.73%, 4=9.67%, 10=38.79%, 20=6.11%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=11.15%, sys=29.89%, ctx=489789, majf=0, minf=31
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=7715158,7714163,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=251MiB/s (263MB/s), 251MiB/s-251MiB/s (263MB/s-263MB/s), io=29.4GiB (31.6GB), run=120007-120007msec
WRITE: bw=251MiB/s (263MB/s), 251MiB/s-251MiB/s (263MB/s-263MB/s), io=29.4GiB (31.6GB), run=120007-120007msec
Disk stats (read/write):
nvme0n1: ios=7707587/7706636, merge=0/3, ticks=29639624/29002029, in_queue=50970659, util=100.00%
# cat test_sequential_read_iops.fio
[global]
bs=4K
iodepth=256
direct=1
ioengine=libaio
group_reporting
time_based
numjobs=2
runtime=120
size=15g
name=raw-seqread
rw=read
[job1]
filename=nvme0n1
# fio test_sequential_read_iops.fio
job1: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [R(2)][100.0%][r=704MiB/s,w=0KiB/s][r=180k,w=0 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=2): err= 0: pid=5951: Mon Mar 8 11:51:55 2021
read: IOPS=180k, BW=703MiB/s (737MB/s)(82.4GiB/120004msec)
slat (nsec): min=1361, max=3174.7k, avg=3281.28, stdev=2649.75
clat (usec): min=116, max=15013, avg=2840.29, stdev=496.78
lat (usec): min=123, max=15020, avg=2843.65, stdev=496.87
clat percentiles (usec):
| 1.00th=[ 799], 5.00th=[ 2057], 10.00th=[ 2212], 20.00th=[ 2540],
| 30.00th=[ 2737], 40.00th=[ 2835], 50.00th=[ 2933], 60.00th=[ 2999],
| 70.00th=[ 3064], 80.00th=[ 3130], 90.00th=[ 3261], 95.00th=[ 3359],
| 99.00th=[ 4080], 99.50th=[ 4293], 99.90th=[ 5145], 99.95th=[ 5276],
| 99.99th=[ 6128]
bw ( KiB/s): min=349872, max=368702, per=50.00%, avg=360024.16, stdev=2934.99, samples=480
iops : min=87468, max=92175, avg=90006.03, stdev=733.75, samples=480
lat (usec) : 250=0.08%, 500=0.47%, 750=0.38%, 1000=0.27%
lat (msec) : 2=2.30%, 4=95.12%, 10=1.37%, 20=0.01%
cpu : usr=11.03%, sys=31.34%, ctx=897476, majf=0, minf=539
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=21603764,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=703MiB/s (737MB/s), 703MiB/s-703MiB/s (737MB/s-737MB/s), io=82.4GiB (88.5GB), run=120004-120004msec
Disk stats (read/write):
nvme0n1: ios=21582883/7, merge=0/3, ticks=59605757/6, in_queue=47956580, util=100.00%
Throughput performance tests # cat test_random_read_throughput.fio
[global]
bs=64K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
numjobs=2
runtime=120
size=15g
name=raw-randread
rw=randread
[job1]
filename=nvme0n1
# fio test_random_read_throughput.fio
job1: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [r(2)][100.0%][r=704MiB/s,w=0KiB/s][r=11.3k,w=0 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=2): err= 0: pid=6439: Mon Mar 8 11:57:37 2021
read: IOPS=11.3k, BW=703MiB/s (737MB/s)(82.4GiB/120008msec)
slat (nsec): min=1794, max=660096, avg=5161.71, stdev=3212.55
clat (usec): min=442, max=34783, avg=11369.45, stdev=1819.99
lat (usec): min=447, max=34789, avg=11374.70, stdev=1819.83
clat percentiles (usec):
| 1.00th=[ 2040], 5.00th=[10159], 10.00th=[10552], 20.00th=[10814],
| 30.00th=[10945], 40.00th=[11076], 50.00th=[11338], 60.00th=[11469],
| 70.00th=[11731], 80.00th=[12125], 90.00th=[12387], 95.00th=[12911],
| 99.00th=[16909], 99.50th=[18482], 99.90th=[23462], 99.95th=[26346],
| 99.99th=[29754]
bw ( KiB/s): min=333824, max=387200, per=50.00%, avg=360054.97, stdev=7782.42, samples=480
iops : min= 5216, max= 6050, avg=5625.86, stdev=121.59, samples=480
lat (usec) : 500=0.05%, 750=0.25%, 1000=0.17%
lat (msec) : 2=0.51%, 4=0.55%, 10=2.78%, 20=95.30%, 50=0.39%
cpu : usr=1.05%, sys=4.15%, ctx=515674, majf=0, minf=1050
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=1350352,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=703MiB/s (737MB/s), 703MiB/s-703MiB/s (737MB/s-737MB/s), io=82.4GiB (88.5GB), run=120008-120008msec
Disk stats (read/write):
nvme0n1: ios=1348944/8, merge=5/3, ticks=15321948/103, in_queue=14650305, util=99.96%
# cat test_random_read_write_throughput.fio
[global]
bs=64K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
numjobs=2
runtime=120
size=15g
name=raw-randreadwrite
rw=randrw
[job1]
filename=nvme0n1
# fio test_random_read_write_throughput.fio
job1: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [m(2)][100.0%][r=249MiB/s,w=248MiB/s][r=3983,w=3961 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=2): err= 0: pid=6719: Mon Mar 8 12:00:49 2021
read: IOPS=4013, BW=251MiB/s (263MB/s)(29.4GiB/120024msec)
slat (nsec): min=1814, max=117638, avg=5084.58, stdev=3571.32
clat (usec): min=178, max=64977, avg=16089.49, stdev=13200.31
lat (usec): min=186, max=64987, avg=16094.67, stdev=13200.82
clat percentiles (usec):
| 1.00th=[ 502], 5.00th=[ 725], 10.00th=[ 1057], 20.00th=[ 1745],
| 30.00th=[ 3261], 40.00th=[ 6521], 50.00th=[15270], 60.00th=[24249],
| 70.00th=[27132], 80.00th=[28967], 90.00th=[31065], 95.00th=[34866],
| 99.00th=[45876], 99.50th=[49546], 99.90th=[54789], 99.95th=[55837],
| 99.99th=[60556]
bw ( KiB/s): min=99584, max=153728, per=50.00%, avg=128417.76, stdev=8306.12, samples=480
iops : min= 1556, max= 2402, avg=2006.51, stdev=129.77, samples=480
write: IOPS=4021, BW=251MiB/s (264MB/s)(29.5GiB/120024msec)
slat (usec): min=2, max=116, avg= 9.00, stdev= 4.71
clat (usec): min=72, max=69282, avg=15755.81, stdev=13277.92
lat (usec): min=93, max=69294, avg=15764.91, stdev=13278.60
clat percentiles (usec):
| 1.00th=[ 310], 5.00th=[ 502], 10.00th=[ 611], 20.00th=[ 1106],
| 30.00th=[ 2704], 40.00th=[ 6325], 50.00th=[15139], 60.00th=[23987],
| 70.00th=[26870], 80.00th=[28705], 90.00th=[30802], 95.00th=[34866],
| 99.00th=[45876], 99.50th=[49546], 99.90th=[54789], 99.95th=[55837],
| 99.99th=[60031]
bw ( KiB/s): min=105344, max=151936, per=50.00%, avg=128686.17, stdev=8007.27, samples=480
iops : min= 1646, max= 2374, avg=2010.70, stdev=125.10, samples=480
lat (usec) : 100=0.01%, 250=0.36%, 500=2.62%, 750=7.84%, 1000=3.58%
lat (msec) : 2=10.26%, 4=9.16%, 10=11.22%, 20=8.84%, 50=45.65%
lat (msec) : 100=0.47%
cpu : usr=1.53%, sys=3.12%, ctx=226178, majf=0, minf=30
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=481658,482675,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=251MiB/s (263MB/s), 251MiB/s-251MiB/s (263MB/s-263MB/s), io=29.4GiB (31.6GB), run=120024-120024msec
WRITE: bw=251MiB/s (264MB/s), 251MiB/s-251MiB/s (264MB/s-264MB/s), io=29.5GiB (31.6GB), run=120024-120024msec
Disk stats (read/write):
nvme0n1: ios=481091/482110, merge=1/5, ticks=7726178/7558547, in_queue=14788990, util=99.92%
# cat test_squential_read_throughput.fio
[global]
bs=64K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
numjobs=2
runtime=120
size=15g
name=raw-squentialread
rw=read
[job1]
filename=nvme0n1
# fio test_squential_read_throughput.fio
job1: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [R(2)][100.0%][r=704MiB/s,w=0KiB/s][r=11.3k,w=0 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=2): err= 0: pid=7031: Mon Mar 8 12:04:14 2021
read: IOPS=11.3k, BW=703MiB/s (737MB/s)(82.4GiB/120006msec)
slat (nsec): min=1753, max=673261, avg=4024.62, stdev=3402.57
clat (usec): min=165, max=34226, avg=11370.27, stdev=1308.25
lat (usec): min=172, max=34266, avg=11374.38, stdev=1308.15
clat percentiles (usec):
| 1.00th=[ 6521], 5.00th=[10945], 10.00th=[10945], 20.00th=[10945],
| 30.00th=[11076], 40.00th=[11076], 50.00th=[11076], 60.00th=[11207],
| 70.00th=[11863], 80.00th=[11994], 90.00th=[12125], 95.00th=[12256],
| 99.00th=[16057], 99.50th=[16909], 99.90th=[21103], 99.95th=[22414],
| 99.99th=[26870]
bw ( KiB/s): min=335872, max=383104, per=50.00%, avg=360069.33, stdev=5954.38, samples=480
iops : min= 5248, max= 5986, avg=5626.09, stdev=93.04, samples=480
lat (usec) : 250=0.04%, 500=0.08%, 750=0.15%, 1000=0.12%
lat (msec) : 2=0.18%, 4=0.21%, 10=0.51%, 20=98.56%, 50=0.15%
cpu : usr=0.81%, sys=2.86%, ctx=198342, majf=0, minf=1053
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=1350389,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=703MiB/s (737MB/s), 703MiB/s-703MiB/s (737MB/s-737MB/s), io=82.4GiB (88.5GB), run=120006-120006msec
Disk stats (read/write):
nvme0n1: ios=1348981/8, merge=6/3, ticks=15294070/1, in_queue=14641195, util=99.95%
