Skip to content

Instantly share code, notes, and snippets.

@ryozi-tn

ryozi-tn/2dai-dump.log Secret

Last active November 25, 2023 03:15
Show Gist options
  • Save ryozi-tn/fdf7dfa820c5e8a93e7177c0c3ae2680 to your computer and use it in GitHub Desktop.
Save ryozi-tn/fdf7dfa820c5e8a93e7177c0c3ae2680 to your computer and use it in GitHub Desktop.
Download ZIP
8台のHDDに対してfioでテストしたやつ
Raw
2dai-dump.log
$ for DEVICES in "/dev/sdb:/dev/sdc" "/dev/sdd:/dev/sde" "/dev/sdf:/dev/sdg" "/dev/sdh:/dev/sdi"; do
> for RW in read write; do
> sudo fio -direct=1 -iodepth=16 -ioengine=libaio -rw=$RW -bs=1024k -size=10G -runtime=10s -filename="$DEVICES" -name="Sequential_${RW}_${DEVICES}"
> done
> done
Sequential_read_/dev/sdb:/dev/sdc: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=2): [R(1)][100.0%][r=374MiB/s][r=374 IOPS][eta 00m:00s]
Sequential_read_/dev/sdb:/dev/sdc: (groupid=0, jobs=1): err= 0: pid=6767: Sat Nov 25 07:37:07 2023
read: IOPS=376, BW=377MiB/s (395MB/s)(3795MiB/10079msec)
slat (usec): min=7, max=429, avg=10.47, stdev=15.97
clat (usec): min=1856, max=596396, avg=42453.90, stdev=43849.47
lat (usec): min=1864, max=596404, avg=42464.37, stdev=43853.10
clat percentiles (usec):
| 1.00th=[ 1860], 5.00th=[ 1860], 10.00th=[ 1860], 20.00th=[ 4686],
| 30.00th=[ 10028], 40.00th=[ 19792], 50.00th=[ 42730], 60.00th=[ 58459],
| 70.00th=[ 68682], 80.00th=[ 74974], 90.00th=[ 80217], 95.00th=[ 83362],
| 99.00th=[115868], 99.50th=[154141], 99.90th=[522191], 99.95th=[530580],
| 99.99th=[599786]
bw ( KiB/s): min=30720, max=432128, per=100.00%, avg=387072.00, stdev=87033.98, samples=20
iops : min= 30, max= 422, avg=378.00, stdev=84.99, samples=20
lat (msec) : 2=14.18%, 4=2.32%, 10=12.94%, 20=10.67%, 50=13.15%
lat (msec) : 100=45.61%, 250=0.71%, 500=0.24%, 750=0.18%
cpu : usr=0.00%, sys=0.34%, ctx=7588, majf=0, minf=532
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.6%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3795,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=377MiB/s (395MB/s), 377MiB/s-377MiB/s (395MB/s-395MB/s), io=3795MiB (3979MB), run=10079-10079msec
Disk stats (read/write):
sdb: ios=1986/0, merge=0/0, ticks=45802/0, in_queue=45802, util=76.92%
sdc: ios=1966/0, merge=0/0, ticks=121332/0, in_queue=121332, util=82.48%
Sequential_write_/dev/sdb:/dev/sdc: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=2): [W(1)][100.0%][w=372MiB/s][w=372 IOPS][eta 00m:00s]
Sequential_write_/dev/sdb:/dev/sdc: (groupid=0, jobs=1): err= 0: pid=6784: Sat Nov 25 07:37:17 2023
write: IOPS=394, BW=395MiB/s (414MB/s)(3967MiB/10054msec); 0 zone resets
slat (nsec): min=10049, max=40045, avg=16057.70, stdev=2986.38
clat (usec): min=1941, max=121770, avg=40516.91, stdev=32530.67
lat (usec): min=1965, max=121792, avg=40532.97, stdev=32529.74
clat percentiles (usec):
| 1.00th=[ 1958], 5.00th=[ 1958], 10.00th=[ 1958], 20.00th=[ 1958],
| 30.00th=[ 9110], 40.00th=[ 20055], 50.00th=[ 42730], 60.00th=[ 57410],
| 70.00th=[ 70779], 80.00th=[ 76022], 90.00th=[ 82314], 95.00th=[ 83362],
| 99.00th=[ 89654], 99.50th=[102237], 99.90th=[116917], 99.95th=[116917],
| 99.99th=[122160]
bw ( KiB/s): min=342016, max=432128, per=100.00%, avg=404684.80, stdev=21931.17, samples=20
iops : min= 334, max= 422, avg=395.20, stdev=21.42, samples=20
lat (msec) : 2=23.47%, 4=1.39%, 10=7.26%, 20=7.89%, 50=13.31%
lat (msec) : 100=46.18%, 250=0.50%
cpu : usr=0.44%, sys=0.27%, ctx=3959, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.6%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,3967,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=395MiB/s (414MB/s), 395MiB/s-395MiB/s (414MB/s-414MB/s), io=3967MiB (4160MB), run=10054-10054msec
Disk stats (read/write):
sdb: ios=47/1972, merge=0/0, ticks=28/32637, in_queue=32664, util=71.20%
sdc: ios=47/1956, merge=0/0, ticks=62/124860, in_queue=124922, util=80.85%
Sequential_read_/dev/sdd:/dev/sde: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=2): [R(1)][100.0%][r=391MiB/s][r=391 IOPS][eta 00m:00s]
Sequential_read_/dev/sdd:/dev/sde: (groupid=0, jobs=1): err= 0: pid=6824: Sat Nov 25 07:37:28 2023
read: IOPS=370, BW=371MiB/s (389MB/s)(3734MiB/10073msec)
slat (usec): min=6, max=344, avg= 8.15, stdev=12.44
clat (usec): min=1855, max=614407, avg=43126.89, stdev=45406.25
lat (usec): min=1863, max=614415, avg=43135.04, stdev=45410.44
clat percentiles (usec):
| 1.00th=[ 1860], 5.00th=[ 1860], 10.00th=[ 1860], 20.00th=[ 4490],
| 30.00th=[ 14746], 40.00th=[ 28443], 50.00th=[ 39584], 60.00th=[ 52691],
| 70.00th=[ 65799], 80.00th=[ 76022], 90.00th=[ 79168], 95.00th=[ 87557],
| 99.00th=[ 94897], 99.50th=[152044], 99.90th=[599786], 99.95th=[608175],
| 99.99th=[616563]
bw ( KiB/s): min=20480, max=421888, per=100.00%, avg=380825.60, stdev=87381.97, samples=20
iops : min= 20, max= 412, avg=371.90, stdev=85.33, samples=20
lat (msec) : 2=16.01%, 4=1.87%, 10=6.19%, 20=9.80%, 50=24.32%
lat (msec) : 100=40.95%, 250=0.43%, 500=0.16%, 750=0.27%
cpu : usr=0.00%, sys=0.27%, ctx=7459, majf=0, minf=20
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.6%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3734,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=371MiB/s (389MB/s), 371MiB/s-371MiB/s (389MB/s-389MB/s), io=3734MiB (3915MB), run=10073-10073msec
Disk stats (read/write):
sdd: ios=1830/0, merge=0/0, ticks=125766/0, in_queue=125766, util=81.46%
sde: ios=1845/0, merge=0/0, ticks=31913/0, in_queue=31913, util=75.20%
Sequential_write_/dev/sdd:/dev/sde: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=2): [W(1)][100.0%][w=391MiB/s][w=391 IOPS][eta 00m:00s]
Sequential_write_/dev/sdd:/dev/sde: (groupid=0, jobs=1): err= 0: pid=6846: Sat Nov 25 07:37:38 2023
write: IOPS=390, BW=390MiB/s (409MB/s)(3933MiB/10074msec); 0 zone resets
slat (nsec): min=9338, max=40236, avg=16113.13, stdev=3400.12
clat (usec): min=1939, max=144942, avg=40944.10, stdev=32549.62
lat (usec): min=1965, max=144957, avg=40960.21, stdev=32548.42
clat percentiles (usec):
| 1.00th=[ 1958], 5.00th=[ 1958], 10.00th=[ 1958], 20.00th=[ 1958],
| 30.00th=[ 9634], 40.00th=[ 23200], 50.00th=[ 39584], 60.00th=[ 56886],
| 70.00th=[ 70779], 80.00th=[ 76022], 90.00th=[ 78119], 95.00th=[ 86508],
| 99.00th=[ 90702], 99.50th=[120062], 99.90th=[129500], 99.95th=[139461],
| 99.99th=[145753]
bw ( KiB/s): min=364544, max=423936, per=100.00%, avg=401203.20, stdev=16274.50, samples=20
iops : min= 356, max= 414, avg=391.80, stdev=15.89, samples=20
lat (msec) : 2=23.21%, 4=0.48%, 10=7.04%, 20=7.30%, 50=17.54%
lat (msec) : 100=43.76%, 250=0.66%
cpu : usr=0.47%, sys=0.24%, ctx=3919, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.6%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,3933,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=390MiB/s (409MB/s), 390MiB/s-390MiB/s (409MB/s-409MB/s), io=3933MiB (4124MB), run=10074-10074msec
Disk stats (read/write):
sdd: ios=46/1929, merge=0/0, ticks=51/131958, in_queue=132009, util=81.99%
sde: ios=47/1944, merge=0/0, ticks=23/25320, in_queue=25343, util=71.20%
Sequential_read_/dev/sdf:/dev/sdg: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=2): [R(1)][100.0%][r=403MiB/s][r=403 IOPS][eta 00m:00s]
Sequential_read_/dev/sdf:/dev/sdg: (groupid=0, jobs=1): err= 0: pid=6881: Sat Nov 25 07:37:48 2023
read: IOPS=375, BW=375MiB/s (393MB/s)(3774MiB/10064msec)
slat (usec): min=8, max=319, avg= 9.78, stdev=12.56
clat (msec): min=2, max=626, avg=42.63, stdev=41.33
lat (msec): min=2, max=626, avg=42.64, stdev=41.34
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 5], 20.00th=[ 9],
| 30.00th=[ 18], 40.00th=[ 34], 50.00th=[ 41], 60.00th=[ 48],
| 70.00th=[ 64], 80.00th=[ 69], 90.00th=[ 79], 95.00th=[ 81],
| 99.00th=[ 89], 99.50th=[ 167], 99.90th=[ 514], 99.95th=[ 625],
| 99.99th=[ 625]
bw ( KiB/s): min=24576, max=440320, per=100.00%, avg=384819.20, stdev=88649.63, samples=20
iops : min= 24, max= 430, avg=375.80, stdev=86.57, samples=20
lat (msec) : 4=7.90%, 10=14.52%, 20=8.93%, 50=30.31%, 100=37.41%
lat (msec) : 250=0.50%, 500=0.26%, 750=0.16%
cpu : usr=0.02%, sys=0.27%, ctx=7529, majf=0, minf=21
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.6%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3774,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=375MiB/s (393MB/s), 375MiB/s-375MiB/s (393MB/s-393MB/s), io=3774MiB (3957MB), run=10064-10064msec
Disk stats (read/write):
sdf: ios=1863/0, merge=0/0, ticks=36723/0, in_queue=36723, util=81.86%
sdg: ios=1853/0, merge=0/0, ticks=121156/0, in_queue=121157, util=78.49%
Sequential_write_/dev/sdf:/dev/sdg: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=2): [W(1)][100.0%][w=385MiB/s][w=385 IOPS][eta 00m:00s]
Sequential_write_/dev/sdf:/dev/sdg: (groupid=0, jobs=1): err= 0: pid=6908: Sat Nov 25 07:37:59 2023
write: IOPS=394, BW=395MiB/s (414MB/s)(3977MiB/10077msec); 0 zone resets
slat (nsec): min=10810, max=53871, avg=17416.18, stdev=2949.49
clat (msec): min=2, max=151, avg=40.50, stdev=32.82
lat (msec): min=2, max=151, avg=40.52, stdev=32.82
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 5], 40.00th=[ 14], 50.00th=[ 41], 60.00th=[ 68],
| 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 80], 95.00th=[ 81],
| 99.00th=[ 88], 99.50th=[ 102], 99.90th=[ 138], 99.95th=[ 146],
| 99.99th=[ 153]
bw ( KiB/s): min=360448, max=438272, per=100.00%, avg=405606.40, stdev=18359.70, samples=20
iops : min= 352, max= 428, avg=396.10, stdev=17.93, samples=20
lat (msec) : 4=28.01%, 10=7.07%, 20=7.49%, 50=12.25%, 100=44.66%
lat (msec) : 250=0.53%
cpu : usr=0.40%, sys=0.37%, ctx=3974, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.6%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,3977,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=395MiB/s (414MB/s), 395MiB/s-395MiB/s (414MB/s-414MB/s), io=3977MiB (4170MB), run=10077-10077msec
Disk stats (read/write):
sdf: ios=47/1967, merge=0/0, ticks=33/19693, in_queue=19725, util=86.78%
sdg: ios=47/1952, merge=0/0, ticks=21/137857, in_queue=137877, util=63.30%
Sequential_read_/dev/sdh:/dev/sdi: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=2): [R(1)][100.0%][r=380MiB/s][r=380 IOPS][eta 00m:00s]
Sequential_read_/dev/sdh:/dev/sdi: (groupid=0, jobs=1): err= 0: pid=6953: Sat Nov 25 07:38:09 2023
read: IOPS=370, BW=370MiB/s (388MB/s)(3719MiB/10043msec)
slat (usec): min=4, max=416, avg=11.28, stdev=15.66
clat (usec): min=1913, max=640441, avg=43176.79, stdev=46825.81
lat (usec): min=1923, max=640451, avg=43188.07, stdev=46831.66
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 6],
| 30.00th=[ 12], 40.00th=[ 32], 50.00th=[ 42], 60.00th=[ 51],
| 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 79], 95.00th=[ 80],
| 99.00th=[ 94], 99.50th=[ 167], 99.90th=[ 634], 99.95th=[ 642],
| 99.99th=[ 642]
bw ( KiB/s): min=16384, max=423936, per=100.00%, avg=379289.60, stdev=88566.67, samples=20
iops : min= 16, max= 414, avg=370.40, stdev=86.49, samples=20
lat (msec) : 2=0.08%, 4=16.48%, 10=12.37%, 20=4.17%, 50=26.27%
lat (msec) : 100=40.09%, 250=0.11%, 500=0.11%, 750=0.32%
cpu : usr=0.02%, sys=0.32%, ctx=7401, majf=0, minf=531
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.6%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3719,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=370MiB/s (388MB/s), 370MiB/s-370MiB/s (388MB/s-388MB/s), io=3719MiB (3900MB), run=10043-10043msec
Disk stats (read/write):
sdh: ios=1832/0, merge=0/0, ticks=33534/0, in_queue=33534, util=77.38%
sdi: ios=1830/0, merge=0/0, ticks=124350/0, in_queue=124349, util=74.24%
Sequential_write_/dev/sdh:/dev/sdi: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=2): [W(1)][100.0%][w=371MiB/s][w=371 IOPS][eta 00m:00s]
Sequential_write_/dev/sdh:/dev/sdi: (groupid=0, jobs=1): err= 0: pid=6974: Sat Nov 25 07:38:19 2023
write: IOPS=392, BW=393MiB/s (412MB/s)(3949MiB/10055msec); 0 zone resets
slat (nsec): min=11492, max=52388, avg=17736.27, stdev=2841.38
clat (msec): min=2, max=101, avg=40.70, stdev=27.01
lat (msec): min=2, max=101, avg=40.72, stdev=27.01
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 7],
| 30.00th=[ 23], 40.00th=[ 33], 50.00th=[ 41], 60.00th=[ 49],
| 70.00th=[ 61], 80.00th=[ 74], 90.00th=[ 78], 95.00th=[ 79],
| 99.00th=[ 87], 99.50th=[ 89], 99.90th=[ 92], 99.95th=[ 96],
| 99.99th=[ 102]
bw ( KiB/s): min=370688, max=425984, per=100.00%, avg=402841.60, stdev=16385.35, samples=20
iops : min= 362, max= 416, avg=393.40, stdev=16.00, samples=20
lat (msec) : 4=18.18%, 10=3.90%, 20=6.13%, 50=35.17%, 100=36.59%
lat (msec) : 250=0.03%
cpu : usr=0.48%, sys=0.30%, ctx=3945, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.6%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,3949,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=393MiB/s (412MB/s), 393MiB/s-393MiB/s (412MB/s-412MB/s), io=3949MiB (4141MB), run=10055-10055msec
Disk stats (read/write):
sdh: ios=46/1946, merge=0/0, ticks=46/35475, in_queue=35521, util=83.06%
sdi: ios=33/1942, merge=0/0, ticks=46/122359, in_queue=122404, util=72.04%
Raw
4dai-dumo.log
$ for DEVICES in "/dev/sdf:/dev/sdg:/dev/sdh:/dev/sdi" "/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde"; do
> for RW in read write; do
> sudo fio -direct=1 -iodepth=16 -ioengine=libaio -rw=$RW -bs=1024k -size=10G -runtime=10s -filename="$DEVICES" -name="Sequential_${RW}_"
> done
> done
Sequential_read_: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=4): [R(1)][100.0%][r=540MiB/s][r=540 IOPS][eta 00m:00s]
Sequential_read_: (groupid=0, jobs=1): err= 0: pid=6583: Sat Nov 25 07:28:48 2023
read: IOPS=517, BW=518MiB/s (543MB/s)(5194MiB/10028msec)
slat (usec): min=5, max=340, avg=10.55, stdev=10.85
clat (msec): min=2, max=625, avg=30.87, stdev=28.85
lat (msec): min=2, max=625, avg=30.88, stdev=28.86
clat percentiles (msec):
| 1.00th=[ 15], 5.00th=[ 20], 10.00th=[ 21], 20.00th=[ 22],
| 30.00th=[ 25], 40.00th=[ 27], 50.00th=[ 28], 60.00th=[ 30],
| 70.00th=[ 33], 80.00th=[ 35], 90.00th=[ 41], 95.00th=[ 46],
| 99.00th=[ 65], 99.50th=[ 79], 99.90th=[ 535], 99.95th=[ 584],
| 99.99th=[ 625]
bw ( KiB/s): min=22528, max=567296, per=99.99%, avg=530329.60, stdev=120983.50, samples=20
iops : min= 22, max= 554, avg=517.90, stdev=118.15, samples=20
lat (msec) : 4=0.10%, 10=0.12%, 20=6.72%, 50=90.41%, 100=2.21%
lat (msec) : 250=0.13%, 500=0.13%, 750=0.17%
cpu : usr=0.04%, sys=0.35%, ctx=9193, majf=0, minf=22
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5194,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=518MiB/s (543MB/s), 518MiB/s-518MiB/s (543MB/s-543MB/s), io=5194MiB (5446MB), run=10028-10028msec
Disk stats (read/write):
sdf: ios=1281/0, merge=0/0, ticks=39481/0, in_queue=39481, util=64.85%
sdg: ios=1279/0, merge=0/0, ticks=37127/0, in_queue=37127, util=66.63%
sdh: ios=1279/0, merge=0/0, ticks=35533/0, in_queue=35533, util=67.82%
sdi: ios=1278/0, merge=0/0, ticks=45796/0, in_queue=45796, util=64.74%
Sequential_write_: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=4): [W(1)][100.0%][w=487MiB/s][w=487 IOPS][eta 00m:00s]
Sequential_write_: (groupid=0, jobs=1): err= 0: pid=6600: Sat Nov 25 07:28:58 2023
write: IOPS=485, BW=486MiB/s (509MB/s)(4870MiB/10030msec); 0 zone resets
slat (nsec): min=9347, max=44354, avg=18450.24, stdev=4264.80
clat (usec): min=8193, max=90071, avg=32924.32, stdev=15347.27
lat (usec): min=8230, max=90085, avg=32942.77, stdev=15346.52
clat percentiles (usec):
| 1.00th=[16319], 5.00th=[16319], 10.00th=[16450], 20.00th=[20317],
| 30.00th=[26346], 40.00th=[27132], 50.00th=[27657], 60.00th=[31851],
| 70.00th=[32900], 80.00th=[39584], 90.00th=[67634], 95.00th=[67634],
| 99.00th=[68682], 99.50th=[68682], 99.90th=[80217], 99.95th=[80217],
| 99.99th=[89654]
bw ( KiB/s): min=450560, max=499712, per=99.99%, avg=497152.00, stdev=10976.15, samples=20
iops : min= 440, max= 488, avg=485.50, stdev=10.72, samples=20
lat (msec) : 10=0.08%, 20=14.66%, 50=71.58%, 100=13.68%
cpu : usr=0.54%, sys=0.45%, ctx=4868, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4870,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=486MiB/s (509MB/s), 486MiB/s-486MiB/s (509MB/s-509MB/s), io=4870MiB (5107MB), run=10030-10030msec
Disk stats (read/write):
sdf: ios=47/1200, merge=0/0, ticks=67/39332, in_queue=39399, util=64.30%
sdg: ios=47/1201, merge=0/0, ticks=24/31725, in_queue=31749, util=77.25%
sdh: ios=47/1202, merge=0/0, ticks=53/26287, in_queue=26340, util=73.78%
sdi: ios=47/1195, merge=0/0, ticks=44/60378, in_queue=60422, util=68.65%
Sequential_read_: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=4): [R(1)][100.0%][r=761MiB/s][r=761 IOPS][eta 00m:00s]
Sequential_read_: (groupid=0, jobs=1): err= 0: pid=6653: Sat Nov 25 07:29:08 2023
read: IOPS=740, BW=740MiB/s (776MB/s)(7432MiB/10042msec)
slat (usec): min=6, max=284, avg= 7.83, stdev= 7.55
clat (usec): min=1855, max=582311, avg=21602.27, stdev=29757.11
lat (usec): min=1862, max=582320, avg=21610.09, stdev=29759.42
clat percentiles (usec):
| 1.00th=[ 1860], 5.00th=[ 1860], 10.00th=[ 1860], 20.00th=[ 1860],
| 30.00th=[ 4293], 40.00th=[ 7504], 50.00th=[ 13829], 60.00th=[ 21627],
| 70.00th=[ 28967], 80.00th=[ 39060], 90.00th=[ 47449], 95.00th=[ 61604],
| 99.00th=[ 83362], 99.50th=[ 89654], 99.90th=[471860], 99.95th=[497026],
| 99.99th=[583009]
bw ( KiB/s): min=40960, max=845824, per=100.00%, avg=759500.80, stdev=174407.85, samples=20
iops : min= 40, max= 826, avg=741.70, stdev=170.32, samples=20
lat (msec) : 2=23.80%, 4=5.10%, 10=15.82%, 20=13.41%, 50=33.58%
lat (msec) : 100=7.87%, 250=0.19%, 500=0.19%, 750=0.03%
cpu : usr=0.02%, sys=0.51%, ctx=14800, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=7432,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=740MiB/s (776MB/s), 740MiB/s-740MiB/s (776MB/s-776MB/s), io=7432MiB (7793MB), run=10042-10042msec
Disk stats (read/write):
sdb: ios=1837/0, merge=0/0, ticks=16916/0, in_queue=16915, util=67.60%
sdc: ios=1831/0, merge=0/0, ticks=47353/0, in_queue=47353, util=73.91%
sdd: ios=1826/0, merge=0/0, ticks=64749/0, in_queue=64750, util=74.27%
sde: ios=1836/0, merge=0/0, ticks=28813/0, in_queue=28813, util=69.31%
Sequential_write_: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=4): [W(1)][100.0%][w=760MiB/s][w=760 IOPS][eta 00m:00s]
Sequential_write_: (groupid=0, jobs=1): err= 0: pid=6690: Sat Nov 25 07:29:19 2023
write: IOPS=779, BW=780MiB/s (818MB/s)(7827MiB/10036msec); 0 zone resets
slat (nsec): min=8496, max=35357, avg=15940.61, stdev=3326.22
clat (usec): min=1947, max=85764, avg=20493.36, stdev=22303.27
lat (usec): min=1959, max=85781, avg=20509.30, stdev=22303.09
clat percentiles (usec):
| 1.00th=[ 1958], 5.00th=[ 1958], 10.00th=[ 1958], 20.00th=[ 1958],
| 30.00th=[ 1958], 40.00th=[ 1958], 50.00th=[10814], 60.00th=[22938],
| 70.00th=[30016], 80.00th=[37487], 90.00th=[52691], 95.00th=[70779],
| 99.00th=[83362], 99.50th=[83362], 99.90th=[84411], 99.95th=[85459],
| 99.99th=[85459]
bw ( KiB/s): min=714752, max=847872, per=100.00%, avg=799948.80, stdev=36402.41, samples=20
iops : min= 698, max= 828, avg=781.20, stdev=35.55, samples=20
lat (msec) : 2=41.05%, 4=2.22%, 10=6.07%, 20=7.82%, 50=31.17%
lat (msec) : 100=11.66%
cpu : usr=1.03%, sys=0.42%, ctx=7736, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,7827,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=780MiB/s (818MB/s), 780MiB/s-780MiB/s (818MB/s-818MB/s), io=7827MiB (8207MB), run=10036-10036msec
Disk stats (read/write):
sdb: ios=47/1934, merge=0/0, ticks=18/17286, in_queue=17303, util=62.90%
sdc: ios=47/1928, merge=0/0, ticks=51/29663, in_queue=29714, util=67.55%
sdd: ios=47/1937, merge=0/0, ticks=42/92905, in_queue=92947, util=75.67%
sde: ios=47/1938, merge=0/0, ticks=58/17741, in_queue=17798, util=65.22%
Raw
4dai-irekae-dump.log
$ for DEVICES in "/dev/sdf:/dev/sdg:/dev/sdd:/dev/sde" "/dev/sdb:/dev/sdc:/dev/sdh:/dev/sdi"; do
> for RW in read write; do
> sudo fio -direct=1 -iodepth=16 -ioengine=libaio -rw=$RW -bs=1024k -size=10G -runtime=10s -filename="$DEVICES" -name="Sequential_${RW}_${DEVICES}"
> done
> done
Sequential_read_/dev/sdf:/dev/sdg:/dev/sdd:/dev/sde: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=4): [R(1)][100.0%][r=781MiB/s][r=780 IOPS][eta 00m:00s]
Sequential_read_/dev/sdf:/dev/sdg:/dev/sdd:/dev/sde: (groupid=0, jobs=1): err= 0: pid=7217: Sat Nov 25 08:07:47 2023
read: IOPS=728, BW=728MiB/s (764MB/s)(7329MiB/10063msec)
slat (usec): min=6, max=303, avg= 8.92, stdev= 8.72
clat (usec): min=1855, max=396354, avg=21946.63, stdev=32256.56
lat (usec): min=1863, max=396365, avg=21955.55, stdev=32258.20
clat percentiles (usec):
| 1.00th=[ 1860], 5.00th=[ 1860], 10.00th=[ 1909], 20.00th=[ 2737],
| 30.00th=[ 3392], 40.00th=[ 5407], 50.00th=[ 9634], 60.00th=[ 13435],
| 70.00th=[ 25035], 80.00th=[ 46400], 90.00th=[ 61080], 95.00th=[ 69731],
| 99.00th=[ 77071], 99.50th=[104334], 99.90th=[379585], 99.95th=[392168],
| 99.99th=[396362]
bw ( KiB/s): min=174080, max=831488, per=100.00%, avg=748953.60, stdev=173506.63, samples=20
iops : min= 170, max= 812, avg=731.40, stdev=169.44, samples=20
lat (msec) : 2=14.38%, 4=18.30%, 10=18.41%, 20=15.30%, 50=16.22%
lat (msec) : 100=16.89%, 250=0.07%, 500=0.44%
cpu : usr=0.05%, sys=0.50%, ctx=14584, majf=0, minf=22
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=7329,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=728MiB/s (764MB/s), 728MiB/s-728MiB/s (764MB/s-764MB/s), io=7329MiB (7685MB), run=10063-10063msec
Disk stats (read/write):
sdf: ios=1809/0, merge=0/0, ticks=11344/0, in_queue=11344, util=70.68%
sdg: ios=1810/0, merge=0/0, ticks=26072/0, in_queue=26072, util=72.34%
sdd: ios=1797/0, merge=0/0, ticks=85961/0, in_queue=85961, util=68.11%
sde: ios=1810/0, merge=0/0, ticks=34385/0, in_queue=34385, util=62.75%
Sequential_write_/dev/sdf:/dev/sdg:/dev/sdd:/dev/sde: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=4): [W(1)][100.0%][w=781MiB/s][w=781 IOPS][eta 00m:00s]
Sequential_write_/dev/sdf:/dev/sdg:/dev/sdd:/dev/sde: (groupid=0, jobs=1): err= 0: pid=7234: Sat Nov 25 08:07:57 2023
write: IOPS=771, BW=772MiB/s (809MB/s)(7771MiB/10068msec); 0 zone resets
slat (nsec): min=10139, max=36519, avg=16203.21, stdev=3536.74
clat (usec): min=1948, max=131647, avg=20703.29, stdev=26241.18
lat (usec): min=1960, max=131676, avg=20719.49, stdev=26241.09
clat percentiles (usec):
| 1.00th=[ 1958], 5.00th=[ 1958], 10.00th=[ 1958], 20.00th=[ 2114],
| 30.00th=[ 2769], 40.00th=[ 3392], 50.00th=[ 4015], 60.00th=[ 10159],
| 70.00th=[ 23462], 80.00th=[ 50594], 90.00th=[ 69731], 95.00th=[ 71828],
| 99.00th=[ 80217], 99.50th=[ 83362], 99.90th=[112722], 99.95th=[116917],
| 99.99th=[131597]
bw ( KiB/s): min=733184, max=837632, per=100.00%, avg=794214.40, stdev=36105.26, samples=20
iops : min= 716, max= 818, avg=775.60, stdev=35.26, samples=20
lat (msec) : 2=14.61%, 4=35.23%, 10=9.74%, 20=8.70%, 50=11.45%
lat (msec) : 100=20.11%, 250=0.15%
cpu : usr=0.88%, sys=0.53%, ctx=7714, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,7771,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=772MiB/s (809MB/s), 772MiB/s-772MiB/s (809MB/s-809MB/s), io=7771MiB (8148MB), run=10068-10068msec
Disk stats (read/write):
sdf: ios=47/1919, merge=0/0, ticks=65/10318, in_queue=10384, util=79.53%
sdg: ios=47/1919, merge=0/0, ticks=25/22935, in_queue=22959, util=76.55%
sdd: ios=49/1906, merge=0/0, ticks=1045/112987, in_queue=114032, util=67.14%
sde: ios=47/1918, merge=0/0, ticks=33/11384, in_queue=11417, util=60.81%
Sequential_read_/dev/sdb:/dev/sdc:/dev/sdh:/dev/sdi: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=4): [R(1)][100.0%][r=757MiB/s][r=756 IOPS][eta 00m:00s]
Sequential_read_/dev/sdb:/dev/sdc:/dev/sdh:/dev/sdi: (groupid=0, jobs=1): err= 0: pid=7292: Sat Nov 25 08:08:08 2023
read: IOPS=763, BW=763MiB/s (800MB/s)(7659MiB/10035msec)
slat (usec): min=4, max=359, avg=11.01, stdev= 8.54
clat (usec): min=1852, max=391323, avg=20943.96, stdev=24602.88
lat (usec): min=1864, max=391333, avg=20954.98, stdev=24604.59
clat percentiles (usec):
| 1.00th=[ 1860], 5.00th=[ 1860], 10.00th=[ 1860], 20.00th=[ 2868],
| 30.00th=[ 4424], 40.00th=[ 9110], 50.00th=[ 14746], 60.00th=[ 22414],
| 70.00th=[ 30802], 80.00th=[ 36963], 90.00th=[ 45876], 95.00th=[ 56886],
| 99.00th=[ 72877], 99.50th=[ 73925], 99.90th=[379585], 99.95th=[387974],
| 99.99th=[392168]
bw ( KiB/s): min=204800, max=858112, per=100.00%, avg=782745.60, stdev=138567.33, samples=20
iops : min= 200, max= 838, avg=764.40, stdev=135.32, samples=20
lat (msec) : 2=16.11%, 4=12.69%, 10=15.37%, 20=12.61%, 50=35.49%
lat (msec) : 100=7.52%, 500=0.21%
cpu : usr=0.03%, sys=0.66%, ctx=15263, majf=0, minf=533
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=7659,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=763MiB/s (800MB/s), 763MiB/s-763MiB/s (800MB/s-800MB/s), io=7659MiB (8031MB), run=10035-10035msec
Disk stats (read/write):
sdb: ios=2017/0, merge=0/0, ticks=11877/0, in_queue=11877, util=67.40%
sdc: ios=2004/0, merge=0/0, ticks=41153/0, in_queue=41154, util=72.45%
sdh: ios=2031/0, merge=0/0, ticks=27109/0, in_queue=27110, util=77.96%
sdi: ios=1996/0, merge=0/0, ticks=86532/0, in_queue=86533, util=75.78%
Sequential_write_/dev/sdb:/dev/sdc:/dev/sdh:/dev/sdi: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=4): [W(1)][100.0%][w=747MiB/s][w=747 IOPS][eta 00m:00s]
Sequential_write_/dev/sdb:/dev/sdc:/dev/sdh:/dev/sdi: (groupid=0, jobs=1): err= 0: pid=7324: Sat Nov 25 08:08:18 2023
write: IOPS=785, BW=786MiB/s (824MB/s)(7887MiB/10040msec); 0 zone resets
slat (nsec): min=9758, max=44504, avg=15652.80, stdev=2893.61
clat (usec): min=1949, max=82277, avg=20344.75, stdev=22263.68
lat (usec): min=1961, max=82291, avg=20360.40, stdev=22263.87
clat percentiles (usec):
| 1.00th=[ 1958], 5.00th=[ 1958], 10.00th=[ 1958], 20.00th=[ 1958],
| 30.00th=[ 3130], 40.00th=[ 3785], 50.00th=[10814], 60.00th=[17695],
| 70.00th=[25035], 80.00th=[40633], 90.00th=[58459], 95.00th=[68682],
| 99.00th=[73925], 99.50th=[74974], 99.90th=[76022], 99.95th=[81265],
| 99.99th=[82314]
bw ( KiB/s): min=733184, max=860160, per=100.00%, avg=806092.80, stdev=35256.42, samples=20
iops : min= 716, max= 840, avg=787.20, stdev=34.43, samples=20
lat (msec) : 2=27.22%, 4=14.43%, 10=7.30%, 20=14.86%, 50=20.08%
lat (msec) : 100=16.10%
cpu : usr=0.77%, sys=0.62%, ctx=7803, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,7887,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=786MiB/s (824MB/s), 786MiB/s-786MiB/s (824MB/s-824MB/s), io=7887MiB (8270MB), run=10040-10040msec
Disk stats (read/write):
sdb: ios=47/1952, merge=0/0, ticks=20/18224, in_queue=18243, util=64.84%
sdc: ios=47/1944, merge=0/0, ticks=43/13979, in_queue=14023, util=68.03%
sdh: ios=47/1952, merge=0/0, ticks=50/23155, in_queue=23205, util=83.25%
sdi: ios=47/1944, merge=0/0, ticks=47/102021, in_queue=102068, util=76.04%
Raw
dump.log
$ (# 1個づつ流す
> for RW in read write; do
> for DEVICE in /dev/sd?; do
> sudo fio -direct=1 -iodepth=16 -ioengine=libaio -rw=$RW -bs=1024k -size=10G -runtime=10s -filename="$DEVICE" -name="Sequential_${RW}_${DEVICE##*/}"
> done
> done
>
>
> # ":"で繋げたデバイス一覧を出す
> DEVICES=(/dev/sd?)
> $(printf ':%s' ${DEVICES[@]})
>
> # 一斉に流す（sdaを含めない）
> DEVICES=/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdh:/dev/sdi
> for RW in read write; do
> sudo fio -direct=1 -iodepth=16 -ioengine=libaio -rw=$RW -bs=1024k -size=10G -runtime=10s -filename="$DEVICES" -name="Sequential_${RW}"
> done
>
> # 4台づつ流す（8台だと性能を出し切れていなかったようなので）
> for DEVICES in "/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde" "/dev/sdf:/dev/sdg:/dev/sdh:/dev/sdi"; do
> for RW in read write; do
> sudo fio -direct=1 -iodepth=16 -ioengine=libaio -rw=$RW -bs=1024k -size=10G -runtime=10s -filename="$DEVICES" -name="Sequential_${RW}_"
> done
> done
> ) | tee dump.log
Sequential_read_sda: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_read_sda: (groupid=0, jobs=1): err= 0: pid=5873: Sat Nov 25 07:06:21 2023
read: IOPS=541, BW=542MiB/s (568MB/s)(5434MiB/10030msec)
slat (usec): min=6, max=305, avg= 8.13, stdev=10.21
clat (usec): min=3030, max=55220, avg=29512.63, stdev=1220.06
lat (usec): min=3336, max=55228, avg=29520.76, stdev=1215.07
clat percentiles (usec):
| 1.00th=[29492], 5.00th=[29492], 10.00th=[29492], 20.00th=[29492],
| 30.00th=[29492], 40.00th=[29492], 50.00th=[29492], 60.00th=[29492],
| 70.00th=[29492], 80.00th=[29492], 90.00th=[29492], 95.00th=[29492],
| 99.00th=[31589], 99.50th=[33424], 99.90th=[45876], 99.95th=[51643],
| 99.99th=[55313]
bw ( KiB/s): min=548864, max=557056, per=100.00%, avg=554803.20, stdev=2090.67, samples=20
iops : min= 536, max= 544, avg=541.80, stdev= 2.04, samples=20
lat (msec) : 4=0.02%, 10=0.06%, 20=0.11%, 50=99.76%, 100=0.06%
cpu : usr=0.03%, sys=0.38%, ctx=10869, majf=0, minf=20
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5434,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=542MiB/s (568MB/s), 542MiB/s-542MiB/s (568MB/s-568MB/s), io=5434MiB (5698MB), run=10030-10030msec
Disk stats (read/write):
sda: ios=5361/9, merge=0/1, ticks=158046/4008, in_queue=162147, util=98.77%
Sequential_read_sdb: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_read_sdb: (groupid=0, jobs=1): err= 0: pid=5890: Sat Nov 25 07:06:31 2023
read: IOPS=198, BW=198MiB/s (208MB/s)(1997MiB/10077msec)
slat (usec): min=7, max=355, avg= 8.87, stdev=18.90
clat (msec): min=2, max=440, avg=80.68, stdev=30.29
lat (msec): min=2, max=440, avg=80.69, stdev=30.29
clat percentiles (msec):
| 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 74],
| 30.00th=[ 77], 40.00th=[ 78], 50.00th=[ 79], 60.00th=[ 80],
| 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 81], 95.00th=[ 86],
| 99.00th=[ 134], 99.50th=[ 405], 99.90th=[ 439], 99.95th=[ 443],
| 99.99th=[ 443]
bw ( KiB/s): min=57344, max=223232, per=99.96%, avg=202854.40, stdev=34898.17, samples=20
iops : min= 56, max= 218, avg=198.10, stdev=34.08, samples=20
lat (msec) : 4=0.05%, 10=0.20%, 100=97.75%, 250=1.20%, 500=0.80%
cpu : usr=0.00%, sys=0.16%, ctx=3995, majf=0, minf=19
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=99.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1997,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=198MiB/s (208MB/s), 198MiB/s-198MiB/s (208MB/s-208MB/s), io=1997MiB (2094MB), run=10077-10077msec
Disk stats (read/write):
sdb: ios=1961/0, merge=0/0, ticks=157730/0, in_queue=157730, util=98.32%
Sequential_read_sdc: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_read_sdc: (groupid=0, jobs=1): err= 0: pid=5907: Sat Nov 25 07:06:41 2023
read: IOPS=191, BW=192MiB/s (201MB/s)(1935MiB/10081msec)
slat (usec): min=7, max=447, avg=10.89, stdev=21.91
clat (msec): min=2, max=430, avg=83.29, stdev=30.75
lat (msec): min=2, max=430, avg=83.30, stdev=30.76
clat percentiles (msec):
| 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 77],
| 30.00th=[ 79], 40.00th=[ 79], 50.00th=[ 79], 60.00th=[ 80],
| 70.00th=[ 84], 80.00th=[ 86], 90.00th=[ 86], 95.00th=[ 87],
| 99.00th=[ 167], 99.50th=[ 405], 99.90th=[ 426], 99.95th=[ 430],
| 99.99th=[ 430]
bw ( KiB/s): min=73728, max=217088, per=100.00%, avg=196608.00, stdev=30565.11, samples=20
iops : min= 72, max= 212, avg=192.00, stdev=29.85, samples=20
lat (msec) : 4=0.10%, 10=0.16%, 20=0.26%, 100=97.52%, 250=1.14%
lat (msec) : 500=0.83%
cpu : usr=0.03%, sys=0.15%, ctx=3871, majf=0, minf=529
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=99.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1935,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=192MiB/s (201MB/s), 192MiB/s-192MiB/s (201MB/s-201MB/s), io=1935MiB (2029MB), run=10081-10081msec
Disk stats (read/write):
sdc: ios=2023/0, merge=0/0, ticks=167203/0, in_queue=167202, util=97.69%
Sequential_read_sdd: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_read_sdd: (groupid=0, jobs=1): err= 0: pid=5924: Sat Nov 25 07:06:52 2023
read: IOPS=188, BW=188MiB/s (198MB/s)(1899MiB/10081msec)
slat (usec): min=7, max=336, avg= 9.05, stdev=19.21
clat (msec): min=2, max=454, avg=84.87, stdev=32.53
lat (msec): min=2, max=454, avg=84.88, stdev=32.54
clat percentiles (msec):
| 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 77],
| 30.00th=[ 79], 40.00th=[ 81], 50.00th=[ 81], 60.00th=[ 81],
| 70.00th=[ 83], 80.00th=[ 89], 90.00th=[ 90], 95.00th=[ 90],
| 99.00th=[ 169], 99.50th=[ 418], 99.90th=[ 451], 99.95th=[ 456],
| 99.99th=[ 456]
bw ( KiB/s): min=47104, max=215040, per=99.96%, avg=192819.20, stdev=35703.62, samples=20
iops : min= 46, max= 210, avg=188.30, stdev=34.87, samples=20
lat (msec) : 4=0.05%, 10=0.16%, 100=98.05%, 250=0.90%, 500=0.84%
cpu : usr=0.00%, sys=0.16%, ctx=3799, majf=0, minf=19
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=99.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1899,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=188MiB/s (198MB/s), 188MiB/s-188MiB/s (198MB/s-198MB/s), io=1899MiB (1991MB), run=10081-10081msec
Disk stats (read/write):
sdd: ios=1868/0, merge=0/0, ticks=157722/0, in_queue=157722, util=97.71%
Sequential_read_sde: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_read_sde: (groupid=0, jobs=1): err= 0: pid=5941: Sat Nov 25 07:07:02 2023
read: IOPS=194, BW=194MiB/s (204MB/s)(1957MiB/10070msec)
slat (usec): min=4, max=304, avg= 8.86, stdev=17.12
clat (msec): min=2, max=437, avg=82.27, stdev=31.74
lat (msec): min=2, max=437, avg=82.28, stdev=31.75
clat percentiles (msec):
| 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 74],
| 30.00th=[ 77], 40.00th=[ 77], 50.00th=[ 78], 60.00th=[ 80],
| 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 86], 95.00th=[ 93],
| 99.00th=[ 203], 99.50th=[ 397], 99.90th=[ 435], 99.95th=[ 439],
| 99.99th=[ 439]
bw ( KiB/s): min=51200, max=223232, per=99.93%, avg=198860.80, stdev=37914.36, samples=20
iops : min= 50, max= 218, avg=194.20, stdev=37.03, samples=20
lat (msec) : 4=0.10%, 100=96.99%, 250=2.10%, 500=0.82%
cpu : usr=0.00%, sys=0.16%, ctx=3910, majf=0, minf=21
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=99.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1957,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=194MiB/s (204MB/s), 194MiB/s-194MiB/s (204MB/s-204MB/s), io=1957MiB (2052MB), run=10070-10070msec
Disk stats (read/write):
sde: ios=1919/0, merge=0/0, ticks=157586/0, in_queue=157585, util=98.83%
Sequential_read_sdf: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_read_sdf: (groupid=0, jobs=1): err= 0: pid=5958: Sat Nov 25 07:07:12 2023
read: IOPS=194, BW=195MiB/s (204MB/s)(1962MiB/10073msec)
slat (usec): min=7, max=337, avg= 8.80, stdev=17.74
clat (msec): min=2, max=437, avg=82.09, stdev=30.45
lat (msec): min=2, max=437, avg=82.10, stdev=30.46
clat percentiles (msec):
| 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 77], 20.00th=[ 77],
| 30.00th=[ 77], 40.00th=[ 80], 50.00th=[ 81], 60.00th=[ 81],
| 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 82], 95.00th=[ 88],
| 99.00th=[ 144], 99.50th=[ 397], 99.90th=[ 435], 99.95th=[ 439],
| 99.99th=[ 439]
bw ( KiB/s): min=61440, max=219136, per=99.91%, avg=199270.40, stdev=33382.70, samples=20
iops : min= 60, max= 214, avg=194.60, stdev=32.60, samples=20
lat (msec) : 4=0.05%, 10=0.10%, 100=96.69%, 250=2.34%, 500=0.82%
cpu : usr=0.01%, sys=0.14%, ctx=3926, majf=0, minf=19
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=99.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1962,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=195MiB/s (204MB/s), 195MiB/s-195MiB/s (204MB/s-204MB/s), io=1962MiB (2057MB), run=10073-10073msec
Disk stats (read/write):
sdf: ios=1926/0, merge=0/0, ticks=157750/0, in_queue=157750, util=97.90%
Sequential_read_sdg: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_read_sdg: (groupid=0, jobs=1): err= 0: pid=5975: Sat Nov 25 07:07:23 2023
read: IOPS=191, BW=191MiB/s (201MB/s)(1928MiB/10076msec)
slat (usec): min=6, max=331, avg= 8.98, stdev=18.67
clat (msec): min=2, max=445, avg=83.55, stdev=31.30
lat (msec): min=2, max=445, avg=83.56, stdev=31.30
clat percentiles (msec):
| 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 75], 20.00th=[ 79],
| 30.00th=[ 80], 40.00th=[ 81], 50.00th=[ 81], 60.00th=[ 81],
| 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 84], 95.00th=[ 89],
| 99.00th=[ 136], 99.50th=[ 409], 99.90th=[ 443], 99.95th=[ 447],
| 99.99th=[ 447]
bw ( KiB/s): min=61317, max=221184, per=99.97%, avg=195885.05, stdev=33104.59, samples=20
iops : min= 59, max= 216, avg=191.25, stdev=32.52, samples=20
lat (msec) : 4=0.10%, 10=0.16%, 100=97.20%, 250=1.71%, 500=0.83%
cpu : usr=0.01%, sys=0.14%, ctx=3856, majf=0, minf=19
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=99.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1928,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=191MiB/s (201MB/s), 191MiB/s-191MiB/s (201MB/s-201MB/s), io=1928MiB (2022MB), run=10076-10076msec
Disk stats (read/write):
sdg: ios=1892/0, merge=0/0, ticks=157719/0, in_queue=157719, util=97.99%
Sequential_read_sdh: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_read_sdh: (groupid=0, jobs=1): err= 0: pid=5992: Sat Nov 25 07:07:33 2023
read: IOPS=191, BW=192MiB/s (201MB/s)(1934MiB/10075msec)
slat (usec): min=4, max=421, avg=11.24, stdev=19.40
clat (msec): min=2, max=569, avg=83.29, stdev=42.36
lat (msec): min=2, max=569, avg=83.30, stdev=42.37
clat percentiles (msec):
| 1.00th=[ 73], 5.00th=[ 73], 10.00th=[ 74], 20.00th=[ 74],
| 30.00th=[ 75], 40.00th=[ 78], 50.00th=[ 79], 60.00th=[ 79],
| 70.00th=[ 81], 80.00th=[ 88], 90.00th=[ 88], 95.00th=[ 88],
| 99.00th=[ 136], 99.50th=[ 531], 99.90th=[ 567], 99.95th=[ 567],
| 99.99th=[ 567]
bw ( KiB/s): min= 8192, max=223232, per=99.92%, avg=196403.20, stdev=45948.19, samples=20
iops : min= 8, max= 218, avg=191.80, stdev=44.87, samples=20
lat (msec) : 4=0.05%, 10=0.10%, 100=97.72%, 250=1.29%, 500=0.05%
lat (msec) : 750=0.78%
cpu : usr=0.02%, sys=0.17%, ctx=3852, majf=0, minf=529
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=99.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1934,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=192MiB/s (201MB/s), 192MiB/s-192MiB/s (201MB/s-201MB/s), io=1934MiB (2028MB), run=10075-10075msec
Disk stats (read/write):
sdh: ios=2014/0, merge=0/0, ticks=167301/0, in_queue=167302, util=98.47%
Sequential_read_sdi: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_read_sdi: (groupid=0, jobs=1): err= 0: pid=6009: Sat Nov 25 07:07:44 2023
read: IOPS=187, BW=187MiB/s (196MB/s)(1887MiB/10081msec)
slat (usec): min=4, max=351, avg= 9.15, stdev=18.00
clat (msec): min=2, max=616, avg=85.42, stdev=44.62
lat (msec): min=2, max=616, avg=85.42, stdev=44.63
clat percentiles (msec):
| 1.00th=[ 79], 5.00th=[ 79], 10.00th=[ 79], 20.00th=[ 79],
| 30.00th=[ 81], 40.00th=[ 81], 50.00th=[ 81], 60.00th=[ 82],
| 70.00th=[ 82], 80.00th=[ 83], 90.00th=[ 84], 95.00th=[ 89],
| 99.00th=[ 155], 99.50th=[ 575], 99.90th=[ 617], 99.95th=[ 617],
| 99.99th=[ 617]
bw ( KiB/s): min=14336, max=210944, per=99.96%, avg=191590.40, stdev=42309.26, samples=20
iops : min= 14, max= 206, avg=187.10, stdev=41.32, samples=20
lat (msec) : 4=0.05%, 10=0.05%, 100=97.62%, 250=1.43%, 500=0.26%
lat (msec) : 750=0.58%
cpu : usr=0.02%, sys=0.13%, ctx=3771, majf=0, minf=19
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=99.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1887,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=187MiB/s (196MB/s), 187MiB/s-187MiB/s (196MB/s-196MB/s), io=1887MiB (1979MB), run=10081-10081msec
Disk stats (read/write):
sdi: ios=1850/0, merge=0/0, ticks=157645/0, in_queue=157646, util=97.99%
Sequential_write_sda: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_write_sda: (groupid=0, jobs=1): err= 0: pid=6028: Sat Nov 25 07:07:54 2023
write: IOPS=493, BW=494MiB/s (518MB/s)(4955MiB/10032msec); 0 zone resets
slat (nsec): min=12684, max=57878, avg=16563.41, stdev=2649.58
clat (msec): min=2, max=118, avg=32.37, stdev= 7.65
lat (msec): min=2, max=118, avg=32.38, stdev= 7.65
clat percentiles (msec):
| 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31],
| 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 31],
| 70.00th=[ 32], 80.00th=[ 32], 90.00th=[ 33], 95.00th=[ 45],
| 99.00th=[ 55], 99.50th=[ 92], 99.90th=[ 120], 99.95th=[ 120],
| 99.99th=[ 120]
bw ( KiB/s): min=456704, max=530432, per=100.00%, avg=505856.00, stdev=17866.41, samples=20
iops : min= 446, max= 518, avg=494.00, stdev=17.45, samples=20
lat (msec) : 4=0.02%, 10=0.06%, 20=0.10%, 50=94.87%, 100=4.62%
lat (msec) : 250=0.32%
cpu : usr=0.66%, sys=0.25%, ctx=4959, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4955,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=494MiB/s (518MB/s), 494MiB/s-494MiB/s (518MB/s-518MB/s), io=4955MiB (5196MB), run=10032-10032msec
Disk stats (read/write):
sda: ios=46/4891, merge=0/1, ticks=7/158361, in_queue=158551, util=99.02%
Sequential_write_sdb: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_write_sdb: (groupid=0, jobs=1): err= 0: pid=6061: Sat Nov 25 07:08:04 2023
write: IOPS=190, BW=190MiB/s (199MB/s)(1917MiB/10080msec); 0 zone resets
slat (nsec): min=12243, max=50295, avg=16363.55, stdev=3452.01
clat (msec): min=48, max=165, avg=81.93, stdev=14.78
lat (msec): min=48, max=165, avg=81.95, stdev=14.79
clat percentiles (msec):
| 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 75],
| 30.00th=[ 77], 40.00th=[ 78], 50.00th=[ 79], 60.00th=[ 81],
| 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 90], 95.00th=[ 123],
| 99.00th=[ 144], 99.50th=[ 159], 99.90th=[ 167], 99.95th=[ 167],
| 99.99th=[ 167]
bw ( KiB/s): min=61440, max=219136, per=99.96%, avg=194662.40, stdev=36878.82, samples=20
iops : min= 60, max= 214, avg=190.10, stdev=36.01, samples=20
lat (msec) : 50=0.05%, 100=91.50%, 250=8.45%
cpu : usr=0.22%, sys=0.12%, ctx=1921, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=99.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1917,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=190MiB/s (199MB/s), 190MiB/s-190MiB/s (199MB/s-199MB/s), io=1917MiB (2010MB), run=10080-10080msec
Disk stats (read/write):
sdb: ios=15/1879, merge=0/0, ticks=8017/153520, in_queue=161538, util=99.70%
Sequential_write_sdc: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_write_sdc: (groupid=0, jobs=1): err= 0: pid=6085: Sat Nov 25 07:08:15 2023
write: IOPS=183, BW=184MiB/s (192MB/s)(1851MiB/10086msec); 0 zone resets
slat (nsec): min=8176, max=43802, avg=16775.24, stdev=2711.39
clat (msec): min=71, max=425, avg=87.12, stdev=31.77
lat (msec): min=71, max=425, avg=87.13, stdev=31.77
clat percentiles (msec):
| 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 78],
| 30.00th=[ 79], 40.00th=[ 79], 50.00th=[ 79], 60.00th=[ 81],
| 70.00th=[ 84], 80.00th=[ 86], 90.00th=[ 94], 95.00th=[ 138],
| 99.00th=[ 161], 99.50th=[ 384], 99.90th=[ 422], 99.95th=[ 426],
| 99.99th=[ 426]
bw ( KiB/s): min=63488, max=212992, per=99.99%, avg=187904.00, stdev=34913.35, samples=20
iops : min= 62, max= 208, avg=183.50, stdev=34.10, samples=20
lat (msec) : 100=90.92%, 250=8.21%, 500=0.86%
cpu : usr=0.25%, sys=0.10%, ctx=1857, majf=0, minf=9
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=99.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1851,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=184MiB/s (192MB/s), 184MiB/s-184MiB/s (192MB/s-192MB/s), io=1851MiB (1941MB), run=10086-10086msec
Disk stats (read/write):
sdc: ios=17/1814, merge=0/0, ticks=9845/157520, in_queue=167365, util=99.00%
Sequential_write_sdd: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_write_sdd: (groupid=0, jobs=1): err= 0: pid=6107: Sat Nov 25 07:08:25 2023
write: IOPS=180, BW=180MiB/s (189MB/s)(1818MiB/10076msec); 0 zone resets
slat (nsec): min=12895, max=53010, avg=16990.04, stdev=3821.00
clat (msec): min=13, max=146, avg=86.26, stdev=14.80
lat (msec): min=13, max=146, avg=86.28, stdev=14.80
clat percentiles (msec):
| 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 79],
| 30.00th=[ 81], 40.00th=[ 81], 50.00th=[ 81], 60.00th=[ 83],
| 70.00th=[ 89], 80.00th=[ 90], 90.00th=[ 99], 95.00th=[ 127],
| 99.00th=[ 144], 99.50th=[ 146], 99.90th=[ 146], 99.95th=[ 148],
| 99.99th=[ 148]
bw ( KiB/s): min=69632, max=217088, per=99.87%, avg=184524.80, stdev=33828.07, samples=20
iops : min= 68, max= 212, avg=180.20, stdev=33.04, samples=20
lat (msec) : 20=0.22%, 50=0.06%, 100=90.87%, 250=8.86%
cpu : usr=0.25%, sys=0.09%, ctx=1823, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=99.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1818,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=180MiB/s (189MB/s), 180MiB/s-180MiB/s (189MB/s-189MB/s), io=1818MiB (1906MB), run=10076-10076msec
Disk stats (read/write):
sdd: ios=11/1780, merge=0/0, ticks=9737/153272, in_queue=163009, util=99.70%
Sequential_write_sde: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_write_sde: (groupid=0, jobs=1): err= 0: pid=6130: Sat Nov 25 07:08:35 2023
write: IOPS=188, BW=189MiB/s (198MB/s)(1902MiB/10072msec); 0 zone resets
slat (nsec): min=10410, max=50525, avg=17241.03, stdev=3325.88
clat (msec): min=4, max=194, avg=82.47, stdev=15.55
lat (msec): min=5, max=194, avg=82.49, stdev=15.55
clat percentiles (msec):
| 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 75],
| 30.00th=[ 77], 40.00th=[ 78], 50.00th=[ 78], 60.00th=[ 81],
| 70.00th=[ 81], 80.00th=[ 86], 90.00th=[ 94], 95.00th=[ 126],
| 99.00th=[ 146], 99.50th=[ 155], 99.90th=[ 163], 99.95th=[ 194],
| 99.99th=[ 194]
bw ( KiB/s): min=69632, max=219136, per=99.87%, avg=193126.40, stdev=34225.12, samples=20
iops : min= 68, max= 214, avg=188.60, stdev=33.42, samples=20
lat (msec) : 10=0.16%, 20=0.11%, 100=90.54%, 250=9.20%
cpu : usr=0.23%, sys=0.13%, ctx=1904, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=99.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1902,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=189MiB/s (198MB/s), 189MiB/s-189MiB/s (198MB/s-198MB/s), io=1902MiB (1994MB), run=10072-10072msec
Disk stats (read/write):
sde: ios=12/1872, merge=0/0, ticks=9730/153218, in_queue=162948, util=99.71%
Sequential_write_sdf: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_write_sdf: (groupid=0, jobs=1): err= 0: pid=6153: Sat Nov 25 07:08:46 2023
write: IOPS=190, BW=190MiB/s (199MB/s)(1915MiB/10077msec); 0 zone resets
slat (nsec): min=12133, max=55303, avg=17158.40, stdev=3576.49
clat (msec): min=15, max=153, avg=81.96, stdev=14.00
lat (msec): min=15, max=153, avg=81.98, stdev=14.00
clat percentiles (msec):
| 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 77], 20.00th=[ 77],
| 30.00th=[ 77], 40.00th=[ 80], 50.00th=[ 81], 60.00th=[ 81],
| 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 87], 95.00th=[ 122],
| 99.00th=[ 144], 99.50th=[ 146], 99.90th=[ 155], 99.95th=[ 155],
| 99.99th=[ 155]
bw ( KiB/s): min=65536, max=217088, per=99.93%, avg=194457.60, stdev=34259.77, samples=20
iops : min= 64, max= 212, avg=189.90, stdev=33.46, samples=20
lat (msec) : 20=0.16%, 50=0.10%, 100=92.22%, 250=7.52%
cpu : usr=0.27%, sys=0.09%, ctx=1918, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=99.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1915,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=190MiB/s (199MB/s), 190MiB/s-190MiB/s (199MB/s-199MB/s), io=1915MiB (2008MB), run=10077-10077msec
Disk stats (read/write):
sdf: ios=17/1878, merge=0/0, ticks=8087/153461, in_queue=161547, util=99.72%
Sequential_write_sdg: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_write_sdg: (groupid=0, jobs=1): err= 0: pid=6176: Sat Nov 25 07:08:56 2023
write: IOPS=181, BW=181MiB/s (190MB/s)(1832MiB/10101msec); 0 zone resets
slat (nsec): min=14337, max=56175, avg=18743.12, stdev=3625.88
clat (msec): min=14, max=203, avg=85.78, stdev=16.83
lat (msec): min=14, max=203, avg=85.79, stdev=16.83
clat percentiles (msec):
| 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 75], 20.00th=[ 79],
| 30.00th=[ 81], 40.00th=[ 81], 50.00th=[ 83], 60.00th=[ 84],
| 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 102], 95.00th=[ 132],
| 99.00th=[ 148], 99.50th=[ 159], 99.90th=[ 199], 99.95th=[ 203],
| 99.99th=[ 203]
bw ( KiB/s): min=65536, max=221184, per=100.00%, avg=185958.40, stdev=34708.68, samples=20
iops : min= 64, max= 216, avg=181.60, stdev=33.90, samples=20
lat (msec) : 20=0.16%, 50=0.11%, 100=89.03%, 250=10.70%
cpu : usr=0.26%, sys=0.14%, ctx=1836, majf=0, minf=9
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=99.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1832,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=181MiB/s (190MB/s), 181MiB/s-181MiB/s (190MB/s-190MB/s), io=1832MiB (1921MB), run=10101-10101msec
Disk stats (read/write):
sdg: ios=15/1800, merge=0/0, ticks=9771/152667, in_queue=162438, util=99.70%
Sequential_write_sdh: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_write_sdh: (groupid=0, jobs=1): err= 0: pid=6200: Sat Nov 25 07:09:06 2023
write: IOPS=189, BW=190MiB/s (199MB/s)(1909MiB/10073msec); 0 zone resets
slat (nsec): min=11581, max=62989, avg=16737.97, stdev=3356.04
clat (msec): min=3, max=154, avg=82.35, stdev=15.37
lat (msec): min=4, max=154, avg=82.37, stdev=15.37
clat percentiles (msec):
| 1.00th=[ 73], 5.00th=[ 73], 10.00th=[ 74], 20.00th=[ 74],
| 30.00th=[ 75], 40.00th=[ 78], 50.00th=[ 79], 60.00th=[ 79],
| 70.00th=[ 83], 80.00th=[ 88], 90.00th=[ 88], 95.00th=[ 121],
| 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155],
| 99.99th=[ 155]
bw ( KiB/s): min=73728, max=225280, per=99.89%, avg=193843.20, stdev=33980.27, samples=20
iops : min= 72, max= 220, avg=189.30, stdev=33.18, samples=20
lat (msec) : 4=0.05%, 10=0.10%, 20=0.10%, 100=92.40%, 250=7.33%
cpu : usr=0.23%, sys=0.12%, ctx=1914, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=99.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1909,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=190MiB/s (199MB/s), 190MiB/s-190MiB/s (199MB/s-199MB/s), io=1909MiB (2002MB), run=10073-10073msec
Disk stats (read/write):
sdh: ios=11/1873, merge=0/0, ticks=9547/153661, in_queue=163209, util=100.00%
Sequential_write_sdi: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_write_sdi: (groupid=0, jobs=1): err= 0: pid=6223: Sat Nov 25 07:09:17 2023
write: IOPS=181, BW=182MiB/s (190MB/s)(1840MiB/10133msec); 0 zone resets
slat (nsec): min=11491, max=56606, avg=16708.53, stdev=3166.86
clat (msec): min=9, max=226, avg=85.79, stdev=17.43
lat (msec): min=9, max=226, avg=85.81, stdev=17.43
clat percentiles (msec):
| 1.00th=[ 78], 5.00th=[ 79], 10.00th=[ 79], 20.00th=[ 79],
| 30.00th=[ 81], 40.00th=[ 81], 50.00th=[ 81], 60.00th=[ 82],
| 70.00th=[ 82], 80.00th=[ 84], 90.00th=[ 101], 95.00th=[ 133],
| 99.00th=[ 148], 99.50th=[ 163], 99.90th=[ 226], 99.95th=[ 228],
| 99.99th=[ 228]
bw ( KiB/s): min=71680, max=210944, per=100.00%, avg=186880.00, stdev=30708.32, samples=20
iops : min= 70, max= 206, avg=182.50, stdev=29.99, samples=20
lat (msec) : 10=0.05%, 20=0.22%, 100=89.73%, 250=10.00%
cpu : usr=0.25%, sys=0.09%, ctx=1844, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=99.2%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1840,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=182MiB/s (190MB/s), 182MiB/s-182MiB/s (190MB/s-190MB/s), io=1840MiB (1929MB), run=10133-10133msec
Disk stats (read/write):
sdi: ios=11/1807, merge=0/0, ticks=7705/152745, in_queue=160450, util=99.71%
-bash: :/dev/sda:/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdh:/dev/sdi: No such file or directory
Sequential_read: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_read: (groupid=0, jobs=1): err= 0: pid=6248: Sat Nov 25 07:09:27 2023
read: IOPS=1008, BW=1009MiB/s (1058MB/s)(9.89GiB/10038msec)
slat (usec): min=5, max=333, avg= 8.76, stdev= 8.46
clat (usec): min=1854, max=479666, avg=15842.58, stdev=20490.49
lat (usec): min=1862, max=479675, avg=15851.34, stdev=20490.85
clat percentiles (usec):
| 1.00th=[ 1860], 5.00th=[ 1860], 10.00th=[ 1860], 20.00th=[ 1860],
| 30.00th=[ 2212], 40.00th=[ 2900], 50.00th=[ 9372], 60.00th=[ 21890],
| 70.00th=[ 25560], 80.00th=[ 27919], 90.00th=[ 33817], 95.00th=[ 38011],
| 99.00th=[ 54264], 99.50th=[ 73925], 99.90th=[371196], 99.95th=[387974],
| 99.99th=[404751]
bw ( KiB/s): min=172032, max=1099776, per=100.00%, avg=1035571.20, stdev=203475.59, samples=20
iops : min= 168, max= 1074, avg=1011.30, stdev=198.71, samples=20
lat (msec) : 2=28.39%, 4=18.59%, 10=3.15%, 20=3.54%, 50=45.12%
lat (msec) : 100=0.98%, 250=0.07%, 500=0.16%
cpu : usr=0.02%, sys=0.70%, ctx=18695, majf=0, minf=27
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=10128,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=1009MiB/s (1058MB/s), 1009MiB/s-1009MiB/s (1058MB/s-1058MB/s), io=9.89GiB (10.6GB), run=10038-10038msec
Disk stats (read/write):
sdb: ios=1250/0, merge=0/0, ticks=4152/0, in_queue=4152, util=45.35%
sdc: ios=1248/0, merge=0/0, ticks=4282/0, in_queue=4283, util=43.08%
sdd: ios=1250/0, merge=0/0, ticks=5029/0, in_queue=5029, util=42.83%
sde: ios=1250/0, merge=0/0, ticks=3822/0, in_queue=3822, util=41.94%
sdf: ios=1247/0, merge=0/0, ticks=39129/0, in_queue=39128, util=63.56%
sdg: ios=1247/0, merge=0/0, ticks=36051/0, in_queue=36051, util=63.44%
sdh: ios=1247/0, merge=0/0, ticks=31968/0, in_queue=31968, util=63.74%
sdi: ios=1244/0, merge=0/0, ticks=33373/0, in_queue=33373, util=62.00%
Sequential_write: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_write: (groupid=0, jobs=1): err= 0: pid=6265: Sat Nov 25 07:09:37 2023
write: IOPS=962, BW=963MiB/s (1010MB/s)(9654MiB/10027msec); 0 zone resets
slat (usec): min=8, max=9973, avg=18.25, stdev=101.48
clat (usec): min=1946, max=127588, avg=16580.97, stdev=25261.20
lat (usec): min=1958, max=127601, avg=16599.23, stdev=25260.54
clat percentiles (usec):
| 1.00th=[ 1942], 5.00th=[ 1958], 10.00th=[ 1958], 20.00th=[ 1958],
| 30.00th=[ 1958], 40.00th=[ 1958], 50.00th=[ 6194], 60.00th=[ 8586],
| 70.00th=[ 12518], 80.00th=[ 22676], 90.00th=[ 58459], 95.00th=[ 90702],
| 99.00th=[ 98042], 99.50th=[101188], 99.90th=[115868], 99.95th=[121111],
| 99.99th=[127402]
bw ( KiB/s): min=903168, max=999424, per=100.00%, avg=987033.60, stdev=22370.42, samples=20
iops : min= 882, max= 976, avg=963.90, stdev=21.85, samples=20
lat (msec) : 2=45.87%, 4=3.40%, 10=15.39%, 20=14.67%, 50=8.41%
lat (msec) : 100=11.57%, 250=0.69%
cpu : usr=1.14%, sys=0.71%, ctx=9409, majf=0, minf=17
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,9654,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=963MiB/s (1010MB/s), 963MiB/s-963MiB/s (1010MB/s-1010MB/s), io=9654MiB (10.1GB), run=10027-10027msec
Disk stats (read/write):
sdb: ios=47/1276, merge=0/0, ticks=26/3078, in_queue=3103, util=36.22%
sdc: ios=50/1309, merge=0/0, ticks=319/3298, in_queue=3617, util=36.00%
sdd: ios=47/1272, merge=0/0, ticks=21/2658, in_queue=2678, util=35.85%
sde: ios=47/1287, merge=0/0, ticks=27/2648, in_queue=2675, util=36.28%
sdf: ios=47/1262, merge=0/0, ticks=28/16940, in_queue=16967, util=49.93%
sdg: ios=47/1309, merge=0/0, ticks=32/16583, in_queue=16615, util=47.32%
sdh: ios=47/1225, merge=0/17, ticks=22/58480, in_queue=58502, util=68.77%
sdi: ios=45/1254, merge=0/12, ticks=3921/60528, in_queue=64449, util=45.71%
Sequential_read_: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_read_: (groupid=0, jobs=1): err= 0: pid=6369: Sat Nov 25 07:09:48 2023
read: IOPS=774, BW=774MiB/s (812MB/s)(7787MiB/10058msec)
slat (usec): min=4, max=261, avg= 9.54, stdev= 7.22
clat (usec): min=1854, max=114578, avg=20644.87, stdev=19679.87
lat (usec): min=1863, max=114585, avg=20654.41, stdev=19679.87
clat percentiles (usec):
| 1.00th=[ 1860], 5.00th=[ 1860], 10.00th=[ 1860], 20.00th=[ 1860],
| 30.00th=[ 4424], 40.00th=[ 9241], 50.00th=[ 14353], 60.00th=[ 21627],
| 70.00th=[ 29754], 80.00th=[ 38011], 90.00th=[ 50070], 95.00th=[ 58459],
| 99.00th=[ 79168], 99.50th=[ 83362], 99.90th=[ 85459], 99.95th=[100140],
| 99.99th=[114820]
bw ( KiB/s): min=720896, max=849920, per=100.00%, avg=795852.80, stdev=34181.88, samples=20
iops : min= 704, max= 830, avg=777.20, stdev=33.38, samples=20
lat (msec) : 2=25.34%, 4=3.48%, 10=13.30%, 20=16.21%, 50=31.55%
lat (msec) : 100=10.08%, 250=0.04%
cpu : usr=0.03%, sys=0.62%, ctx=15513, majf=0, minf=531
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=7787,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=774MiB/s (812MB/s), 774MiB/s-774MiB/s (812MB/s-812MB/s), io=7787MiB (8165MB), run=10058-10058msec
Disk stats (read/write):
sdb: ios=2047/0, merge=0/0, ticks=11419/0, in_queue=11420, util=65.48%
sdc: ios=2058/0, merge=0/0, ticks=43443/0, in_queue=43443, util=69.56%
sdd: ios=2078/0, merge=0/0, ticks=70748/0, in_queue=70747, util=76.74%
sde: ios=2079/0, merge=0/0, ticks=41804/0, in_queue=41804, util=70.25%
Sequential_write_: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_write_: (groupid=0, jobs=1): err= 0: pid=6411: Sat Nov 25 07:09:58 2023
write: IOPS=781, BW=782MiB/s (820MB/s)(7851MiB/10040msec); 0 zone resets
slat (nsec): min=9488, max=86012, avg=16598.34, stdev=4559.55
clat (usec): min=1947, max=130905, avg=20437.90, stdev=22163.60
lat (usec): min=1958, max=130925, avg=20454.50, stdev=22163.29
clat percentiles (usec):
| 1.00th=[ 1958], 5.00th=[ 1958], 10.00th=[ 1958], 20.00th=[ 1958],
| 30.00th=[ 1958], 40.00th=[ 1958], 50.00th=[ 11076], 60.00th=[ 21627],
| 70.00th=[ 31065], 80.00th=[ 40109], 90.00th=[ 50594], 95.00th=[ 65274],
| 99.00th=[ 83362], 99.50th=[ 87557], 99.90th=[116917], 99.95th=[121111],
| 99.99th=[130548]
bw ( KiB/s): min=739328, max=856064, per=100.00%, avg=802406.40, stdev=32338.06, samples=20
iops : min= 722, max= 836, avg=783.60, stdev=31.58, samples=20
lat (msec) : 2=41.13%, 4=1.57%, 10=5.74%, 20=10.08%, 50=30.86%
lat (msec) : 100=10.44%, 250=0.18%
cpu : usr=0.84%, sys=0.63%, ctx=7766, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,7851,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=782MiB/s (820MB/s), 782MiB/s-782MiB/s (820MB/s-820MB/s), io=7851MiB (8232MB), run=10040-10040msec
Disk stats (read/write):
sdb: ios=46/1940, merge=0/0, ticks=6/26396, in_queue=26402, util=63.08%
sdc: ios=46/1932, merge=0/0, ticks=6/32032, in_queue=32038, util=67.66%
sdd: ios=46/1933, merge=0/0, ticks=26/82830, in_queue=82856, util=74.50%
sde: ios=47/1939, merge=0/0, ticks=24/16488, in_queue=16511, util=65.73%
Sequential_read_: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_read_: (groupid=0, jobs=1): err= 0: pid=6464: Sat Nov 25 07:10:08 2023
read: IOPS=528, BW=528MiB/s (554MB/s)(5296MiB/10026msec)
slat (usec): min=4, max=390, avg= 9.94, stdev=11.85
clat (usec): min=1946, max=394297, avg=30270.00, stdev=21319.40
lat (usec): min=1957, max=394307, avg=30279.94, stdev=21323.94
clat percentiles (msec):
| 1.00th=[ 13], 5.00th=[ 20], 10.00th=[ 21], 20.00th=[ 22],
| 30.00th=[ 25], 40.00th=[ 26], 50.00th=[ 28], 60.00th=[ 28],
| 70.00th=[ 33], 80.00th=[ 37], 90.00th=[ 42], 95.00th=[ 46],
| 99.00th=[ 64], 99.50th=[ 73], 99.90th=[ 384], 99.95th=[ 393],
| 99.99th=[ 397]
bw ( KiB/s): min=143360, max=567296, per=99.96%, avg=540672.00, stdev=93601.49, samples=20
iops : min= 140, max= 554, avg=528.00, stdev=91.41, samples=20
lat (msec) : 2=0.02%, 4=0.08%, 10=0.57%, 20=5.27%, 50=91.58%
lat (msec) : 100=2.19%, 500=0.30%
cpu : usr=0.04%, sys=0.34%, ctx=9497, majf=0, minf=22
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5296,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=528MiB/s (554MB/s), 528MiB/s-528MiB/s (554MB/s-554MB/s), io=5296MiB (5553MB), run=10026-10026msec
Disk stats (read/write):
sdf: ios=1304/0, merge=0/0, ticks=35258/0, in_queue=35257, util=72.77%
sdg: ios=1306/0, merge=0/0, ticks=42498/0, in_queue=42498, util=69.10%
sdh: ios=1306/0, merge=0/0, ticks=38375/0, in_queue=38375, util=67.26%
sdi: ios=1303/0, merge=0/0, ticks=41770/0, in_queue=41770, util=66.34%
Sequential_write_: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Sequential_write_: (groupid=0, jobs=1): err= 0: pid=6509: Sat Nov 25 07:10:19 2023
write: IOPS=484, BW=485MiB/s (508MB/s)(4866MiB/10036msec); 0 zone resets
slat (nsec): min=10229, max=51808, avg=17142.95, stdev=3146.21
clat (msec): min=7, max=133, avg=32.97, stdev=28.56
lat (msec): min=7, max=133, avg=32.99, stdev=28.56
clat percentiles (msec):
| 1.00th=[ 9], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 11],
| 30.00th=[ 13], 40.00th=[ 16], 50.00th=[ 31], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 39], 90.00th=[ 99], 95.00th=[ 101],
| 99.00th=[ 102], 99.50th=[ 103], 99.90th=[ 104], 99.95th=[ 117],
| 99.99th=[ 134]
bw ( KiB/s): min=444416, max=501760, per=100.00%, avg=496742.40, stdev=12368.13, samples=20
iops : min= 434, max= 490, avg=485.10, stdev=12.08, samples=20
lat (msec) : 10=19.01%, 20=21.87%, 50=45.11%, 100=9.43%, 250=4.58%
cpu : usr=0.57%, sys=0.35%, ctx=4831, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4866,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=485MiB/s (508MB/s), 485MiB/s-485MiB/s (508MB/s-508MB/s), io=4866MiB (5102MB), run=10036-10036msec
Disk stats (read/write):
sdf: ios=47/1202, merge=0/0, ticks=61/29526, in_queue=29586, util=71.79%
sdg: ios=46/1201, merge=0/0, ticks=25/24333, in_queue=24357, util=76.33%
sdh: ios=27/1190, merge=0/0, ticks=32/80830, in_queue=80861, util=82.32%
sdi: ios=43/1201, merge=0/0, ticks=44/22736, in_queue=22780, util=54.66%
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment