fio on revival+ocz raid0 4.10.0-38

gistfile1.txt

fio --loops=5 --size=1000m --filename=/home/chocobo/fiotest.tmp --stonewall --ioengine=libaio --direct=1   --name=Seqread --bs=1m --rw=read   --name=Seqwrite --bs=1m --rw=write   --name=512Kread --bs=512k --rw=randread   --name=512Kwrite --bs=512k --rw=randwrite   --name=4kQD32read --bs=4k --iodepth=32 --rw=randread --name=4kQD32write --bs=4k --iodepth=32 --rw=randwrite
Seqread: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=1
Seqwrite: (g=1): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=1test.tmp --stonewall --ioengine=libaio --direct=1   --name=Seqread --bs=1m --rw=read   --name=Seqwrite --bs=1m --rw=write   --name=512Kread --bs=512k --rw=randread512Kread: (g=2): rw=randread, bs=512K-512K/512K-512K/512K-512K, ioengine=libaio, iodepth=1
512Kwrite: (g=3): rw=randwrite, bs=512K-512K/512K-512K/512K-512K, ioengine=libaio, iodepth=1
4kQD32read: (g=4): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
4kQD32write: (g=5): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.2.10
Starting 6 processes
Seqread: Laying out IO file(s) (1 file(s) / 1000MB)
Jobs: 1 (f=1): [_(5),w(1)] [100.0% done] [0KB/369.5MB/0KB /s] [0/94.5K/0 iops] [eta 00m:00s]       
Seqread: (groupid=0, jobs=1): err= 0: pid=4296: Thu Jan 11 18:04:48 2018
  read : io=5000.0MB, bw=508795KB/s, iops=496, runt= 10063msec
    slat (usec): min=41, max=418, avg=88.95, stdev=13.91
    clat (usec): min=1837, max=3132, avg=1916.49, stdev=44.50
     lat (usec): min=1887, max=3246, avg=2006.10, stdev=48.33
    clat percentiles (usec):
     |  1.00th=[ 1848],  5.00th=[ 1880], 10.00th=[ 1880], 20.00th=[ 1896],
     | 30.00th=[ 1912], 40.00th=[ 1912], 50.00th=[ 1928], 60.00th=[ 1928],
     | 70.00th=[ 1928], 80.00th=[ 1928], 90.00th=[ 1928], 95.00th=[ 1928],
     | 99.00th=[ 1976], 99.50th=[ 2160], 99.90th=[ 2736], 99.95th=[ 2864],
     | 99.99th=[ 3120]
    bw (KB  /s): min=504846, max=518144, per=100.00%, avg=509148.68, stdev=3552.44
    lat (msec) : 2=99.02%, 4=0.98%
  cpu          : usr=0.87%, sys=4.89%, ctx=5009, majf=0, minf=266
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=5000/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Seqwrite: (groupid=1, jobs=1): err= 0: pid=4297: Thu Jan 11 18:04:48 2018
  write: io=5000.0MB, bw=483156KB/s, iops=471, runt= 10597msec
    slat (usec): min=71, max=317, avg=129.19, stdev=15.14
    clat (usec): min=1841, max=20200, avg=1982.08, stdev=363.33
     lat (usec): min=2005, max=20329, avg=2111.96, stdev=363.98
    clat percentiles (usec):
     |  1.00th=[ 1928],  5.00th=[ 1944], 10.00th=[ 1960], 20.00th=[ 1960],
     | 30.00th=[ 1960], 40.00th=[ 1976], 50.00th=[ 1976], 60.00th=[ 1976],
     | 70.00th=[ 1976], 80.00th=[ 1976], 90.00th=[ 1992], 95.00th=[ 1992],
     | 99.00th=[ 2008], 99.50th=[ 2024], 99.90th=[ 4256], 99.95th=[13504],
     | 99.99th=[20096]
    bw (KB  /s): min=461924, max=489472, per=100.00%, avg=483719.70, stdev=6836.90
    lat (msec) : 2=96.44%, 4=3.46%, 10=0.04%, 20=0.04%, 50=0.02%
  cpu          : usr=3.06%, sys=4.34%, ctx=5019, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=5000/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
512Kread: (groupid=2, jobs=1): err= 0: pid=4303: Thu Jan 11 18:04:48 2018
  read : io=5000.0MB, bw=236217KB/s, iops=461, runt= 21675msec
    slat (usec): min=33, max=368, avg=55.74, stdev= 9.78
    clat (usec): min=2003, max=4668, avg=2104.56, stdev=42.83
     lat (usec): min=2041, max=4728, avg=2160.90, stdev=44.65
    clat percentiles (usec):
     |  1.00th=[ 2024],  5.00th=[ 2064], 10.00th=[ 2064], 20.00th=[ 2064],
     | 30.00th=[ 2096], 40.00th=[ 2096], 50.00th=[ 2128], 60.00th=[ 2128],
     | 70.00th=[ 2128], 80.00th=[ 2128], 90.00th=[ 2128], 95.00th=[ 2128],
     | 99.00th=[ 2160], 99.50th=[ 2160], 99.90th=[ 2160], 99.95th=[ 2160],
     | 99.99th=[ 3472]
    bw (KB  /s): min=234027, max=237568, per=100.00%, avg=236370.17, stdev=876.23
    lat (msec) : 4=99.99%, 10=0.01%
  cpu          : usr=0.37%, sys=3.21%, ctx=10019, majf=0, minf=137
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=10000/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
512Kwrite: (groupid=3, jobs=1): err= 0: pid=4318: Thu Jan 11 18:04:48 2018
  write: io=5000.0MB, bw=235890KB/s, iops=460, runt= 21705msec
    slat (usec): min=27, max=173, avg=76.74, stdev=12.80
    clat (msec): min=1, max=20, avg= 2.09, stdev= 1.33
     lat (msec): min=1, max=20, avg= 2.16, stdev= 1.33
    clat percentiles (usec):
     |  1.00th=[ 1912],  5.00th=[ 1928], 10.00th=[ 1944], 20.00th=[ 1960],
     | 30.00th=[ 1960], 40.00th=[ 1960], 50.00th=[ 1976], 60.00th=[ 1976],
     | 70.00th=[ 1976], 80.00th=[ 1976], 90.00th=[ 1992], 95.00th=[ 1992],
     | 99.00th=[ 2064], 99.50th=[16768], 99.90th=[19072], 99.95th=[19072],
     | 99.99th=[19072]
    bw (KB  /s): min=55782, max=254976, per=99.99%, avg=235874.76, stdev=46186.90
    lat (msec) : 2=95.60%, 4=3.54%, 10=0.07%, 20=0.78%, 50=0.01%
  cpu          : usr=1.79%, sys=3.02%, ctx=10019, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=10000/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
4kQD32read: (groupid=4, jobs=1): err= 0: pid=4327: Thu Jan 11 18:04:48 2018
  read : io=5000.0MB, bw=392427KB/s, iops=98106, runt= 13047msec
    slat (usec): min=3, max=334, avg= 7.08, stdev= 3.90
    clat (usec): min=33, max=5344, avg=317.54, stdev=170.38
     lat (usec): min=67, max=5352, avg=324.81, stdev=170.49
    clat percentiles (usec):
     |  1.00th=[  129],  5.00th=[  147], 10.00th=[  165], 20.00th=[  199],
     | 30.00th=[  229], 40.00th=[  258], 50.00th=[  286], 60.00th=[  314],
     | 70.00th=[  350], 80.00th=[  394], 90.00th=[  470], 95.00th=[  604],
     | 99.00th=[ 1048], 99.50th=[ 1224], 99.90th=[ 1576], 99.95th=[ 1672],
     | 99.99th=[ 1864]
    bw (KB  /s): min=370064, max=414424, per=100.00%, avg=396458.24, stdev=12522.98
    lat (usec) : 50=0.01%, 100=0.01%, 250=37.46%, 500=54.47%, 750=5.17%
    lat (usec) : 1000=1.69%
    lat (msec) : 2=1.20%, 4=0.01%, 10=0.01%
  cpu          : usr=16.46%, sys=76.81%, ctx=104932, majf=0, minf=41
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=1280000/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
4kQD32write: (groupid=5, jobs=1): err= 0: pid=4328: Thu Jan 11 18:04:48 2018
  write: io=5000.0MB, bw=213805KB/s, iops=53451, runt= 23947msec
    slat (usec): min=3, max=642, avg= 7.40, stdev= 3.52
    clat (usec): min=2, max=1485.6K, avg=589.68, stdev=17694.03
     lat (usec): min=33, max=1485.6K, avg=597.28, stdev=17694.06
    clat percentiles (usec):
     |  1.00th=[   39],  5.00th=[   65], 10.00th=[   98], 20.00th=[  165],
     | 30.00th=[  221], 40.00th=[  274], 50.00th=[  322], 60.00th=[  366],
     | 70.00th=[  414], 80.00th=[  470], 90.00th=[  532], 95.00th=[  572],
     | 99.00th=[  708], 99.50th=[ 4576], 99.90th=[ 8512], 99.95th=[27008],
     | 99.99th=[1433600]
    bw (KB  /s): min=11110, max=400536, per=100.00%, avg=292837.78, stdev=160896.39
    lat (usec) : 4=0.01%, 20=0.01%, 50=2.60%, 100=7.58%, 250=25.21%
    lat (usec) : 500=49.47%, 750=14.18%, 1000=0.04%
    lat (msec) : 2=0.02%, 4=0.01%, 10=0.79%, 20=0.01%, 50=0.07%
    lat (msec) : 100=0.01%, 2000=0.01%
  cpu          : usr=9.87%, sys=43.05%, ctx=135543, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1280000/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=5000.0MB, aggrb=508794KB/s, minb=508794KB/s, maxb=508794KB/s, mint=10063msec, maxt=10063msec

Run status group 1 (all jobs):
  WRITE: io=5000.0MB, aggrb=483155KB/s, minb=483155KB/s, maxb=483155KB/s, mint=10597msec, maxt=10597msec

Run status group 2 (all jobs):
   READ: io=5000.0MB, aggrb=236216KB/s, minb=236216KB/s, maxb=236216KB/s, mint=21675msec, maxt=21675msec

Run status group 3 (all jobs):
  WRITE: io=5000.0MB, aggrb=235890KB/s, minb=235890KB/s, maxb=235890KB/s, mint=21705msec, maxt=21705msec

Run status group 4 (all jobs):
   READ: io=5000.0MB, aggrb=392427KB/s, minb=392427KB/s, maxb=392427KB/s, mint=13047msec, maxt=13047msec

Run status group 5 (all jobs):
  WRITE: io=5000.0MB, aggrb=213805KB/s, minb=213805KB/s, maxb=213805KB/s, mint=23947msec, maxt=23947msec

Disk stats (read/write):
    md1: ios=1300000/1291493, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=650000/650062, aggrmerge=0/53, aggrticks=201068/370850, aggrin_queue=571872, aggrutil=73.63%
  sdc: ios=650000/650051, merge=0/23, ticks=208640/219848, in_queue=428396, util=66.02%
  sda: ios=650000/650074, merge=0/83, ticks=193496/521852, in_queue=715348, util=73.63%