gistfile1.txt

$ fio ~/sr.fio
sequential-read: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K, ioengine=libaio, iodepth=32
...
fio-2.2.10
Starting 20 processes
Jobs: 20 (f=20), CR=3840/0 IOPS: [R(20)] [100.0% done] [1921MB/0KB/0KB /s] [3841/0/0 iops] [eta 00m:00s]
sequential-read: (groupid=0, jobs=1): err= 0: pid=20500: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170664msec
    slat (usec): min=56, max=2596, avg=142.09, stdev=42.15
    clat (usec): min=371, max=45930, avg=999.17, stdev=1253.47
     lat (usec): min=436, max=46196, avg=1141.68, stdev=1259.29
    clat percentiles (usec):
     |  1.00th=[  446],  5.00th=[  494], 10.00th=[  532], 20.00th=[  596],
     | 30.00th=[  660], 40.00th=[  716], 50.00th=[  780], 60.00th=[  868],
     | 70.00th=[ 1004], 80.00th=[ 1256], 90.00th=[ 1592], 95.00th=[ 1832],
     | 99.00th=[ 2256], 99.50th=[ 5152], 99.90th=[19328], 99.95th=[21120],
     | 99.99th=[38144]
    bw (KB  /s): min=92160, max=104634, per=5.00%, avg=98396.44, stdev=513.45
    lat (usec) : 500=5.96%, 750=39.34%, 1000=24.31%
    lat (msec) : 2=27.76%, 4=2.08%, 10=0.14%, 20=0.33%, 50=0.08%
  cpu          : usr=0.17%, sys=3.39%, ctx=65535, majf=0, minf=528
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20501: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170664msec
    slat (usec): min=42, max=2279, avg=126.01, stdev=36.08
    clat (usec): min=381, max=33790, avg=925.97, stdev=1121.08
     lat (usec): min=436, max=34030, avg=1052.41, stdev=1127.99
    clat percentiles (usec):
     |  1.00th=[  434],  5.00th=[  470], 10.00th=[  494], 20.00th=[  540],
     | 30.00th=[  596], 40.00th=[  660], 50.00th=[  716], 60.00th=[  796],
     | 70.00th=[  908], 80.00th=[ 1112], 90.00th=[ 1528], 95.00th=[ 1800],
     | 99.00th=[ 2256], 99.50th=[ 5984], 99.90th=[18304], 99.95th=[20352],
     | 99.99th=[33536]
    bw (KB  /s): min=93090, max=104448, per=5.00%, avg=98392.40, stdev=468.48
    lat (usec) : 500=11.00%, 750=43.71%, 1000=20.79%
    lat (msec) : 2=21.96%, 4=2.00%, 10=0.16%, 20=0.33%, 50=0.05%
  cpu          : usr=0.21%, sys=3.05%, ctx=65535, majf=0, minf=528
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20502: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170664msec
    slat (usec): min=55, max=1995, avg=141.35, stdev=34.66
    clat (usec): min=359, max=49527, avg=999.11, stdev=1280.18
     lat (usec): min=447, max=49685, avg=1140.89, stdev=1285.43
    clat percentiles (usec):
     |  1.00th=[  446],  5.00th=[  494], 10.00th=[  532], 20.00th=[  604],
     | 30.00th=[  660], 40.00th=[  716], 50.00th=[  780], 60.00th=[  868],
     | 70.00th=[  996], 80.00th=[ 1256], 90.00th=[ 1576], 95.00th=[ 1816],
     | 99.00th=[ 2288], 99.50th=[ 5792], 99.90th=[19328], 99.95th=[21888],
     | 99.99th=[41728]
    bw (KB  /s): min=92458, max=104842, per=5.00%, avg=98400.79, stdev=505.26
    lat (usec) : 500=5.68%, 750=39.17%, 1000=25.37%
    lat (msec) : 2=27.15%, 4=2.07%, 10=0.16%, 20=0.34%, 50=0.07%
  cpu          : usr=0.18%, sys=3.37%, ctx=65543, majf=0, minf=529
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20503: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170664msec
    slat (usec): min=55, max=2427, avg=141.35, stdev=34.98
    clat (usec): min=386, max=43053, avg=1021.60, stdev=1257.95
     lat (usec): min=467, max=43286, avg=1163.36, stdev=1262.94
    clat percentiles (usec):
     |  1.00th=[  450],  5.00th=[  494], 10.00th=[  532], 20.00th=[  604],
     | 30.00th=[  668], 40.00th=[  732], 50.00th=[  804], 60.00th=[  892],
     | 70.00th=[ 1048], 80.00th=[ 1320], 90.00th=[ 1624], 95.00th=[ 1848],
     | 99.00th=[ 2288], 99.50th=[ 5920], 99.90th=[19072], 99.95th=[23168],
     | 99.99th=[42752]
    bw (KB  /s): min=92907, max=103825, per=5.00%, avg=98391.42, stdev=458.33
    lat (usec) : 500=5.48%, 750=37.35%, 1000=24.79%
    lat (msec) : 2=29.60%, 4=2.25%, 10=0.12%, 20=0.33%, 50=0.08%
  cpu          : usr=0.16%, sys=3.39%, ctx=65552, majf=0, minf=528
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20504: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170663msec
    slat (usec): min=56, max=2693, avg=141.89, stdev=38.74
    clat (usec): min=372, max=38905, avg=1048.08, stdev=1229.54
     lat (usec): min=462, max=39157, avg=1190.39, stdev=1234.97
    clat percentiles (usec):
     |  1.00th=[  454],  5.00th=[  502], 10.00th=[  548], 20.00th=[  620],
     | 30.00th=[  684], 40.00th=[  748], 50.00th=[  828], 60.00th=[  924],
     | 70.00th=[ 1096], 80.00th=[ 1368], 90.00th=[ 1672], 95.00th=[ 1880],
     | 99.00th=[ 2288], 99.50th=[ 5280], 99.90th=[20352], 99.95th=[23424],
     | 99.99th=[38656]
    bw (KB  /s): min=91792, max=105261, per=5.00%, avg=98395.08, stdev=544.71
    lat (usec) : 500=4.74%, 750=35.15%, 1000=25.21%
    lat (msec) : 2=31.69%, 4=2.67%, 10=0.14%, 20=0.29%, 50=0.11%
  cpu          : usr=0.22%, sys=3.35%, ctx=65531, majf=0, minf=530
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20505: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170664msec
    slat (usec): min=42, max=1854, avg=119.25, stdev=29.18
    clat (usec): min=376, max=66740, avg=980.69, stdev=1569.28
     lat (usec): min=461, max=67133, avg=1100.37, stdev=1571.96
    clat percentiles (usec):
     |  1.00th=[  442],  5.00th=[  482], 10.00th=[  516], 20.00th=[  580],
     | 30.00th=[  660], 40.00th=[  756], 50.00th=[  876], 60.00th=[  996],
     | 70.00th=[ 1112], 80.00th=[ 1192], 90.00th=[ 1272], 95.00th=[ 1336],
     | 99.00th=[ 1480], 99.50th=[ 6240], 99.90th=[20096], 99.95th=[37120],
     | 99.99th=[59136]
    bw (KB  /s): min=92180, max=105051, per=5.00%, avg=98343.53, stdev=501.20
    lat (usec) : 500=7.75%, 750=31.88%, 1000=20.47%
    lat (msec) : 2=39.25%, 4=0.06%, 10=0.14%, 20=0.34%, 50=0.07%
    lat (msec) : 100=0.03%
  cpu          : usr=0.22%, sys=2.93%, ctx=65430, majf=0, minf=529
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20506: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98305KB/s, iops=192, runt=170665msec
    slat (usec): min=55, max=5470, avg=133.00, stdev=38.83
    clat (usec): min=400, max=120539, avg=1962.41, stdev=3496.68
     lat (usec): min=472, max=120747, avg=2095.76, stdev=3502.66
    clat percentiles (usec):
     |  1.00th=[  502],  5.00th=[  628], 10.00th=[  724], 20.00th=[  884],
     | 30.00th=[ 1064], 40.00th=[ 1320], 50.00th=[ 1816], 60.00th=[ 2288],
     | 70.00th=[ 2544], 80.00th=[ 2736], 90.00th=[ 2896], 95.00th=[ 3024],
     | 99.00th=[ 3248], 99.50th=[ 6880], 99.90th=[42240], 99.95th=[105984],
     | 99.99th=[120320]
    bw (KB  /s): min=92160, max=104634, per=5.00%, avg=98359.62, stdev=487.55
    lat (usec) : 500=0.97%, 750=10.75%, 1000=15.04%
    lat (msec) : 2=26.13%, 4=46.55%, 10=0.13%, 20=0.31%, 50=0.03%
    lat (msec) : 100=0.04%, 250=0.06%
  cpu          : usr=0.15%, sys=3.32%, ctx=65442, majf=0, minf=528
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20507: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170663msec
    slat (usec): min=57, max=2050, avg=135.88, stdev=26.79
    clat (usec): min=2, max=118632, avg=644.53, stdev=3080.44
     lat (usec): min=416, max=119034, avg=780.87, stdev=3086.30
    clat percentiles (usec):
     |  1.00th=[  402],  5.00th=[  426], 10.00th=[  438], 20.00th=[  454],
     | 30.00th=[  466], 40.00th=[  478], 50.00th=[  490], 60.00th=[  498],
     | 70.00th=[  510], 80.00th=[  524], 90.00th=[  556], 95.00th=[  588],
     | 99.00th=[  764], 99.50th=[ 5472], 99.90th=[60672], 99.95th=[94720],
     | 99.99th=[114176]
    bw (KB  /s): min=92907, max=104448, per=5.00%, avg=98336.76, stdev=449.44
    lat (usec) : 4=0.01%, 500=60.68%, 750=38.26%, 1000=0.27%
    lat (msec) : 2=0.16%, 4=0.09%, 10=0.12%, 20=0.29%, 50=0.03%
    lat (msec) : 100=0.06%, 250=0.04%
  cpu          : usr=0.20%, sys=3.12%, ctx=65437, majf=0, minf=529
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20508: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170663msec
    slat (usec): min=55, max=6652, avg=136.47, stdev=50.21
    clat (usec): min=4, max=123670, avg=718.09, stdev=2836.31
     lat (usec): min=421, max=126627, avg=854.99, stdev=2848.10
    clat percentiles (usec):
     |  1.00th=[  410],  5.00th=[  438], 10.00th=[  450], 20.00th=[  470],
     | 30.00th=[  486], 40.00th=[  498], 50.00th=[  516], 60.00th=[  532],
     | 70.00th=[  564], 80.00th=[  628], 90.00th=[  748], 95.00th=[  948],
     | 99.00th=[ 1848], 99.50th=[ 9280], 99.90th=[48384], 99.95th=[78336],
     | 99.99th=[120320]
    bw (KB  /s): min=92444, max=104842, per=5.00%, avg=98365.28, stdev=491.15
    lat (usec) : 10=0.01%, 500=40.93%, 750=49.18%, 1000=5.48%
    lat (msec) : 2=3.58%, 4=0.27%, 10=0.10%, 20=0.34%, 50=0.04%
    lat (msec) : 100=0.07%, 250=0.02%
  cpu          : usr=0.16%, sys=3.19%, ctx=65493, majf=0, minf=528
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20509: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98305KB/s, iops=192, runt=170665msec
    slat (usec): min=55, max=2281, avg=133.94, stdev=30.91
    clat (usec): min=411, max=116031, avg=1895.84, stdev=2163.86
     lat (usec): min=480, max=116142, avg=2030.14, stdev=2167.46
    clat percentiles (usec):
     |  1.00th=[  502],  5.00th=[  628], 10.00th=[  716], 20.00th=[  852],
     | 30.00th=[ 1012], 40.00th=[ 1256], 50.00th=[ 1784], 60.00th=[ 2320],
     | 70.00th=[ 2544], 80.00th=[ 2704], 90.00th=[ 2864], 95.00th=[ 2960],
     | 99.00th=[ 3184], 99.50th=[ 9408], 99.90th=[28800], 99.95th=[53504],
     | 99.99th=[75264]
    bw (KB  /s): min=92724, max=104031, per=5.00%, avg=98374.35, stdev=446.22
    lat (usec) : 500=0.91%, 750=11.57%, 1000=16.87%
    lat (msec) : 2=23.30%, 4=46.78%, 10=0.08%, 20=0.32%, 50=0.12%
    lat (msec) : 100=0.05%, 250=0.01%
  cpu          : usr=0.18%, sys=3.29%, ctx=65445, majf=0, minf=529
  IO depths    : 1=99.5%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20510: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170663msec
    slat (usec): min=56, max=2537, avg=141.98, stdev=47.15
    clat (usec): min=391, max=90267, avg=1151.48, stdev=2342.47
     lat (usec): min=455, max=90498, avg=1293.87, stdev=2349.22
    clat percentiles (usec):
     |  1.00th=[  458],  5.00th=[  510], 10.00th=[  556], 20.00th=[  628],
     | 30.00th=[  700], 40.00th=[  772], 50.00th=[  860], 60.00th=[  996],
     | 70.00th=[ 1240], 80.00th=[ 1496], 90.00th=[ 1768], 95.00th=[ 1960],
     | 99.00th=[ 2352], 99.50th=[ 7200], 99.90th=[41216], 99.95th=[68096],
     | 99.99th=[89600]
    bw (KB  /s): min=92458, max=104842, per=5.00%, avg=98400.35, stdev=504.87
    lat (usec) : 500=4.06%, 750=33.35%, 1000=22.61%
    lat (msec) : 2=35.64%, 4=3.78%, 10=0.10%, 20=0.32%, 50=0.06%
    lat (msec) : 100=0.08%
  cpu          : usr=0.19%, sys=3.37%, ctx=65553, majf=0, minf=528
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20511: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170663msec
    slat (usec): min=56, max=2592, avg=141.51, stdev=35.77
    clat (usec): min=361, max=96017, avg=1148.00, stdev=2347.57
     lat (usec): min=422, max=96408, avg=1289.95, stdev=2353.05
    clat percentiles (usec):
     |  1.00th=[  458],  5.00th=[  510], 10.00th=[  556], 20.00th=[  628],
     | 30.00th=[  700], 40.00th=[  772], 50.00th=[  860], 60.00th=[  996],
     | 70.00th=[ 1224], 80.00th=[ 1480], 90.00th=[ 1768], 95.00th=[ 1960],
     | 99.00th=[ 2320], 99.50th=[ 7008], 99.90th=[38656], 99.95th=[67072],
     | 99.99th=[88576]
    bw (KB  /s): min=92458, max=104842, per=5.00%, avg=98393.28, stdev=503.07
    lat (usec) : 500=4.15%, 750=33.22%, 1000=22.66%
    lat (msec) : 2=35.71%, 4=3.70%, 10=0.11%, 20=0.33%, 50=0.05%
    lat (msec) : 100=0.08%
  cpu          : usr=0.17%, sys=3.39%, ctx=65551, majf=0, minf=530
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20512: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170664msec
    slat (usec): min=55, max=2799, avg=138.52, stdev=39.28
    clat (usec): min=377, max=92808, avg=1474.47, stdev=2446.34
     lat (usec): min=446, max=93045, avg=1613.36, stdev=2453.19
    clat percentiles (usec):
     |  1.00th=[  462],  5.00th=[  524], 10.00th=[  588], 20.00th=[  684],
     | 30.00th=[  764], 40.00th=[  876], 50.00th=[ 1208], 60.00th=[ 1704],
     | 70.00th=[ 1912], 80.00th=[ 2064], 90.00th=[ 2256], 95.00th=[ 2384],
     | 99.00th=[ 2704], 99.50th=[ 6624], 99.90th=[44288], 99.95th=[66048],
     | 99.99th=[92672]
    bw (KB  /s): min=92376, max=104842, per=5.00%, avg=98389.35, stdev=495.88
    lat (usec) : 500=3.12%, 750=25.19%, 1000=17.62%
    lat (msec) : 2=29.03%, 4=24.47%, 10=0.14%, 20=0.29%, 50=0.05%
    lat (msec) : 100=0.09%
  cpu          : usr=0.16%, sys=3.34%, ctx=65507, majf=0, minf=528
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20513: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170664msec
    slat (usec): min=55, max=2507, avg=138.55, stdev=40.23
    clat (usec): min=372, max=113199, avg=1492.31, stdev=2873.62
     lat (usec): min=442, max=113443, avg=1631.23, stdev=2880.23
    clat percentiles (usec):
     |  1.00th=[  462],  5.00th=[  524], 10.00th=[  588], 20.00th=[  684],
     | 30.00th=[  764], 40.00th=[  884], 50.00th=[ 1208], 60.00th=[ 1688],
     | 70.00th=[ 1912], 80.00th=[ 2096], 90.00th=[ 2256], 95.00th=[ 2384],
     | 99.00th=[ 2704], 99.50th=[10560], 99.90th=[48896], 99.95th=[77312],
     | 99.99th=[109056]
    bw (KB  /s): min=92911, max=104448, per=5.00%, avg=98390.47, stdev=461.12
    lat (usec) : 500=2.93%, 750=25.34%, 1000=17.59%
    lat (msec) : 2=28.96%, 4=24.60%, 10=0.07%, 20=0.37%, 50=0.04%
    lat (msec) : 100=0.06%, 250=0.03%
  cpu          : usr=0.22%, sys=3.28%, ctx=65488, majf=0, minf=530
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20514: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170664msec
    slat (usec): min=40, max=2109, avg=122.87, stdev=32.37
    clat (usec): min=373, max=140079, avg=1691.19, stdev=4049.15
     lat (usec): min=446, max=140288, avg=1814.43, stdev=4052.63
    clat percentiles (usec):
     |  1.00th=[  466],  5.00th=[  532], 10.00th=[  596], 20.00th=[  700],
     | 30.00th=[  796], 40.00th=[  940], 50.00th=[ 1512], 60.00th=[ 2024],
     | 70.00th=[ 2192], 80.00th=[ 2352], 90.00th=[ 2512], 95.00th=[ 2640],
     | 99.00th=[ 2896], 99.50th=[ 5984], 99.90th=[34048], 99.95th=[125440],
     | 99.99th=[136192]
    bw (KB  /s): min=92458, max=104842, per=5.00%, avg=98384.08, stdev=492.74
    lat (usec) : 500=2.88%, 750=22.97%, 1000=16.53%
    lat (msec) : 2=16.91%, 4=40.16%, 10=0.12%, 20=0.30%, 50=0.03%
    lat (msec) : 250=0.10%
  cpu          : usr=0.18%, sys=3.06%, ctx=65463, majf=0, minf=529
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20515: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170664msec
    slat (usec): min=55, max=2400, avg=135.31, stdev=33.47
    clat (usec): min=381, max=134107, avg=1708.29, stdev=3833.16
     lat (usec): min=460, max=134468, avg=1843.95, stdev=3839.53
    clat percentiles (usec):
     |  1.00th=[  478],  5.00th=[  540], 10.00th=[  604], 20.00th=[  708],
     | 30.00th=[  804], 40.00th=[  964], 50.00th=[ 1560], 60.00th=[ 2064],
     | 70.00th=[ 2224], 80.00th=[ 2384], 90.00th=[ 2544], 95.00th=[ 2640],
     | 99.00th=[ 2928], 99.50th=[ 6240], 99.90th=[51968], 99.95th=[116224],
     | 99.99th=[132096]
    bw (KB  /s): min=92160, max=104448, per=5.00%, avg=98384.25, stdev=484.82
    lat (usec) : 500=2.32%, 750=22.34%, 1000=16.93%
    lat (msec) : 2=16.25%, 4=41.59%, 10=0.13%, 20=0.30%, 50=0.04%
    lat (msec) : 100=0.02%, 250=0.08%
  cpu          : usr=0.19%, sys=3.28%, ctx=65462, majf=0, minf=530
  IO depths    : 1=99.6%, 2=0.3%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20516: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170664msec
    slat (usec): min=55, max=1406, avg=134.57, stdev=27.30
    clat (usec): min=401, max=116502, avg=1835.05, stdev=3329.01
     lat (usec): min=464, max=116669, avg=1969.96, stdev=3333.28
    clat percentiles (usec):
     |  1.00th=[  482],  5.00th=[  572], 10.00th=[  652], 20.00th=[  772],
     | 30.00th=[  908], 40.00th=[ 1112], 50.00th=[ 1720], 60.00th=[ 2256],
     | 70.00th=[ 2448], 80.00th=[ 2576], 90.00th=[ 2736], 95.00th=[ 2832],
     | 99.00th=[ 3056], 99.50th=[ 7008], 99.90th=[49920], 99.95th=[99840],
     | 99.99th=[116224]
    bw (KB  /s): min=92724, max=104239, per=5.00%, avg=98384.28, stdev=458.28
    lat (usec) : 500=1.57%, 750=16.25%, 1000=17.74%
    lat (msec) : 2=17.89%, 4=45.97%, 10=0.14%, 20=0.31%, 50=0.03%
    lat (msec) : 100=0.05%, 250=0.05%
  cpu          : usr=0.19%, sys=3.29%, ctx=65454, majf=0, minf=529
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20517: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170664msec
    slat (usec): min=55, max=6712, avg=135.04, stdev=53.86
    clat (usec): min=412, max=101019, avg=1804.05, stdev=2479.16
     lat (usec): min=473, max=101228, avg=1939.46, stdev=2495.72
    clat percentiles (usec):
     |  1.00th=[  482],  5.00th=[  580], 10.00th=[  660], 20.00th=[  772],
     | 30.00th=[  908], 40.00th=[ 1112], 50.00th=[ 1720], 60.00th=[ 2256],
     | 70.00th=[ 2448], 80.00th=[ 2576], 90.00th=[ 2736], 95.00th=[ 2864],
     | 99.00th=[ 3088], 99.50th=[ 8256], 99.90th=[23168], 99.95th=[49408],
     | 99.99th=[98816]
    bw (KB  /s): min=92542, max=104448, per=5.00%, avg=98378.01, stdev=472.71
    lat (usec) : 500=1.57%, 750=16.49%, 1000=17.29%
    lat (msec) : 2=18.18%, 4=45.90%, 10=0.11%, 20=0.34%, 50=0.07%
    lat (msec) : 100=0.04%, 250=0.01%
  cpu          : usr=0.18%, sys=3.29%, ctx=65464, majf=0, minf=529
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20518: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170663msec
    slat (usec): min=56, max=3221, avg=136.06, stdev=67.72
    clat (usec): min=360, max=108597, avg=1714.34, stdev=2240.23
     lat (usec): min=425, max=108693, avg=1850.80, stdev=2245.63
    clat percentiles (usec):
     |  1.00th=[  466],  5.00th=[  540], 10.00th=[  636], 20.00th=[  820],
     | 30.00th=[ 1032], 40.00th=[ 1288], 50.00th=[ 1592], 60.00th=[ 1896],
     | 70.00th=[ 2160], 80.00th=[ 2384], 90.00th=[ 2576], 95.00th=[ 2672],
     | 99.00th=[ 2864], 99.50th=[ 9920], 99.90th=[31616], 99.95th=[61184],
     | 99.99th=[81408]
    bw (KB  /s): min=92458, max=104842, per=5.00%, avg=98356.16, stdev=483.88
    lat (usec) : 500=2.49%, 750=13.84%, 1000=12.42%
    lat (msec) : 2=35.14%, 4=35.54%, 10=0.08%, 20=0.38%, 50=0.05%
    lat (msec) : 100=0.06%, 250=0.01%
  cpu          : usr=0.21%, sys=3.29%, ctx=65532, majf=0, minf=528
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
sequential-read: (groupid=0, jobs=1): err= 0: pid=20519: Sun Jul  2 18:43:23 2017
  read : io=16384MB, bw=98306KB/s, iops=192, runt=170663msec
    slat (usec): min=40, max=109766, avg=125.14, stdev=617.54
    clat (usec): min=369, max=51365, avg=1350.74, stdev=1690.32
     lat (usec): min=455, max=140421, avg=1476.30, stdev=1853.90
    clat percentiles (usec):
     |  1.00th=[  462],  5.00th=[  524], 10.00th=[  588], 20.00th=[  716],
     | 30.00th=[  876], 40.00th=[ 1048], 50.00th=[ 1240], 60.00th=[ 1432],
     | 70.00th=[ 1624], 80.00th=[ 1784], 90.00th=[ 1944], 95.00th=[ 2024],
     | 99.00th=[ 2192], 99.50th=[ 6176], 99.90th=[28800], 99.95th=[43264],
     | 99.99th=[50944]
    bw (KB  /s): min=92542, max=104634, per=5.00%, avg=98351.71, stdev=473.18
    lat (usec) : 500=3.31%, 750=18.93%, 1000=14.82%
    lat (msec) : 2=56.76%, 4=5.63%, 10=0.11%, 20=0.31%, 50=0.10%
    lat (msec) : 100=0.03%
  cpu          : usr=0.16%, sys=2.98%, ctx=65714, majf=0, minf=528
  IO depths    : 1=99.6%, 2=0.2%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=327680MB, aggrb=1920.2MB/s, minb=98304KB/s, maxb=98306KB/s, mint=170663msec, maxt=170665msec

sr.fio

[sequential-read]
ioengine=libaio
invalidate=1
rw=read
size=16G
direct=1
directory=/media/firefish/3a759c12-8025-4fbc-9ece-b03908079b43/test
numjobs=20
name=sequential-read
bs=512k
iodepth=32
rate_iops=192