Checking IOPS with fio

beta_test_rand_mix_rw

test_rand_mix_rw: (groupid=0, jobs=1): err= 0: pid=29890: Tue Jan  7 12:33:06 2020
  read: IOPS=57.5k, BW=449MiB/s (471MB/s)(2046MiB/4557msec)
    slat (nsec): min=1550, max=48849, avg=2505.48, stdev=1810.42
    clat (usec): min=70, max=10830, avg=496.39, stdev=482.92
     lat (usec): min=73, max=10832, avg=498.94, stdev=482.93
    clat percentiles (usec):
     |  1.00th=[   88],  5.00th=[  113], 10.00th=[  137], 20.00th=[  176],
     | 30.00th=[  215], 40.00th=[  258], 50.00th=[  318], 60.00th=[  400],
     | 70.00th=[  537], 80.00th=[  766], 90.00th=[ 1123], 95.00th=[ 1385],
     | 99.00th=[ 2278], 99.50th=[ 2868], 99.90th=[ 4080], 99.95th=[ 4555],
     | 99.99th=[ 6718]
   bw (  KiB/s): min=280208, max=541008, per=100.00%, avg=462135.11, stdev=82483.53, samples=9
   iops        : min=35026, max=67626, avg=57766.89, stdev=10310.44, samples=9
  write: IOPS=57.6k, BW=450MiB/s (472MB/s)(2050MiB/4557msec)
    slat (nsec): min=1826, max=41183, avg=3050.49, stdev=2028.72
    clat (usec): min=4, max=10684, avg=53.74, stdev=243.44
     lat (usec): min=13, max=10687, avg=56.83, stdev=243.46
    clat percentiles (usec):
     |  1.00th=[   13],  5.00th=[   14], 10.00th=[   14], 20.00th=[   15],
     | 30.00th=[   15], 40.00th=[   16], 50.00th=[   17], 60.00th=[   18],
     | 70.00th=[   20], 80.00th=[   23], 90.00th=[   32], 95.00th=[   62],
     | 99.00th=[ 1205], 99.50th=[ 1713], 99.90th=[ 2933], 99.95th=[ 3458],
     | 99.99th=[ 6325]
   bw (  KiB/s): min=284624, max=542896, per=100.00%, avg=462782.22, stdev=81537.25, samples=9
   iops        : min=35578, max=67862, avg=57847.78, stdev=10192.16, samples=9
  lat (usec)   : 10=0.01%, 20=37.20%, 50=9.97%, 100=2.21%, 250=18.27%
  lat (usec)   : 500=15.09%, 750=6.12%, 1000=3.95%
  lat (msec)   : 2=6.25%, 4=0.85%, 10=0.07%, 20=0.01%
  cpu          : usr=10.82%, sys=38.96%, ctx=207714, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=261946,262342,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=449MiB/s (471MB/s), 449MiB/s-449MiB/s (471MB/s-471MB/s), io=2046MiB (2146MB), run=4557-4557msec
  WRITE: bw=450MiB/s (472MB/s), 450MiB/s-450MiB/s (472MB/s-472MB/s), io=2050MiB (2149MB), run=4557-4557msec

Disk stats (read/write):
    md2: ios=256518/256975, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=130980/131196, aggrmerge=0/16, aggrticks=64682/7103, aggrin_queue=71661, aggrutil=97.35%
  nvme0n1: ios=130809/131386, merge=0/33, ticks=69098/12095, in_queue=81083, util=97.35%
  nvme1n1: ios=131151/131006, merge=0/0, ticks=60266/2112, in_queue=62240, util=95.70%

beta_test_seq_mix_rw

test_seq_mix_rw: (groupid=0, jobs=1): err= 0: pid=29868: Tue Jan  7 12:32:58 2020
  read: IOPS=52.3k, BW=409MiB/s (429MB/s)(2046MiB/5006msec)
    slat (nsec): min=1492, max=48221, avg=2558.94, stdev=1903.16
    clat (usec): min=11, max=16674, avg=499.36, stdev=680.19
     lat (usec): min=13, max=16676, avg=501.97, stdev=680.19
    clat percentiles (usec):
     |  1.00th=[   37],  5.00th=[   87], 10.00th=[  108], 20.00th=[  139],
     | 30.00th=[  169], 40.00th=[  208], 50.00th=[  265], 60.00th=[  347],
     | 70.00th=[  498], 80.00th=[  783], 90.00th=[ 1156], 95.00th=[ 1418],
     | 99.00th=[ 3326], 99.50th=[ 4817], 99.90th=[ 7635], 99.95th=[ 8586],
     | 99.99th=[11076]
   bw (  KiB/s): min=202192, max=654880, per=99.96%, avg=418460.80, stdev=119160.04, samples=10
   iops        : min=25274, max=81860, avg=52307.60, stdev=14895.00, samples=10
  write: IOPS=52.4k, BW=409MiB/s (429MB/s)(2050MiB/5006msec)
    slat (nsec): min=1776, max=65575, avg=3163.06, stdev=2178.53
    clat (nsec): min=478, max=12445k, avg=105517.71, stdev=455493.63
     lat (usec): min=14, max=12473, avg=108.73, stdev=455.56
    clat percentiles (usec):
     |  1.00th=[   14],  5.00th=[   15], 10.00th=[   17], 20.00th=[   23],
     | 30.00th=[   31], 40.00th=[   41], 50.00th=[   52], 60.00th=[   64],
     | 70.00th=[   80], 80.00th=[   99], 90.00th=[  147], 95.00th=[  219],
     | 99.00th=[  420], 99.50th=[ 4293], 99.90th=[ 6783], 99.95th=[ 7832],
     | 99.99th=[10159]
   bw (  KiB/s): min=198656, max=662272, per=99.97%, avg=419113.60, stdev=121257.72, samples=10
   iops        : min=24832, max=82782, avg=52389.20, stdev=15156.68, samples=10
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=8.39%, 50=16.96%, 100=18.94%
  lat (usec)   : 250=27.89%, 500=12.42%, 750=4.45%, 1000=3.62%
  lat (msec)   : 2=5.92%, 4=0.78%, 10=0.59%, 20=0.02%
  cpu          : usr=9.69%, sys=32.13%, ctx=86017, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=261946,262342,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=409MiB/s (429MB/s), 409MiB/s-409MiB/s (429MB/s-429MB/s), io=2046MiB (2146MB), run=5006-5006msec
  WRITE: bw=409MiB/s (429MB/s), 409MiB/s-409MiB/s (429MB/s-429MB/s), io=2050MiB (2149MB), run=5006-5006msec

Disk stats (read/write):
    md2: ios=253074/253423, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=130982/131288, aggrmerge=0/8, aggrticks=62314/11824, aggrin_queue=68210, aggrutil=71.61%
  nvme0n1: ios=131012/131317, merge=0/17, ticks=59965/12356, in_queue=65758, util=68.76%
  nvme1n1: ios=130952/131260, merge=0/0, ticks=64663/11292, in_queue=70662, util=71.61%

eks_test_rand_mix_rw

test_rand_mix_rw: (groupid=0, jobs=1): err= 0: pid=6082: Tue Jan  7 11:58:21 2020
  read: IOPS=36.3k, BW=284MiB/s (297MB/s)(2046MiB/7214msec)
    slat (nsec): min=2102, max=80192, avg=4381.26, stdev=4289.24
    clat (usec): min=70, max=4805, avg=762.87, stdev=822.93
     lat (usec): min=84, max=4808, avg=767.36, stdev=822.78
    clat percentiles (usec):
     |  1.00th=[   90],  5.00th=[  108], 10.00th=[  130], 20.00th=[  196],
     | 30.00th=[  249], 40.00th=[  273], 50.00th=[  310], 60.00th=[  461],
     | 70.00th=[  709], 80.00th=[ 1713], 90.00th=[ 2278], 95.00th=[ 2474],
     | 99.00th=[ 2737], 99.50th=[ 2835], 99.90th=[ 3064], 99.95th=[ 3163],
     | 99.99th=[ 4015]
   bw (  KiB/s): min=281248, max=304272, per=99.97%, avg=290411.43, stdev=6666.86, samples=14
   iops        : min=35156, max=38034, avg=36301.43, stdev=833.36, samples=14
  write: IOPS=36.4k, BW=284MiB/s (298MB/s)(2050MiB/7214msec); 0 zone resets
    slat (nsec): min=2432, max=77769, avg=5255.77, stdev=4795.95
    clat (usec): min=4, max=451, avg=107.01, stdev=64.51
     lat (usec): min=25, max=455, avg=112.37, stdev=65.32
    clat percentiles (usec):
     |  1.00th=[   26],  5.00th=[   29], 10.00th=[   32], 20.00th=[   40],
     | 30.00th=[   51], 40.00th=[   72], 50.00th=[   96], 60.00th=[  123],
     | 70.00th=[  155], 80.00th=[  182], 90.00th=[  200], 95.00th=[  212],
     | 99.00th=[  229], 99.50th=[  235], 99.90th=[  251], 99.95th=[  269],
     | 99.99th=[  322]
   bw (  KiB/s): min=277184, max=299312, per=99.93%, avg=290734.86, stdev=5994.06, samples=14
   iops        : min=34648, max=37414, avg=36341.86, stdev=749.26, samples=14
  lat (usec)   : 10=0.01%, 20=0.01%, 50=14.69%, 100=12.56%, 250=38.04%
  lat (usec)   : 500=15.70%, 750=4.65%, 1000=2.14%
  lat (msec)   : 2=3.65%, 4=8.55%, 10=0.01%
  cpu          : usr=18.51%, sys=35.64%, ctx=47517, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=261946,262342,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=284MiB/s (297MB/s), 284MiB/s-284MiB/s (297MB/s-297MB/s), io=2046MiB (2146MB), run=7214-7214msec
  WRITE: bw=284MiB/s (298MB/s), 284MiB/s-284MiB/s (298MB/s-298MB/s), io=2050MiB (2149MB), run=7214-7214msec

Disk stats (read/write):
    md0: ios=259491/260369, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=130973/131431, aggrmerge=0/10, aggrticks=88292/3524, aggrin_queue=87120, aggrutil=87.09%
  nvme2n1: ios=131149/131246, merge=0/12, ticks=96020/3452, in_queue=94768, util=87.09%
  nvme1n1: ios=130797/131616, merge=0/9, ticks=80564/3596, in_queue=79472, util=83.13%

eks_test_seq_mix_rw

test_seq_mix_rw: (groupid=0, jobs=1): err= 0: pid=6063: Tue Jan  7 11:58:02 2020
  read: IOPS=39.7k, BW=310MiB/s (325MB/s)(2046MiB/6594msec)
    slat (usec): min=2, max=103, avg= 4.69, stdev= 4.39
    clat (usec): min=21, max=4881, avg=627.42, stdev=750.17
     lat (usec): min=28, max=4884, avg=632.23, stdev=749.80
    clat percentiles (usec):
     |  1.00th=[   83],  5.00th=[  169], 10.00th=[  204], 20.00th=[  227],
     | 30.00th=[  239], 40.00th=[  247], 50.00th=[  260], 60.00th=[  293],
     | 70.00th=[  404], 80.00th=[  807], 90.00th=[ 2147], 95.00th=[ 2507],
     | 99.00th=[ 2835], 99.50th=[ 2933], 99.90th=[ 3130], 99.95th=[ 3195],
     | 99.99th=[ 3458]
   bw (  KiB/s): min=251504, max=494032, per=99.28%, avg=315504.00, stdev=80847.18, samples=13
   iops        : min=31438, max=61754, avg=39438.00, stdev=10105.90, samples=13
  write: IOPS=39.8k, BW=311MiB/s (326MB/s)(2050MiB/6594msec); 0 zone resets
    slat (nsec): min=2417, max=96305, avg=5623.66, stdev=4934.48
    clat (usec): min=15, max=891, avg=166.05, stdev=73.75
     lat (usec): min=26, max=899, avg=171.79, stdev=74.60
    clat percentiles (usec):
     |  1.00th=[   29],  5.00th=[   35], 10.00th=[   44], 20.00th=[   83],
     | 30.00th=[  127], 40.00th=[  163], 50.00th=[  190], 60.00th=[  208],
     | 70.00th=[  223], 80.00th=[  235], 90.00th=[  245], 95.00th=[  251],
     | 99.00th=[  265], 99.50th=[  269], 99.90th=[  318], 99.95th=[  334],
     | 99.99th=[  742]
   bw (  KiB/s): min=250560, max=494096, per=99.28%, avg=315993.85, stdev=81506.60, samples=13
   iops        : min=31320, max=61762, avg=39499.23, stdev=10188.33, samples=13
  lat (usec)   : 20=0.01%, 50=6.31%, 100=6.39%, 250=55.85%, 500=18.16%
  lat (usec)   : 750=2.81%, 1000=1.61%
  lat (msec)   : 2=3.08%, 4=5.78%, 10=0.01%
  cpu          : usr=19.66%, sys=39.86%, ctx=20572, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=261946,262342,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=310MiB/s (325MB/s), 310MiB/s-310MiB/s (325MB/s-325MB/s), io=2046MiB (2146MB), run=6594-6594msec
  WRITE: bw=311MiB/s (326MB/s), 311MiB/s-311MiB/s (326MB/s-326MB/s), io=2050MiB (2149MB), run=6594-6594msec

Disk stats (read/write):
    md0: ios=250204/250866, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=130979/131411, aggrmerge=0/8, aggrticks=61208/4112, aggrin_queue=61360, aggrutil=64.22%
  nvme2n1: ios=130949/131390, merge=0/0, ticks=73392/4148, in_queue=73356, util=64.22%
  nvme1n1: ios=131009/131432, merge=0/16, ticks=49024/4076, in_queue=49364, util=55.02%

How to reproduce https://bit.ly/2QwHTU0