3par-linstor-benchmark

3par

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=1 -randrepeat=0 -rw=randwrite -filename=/dev/mapper/mpathc -runtime=60
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [w(1)][0.6%][w=176KiB/s][w=44 IOPS][eta 02h:57m:30s]
test: (groupid=0, jobs=1): err= 0: pid=4129882: Thu Nov 12 16:02:14 2020
  write: IOPS=2448, BW=9796KiB/s (10.0MB/s)(583MiB/60958msec); 0 zone resets
    slat (usec): min=4, max=685, avg=36.64, stdev=19.01
    clat (usec): min=198, max=1012.8k, avg=366.08, stdev=3744.08
     lat (usec): min=208, max=1012.8k, avg=403.32, stdev=3744.27
    clat percentiles (usec):
     |  1.00th=[  235],  5.00th=[  255], 10.00th=[  269], 20.00th=[  285],
     | 30.00th=[  297], 40.00th=[  314], 50.00th=[  326], 60.00th=[  343],
     | 70.00th=[  363], 80.00th=[  388], 90.00th=[  433], 95.00th=[  478],
     | 99.00th=[  840], 99.50th=[  914], 99.90th=[ 1303], 99.95th=[ 1680],
     | 99.99th=[ 8455]
   bw (  KiB/s): min=  352, max=12200, per=100.00%, avg=10034.05, stdev=1313.22, samples=119
   iops        : min=   88, max= 3050, avg=2508.48, stdev=328.29, samples=119
  lat (usec)   : 250=3.55%, 500=92.42%, 750=2.02%, 1000=1.67%
  lat (msec)   : 2=0.29%, 4=0.01%, 10=0.02%, 20=0.01%, 50=0.01%
  lat (msec)   : 250=0.01%, 2000=0.01%
  cpu          : usr=4.00%, sys=12.70%, ctx=149381, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,149284,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=9796KiB/s (10.0MB/s), 9796KiB/s-9796KiB/s (10.0MB/s-10.0MB/s), io=583MiB (611MB), run=60958-60958msec

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=32 -randrepeat=0 -rw=randwrite -filename=/dev/mapper/mpathc -runtime=60
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=221MiB/s][w=56.5k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=4133653: Thu Nov 12 16:05:39 2020
  write: IOPS=53.3k, BW=208MiB/s (218MB/s)(12.2GiB/60001msec); 0 zone resets
    slat (usec): min=2, max=1484, avg= 6.24, stdev= 4.16
    clat (usec): min=151, max=15114k, avg=593.22, stdev=25200.81
     lat (usec): min=193, max=15114k, avg=599.57, stdev=25200.81
    clat percentiles (usec):
     |  1.00th=[  310],  5.00th=[  355], 10.00th=[  383], 20.00th=[  420],
     | 30.00th=[  453], 40.00th=[  478], 50.00th=[  506], 60.00th=[  537],
     | 70.00th=[  570], 80.00th=[  611], 90.00th=[  701], 95.00th=[  898],
     | 99.00th=[ 1319], 99.50th=[ 1614], 99.90th=[ 3392], 99.95th=[ 4113],
     | 99.99th=[ 5932]
   bw (  KiB/s): min=169560, max=231376, per=99.93%, avg=213103.70, stdev=16034.65, samples=119
   iops        : min=42390, max=57844, avg=53275.92, stdev=4008.64, samples=119
  lat (usec)   : 250=0.04%, 500=47.80%, 750=44.09%, 1000=4.84%
  lat (msec)   : 2=2.92%, 4=0.25%, 10=0.05%, 20=0.01%, 50=0.01%
  lat (msec)   : 250=0.01%, 2000=0.01%, >=2000=0.01%
  cpu          : usr=10.56%, sys=36.16%, ctx=1106473, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3198818,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=208MiB/s (218MB/s), 208MiB/s-208MiB/s (218MB/s-218MB/s), io=12.2GiB (13.1GB), run=60001-60001msec

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=512 -randrepeat=0 -rw=randwrite -filename=/dev/mapper/mpathc -runtime=60
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=512
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=323MiB/s][w=82.6k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=4135506: Thu Nov 12 16:06:54 2020
  write: IOPS=74.0k, BW=293MiB/s (307MB/s)(17.2GiB/60008msec); 0 zone resets
    slat (usec): min=2, max=42663, avg=11.73, stdev=78.84
    clat (msec): min=2, max=1320, avg= 6.82, stdev= 5.22
     lat (msec): min=2, max=1320, avg= 6.83, stdev= 5.22
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    5], 20.00th=[    5],
     | 30.00th=[    6], 40.00th=[    6], 50.00th=[    6], 60.00th=[    7],
     | 70.00th=[    7], 80.00th=[    8], 90.00th=[   11], 95.00th=[   14],
     | 99.00th=[   23], 99.50th=[   30], 99.90th=[   62], 99.95th=[   80],
     | 99.99th=[  169]
   bw (  KiB/s): min=207632, max=367392, per=99.99%, avg=299875.37, stdev=22630.80, samples=120
   iops        : min=51908, max=91848, avg=74968.83, stdev=5657.72, samples=120
  lat (msec)   : 4=3.22%, 10=86.66%, 20=8.86%, 50=1.12%, 100=0.13%
  lat (msec)   : 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
  cpu          : usr=13.01%, sys=46.06%, ctx=168089, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,4499082,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=512

Run status group 0 (all jobs):
  WRITE: bw=293MiB/s (307MB/s), 293MiB/s-293MiB/s (307MB/s-307MB/s), io=17.2GiB (18.4GB), run=60008-60008msec

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=1024 -randrepeat=0 -rw=randwrite -filename=/dev/mapper/mpathc -runtime=60
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1024
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=350MiB/s][w=89.7k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=4138687: Thu Nov 12 16:09:24 2020
  write: IOPS=81.0k, BW=317MiB/s (332MB/s)(18.6GiB/60009msec); 0 zone resets
    slat (usec): min=2, max=67441, avg=10.70, stdev=50.70
    clat (msec): min=6, max=150, avg=12.62, stdev= 3.28
     lat (msec): min=6, max=150, avg=12.63, stdev= 3.28
    clat percentiles (msec):
     |  1.00th=[    9],  5.00th=[   10], 10.00th=[   10], 20.00th=[   11],
     | 30.00th=[   11], 40.00th=[   12], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   14], 80.00th=[   15], 90.00th=[   17], 95.00th=[   19],
     | 99.00th=[   23], 99.50th=[   24], 99.90th=[   35], 99.95th=[   48],
     | 99.99th=[  101]
   bw (  KiB/s): min=283584, max=424040, per=99.99%, avg=324141.71, stdev=19925.69, samples=120
   iops        : min=70896, max=106010, avg=81035.45, stdev=4981.40, samples=120
  lat (msec)   : 10=12.92%, 20=84.54%, 50=2.48%, 100=0.04%, 250=0.01%
  cpu          : usr=15.41%, sys=48.86%, ctx=168415, majf=0, minf=27
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,4863559,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1024

Run status group 0 (all jobs):
  WRITE: bw=317MiB/s (332MB/s), 317MiB/s-317MiB/s (332MB/s-332MB/s), io=18.6GiB (19.9GB), run=60009-60009msec

# -----------------------------------------------------

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=1 -randrepeat=0 -rw=randread -filename=/dev/mapper/mpathc -runtime=60
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=7508KiB/s][r=1877 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=4136246: Thu Nov 12 16:08:19 2020
  read: IOPS=1932, BW=7731KiB/s (7917kB/s)(453MiB/60001msec)
    slat (usec): min=4, max=676, avg=43.91, stdev=18.76
    clat (usec): min=146, max=91088, avg=466.63, stdev=434.25
     lat (usec): min=154, max=91114, avg=511.45, stdev=436.18
    clat percentiles (usec):
     |  1.00th=[  223],  5.00th=[  289], 10.00th=[  322], 20.00th=[  367],
     | 30.00th=[  400], 40.00th=[  429], 50.00th=[  453], 60.00th=[  478],
     | 70.00th=[  502], 80.00th=[  529], 90.00th=[  570], 95.00th=[  627],
     | 99.00th=[  988], 99.50th=[ 1090], 99.90th=[ 2769], 99.95th=[ 4146],
     | 99.99th=[ 9110]
   bw (  KiB/s): min= 4760, max=11520, per=100.00%, avg=7733.05, stdev=771.16, samples=119
   iops        : min= 1190, max= 2880, avg=1933.25, stdev=192.79, samples=119
  lat (usec)   : 250=2.14%, 500=66.65%, 750=27.77%, 1000=2.52%
  lat (msec)   : 2=0.78%, 4=0.09%, 10=0.04%, 20=0.01%, 100=0.01%
  cpu          : usr=3.77%, sys=13.76%, ctx=116015, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=115967,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=7731KiB/s (7917kB/s), 7731KiB/s-7731KiB/s (7917kB/s-7917kB/s), io=453MiB (475MB), run=60001-60001msec

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=32 -randrepeat=0 -rw=randread -filename=/dev/mapper/mpathc -runtime=60
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [r(1)][11.7%][eta 09m:33s]                           
test: (groupid=0, jobs=1): err= 0: pid=4140737: Thu Nov 12 16:11:26 2020
  read: IOPS=39.0k, BW=156MiB/s (164MB/s)(11.7GiB/76805msec)
    slat (usec): min=2, max=630, avg= 5.63, stdev= 4.60
    clat (usec): min=142, max=24974k, avg=789.10, stdev=67739.30
     lat (usec): min=149, max=24974k, avg=794.84, stdev=67739.31
    clat percentiles (usec):
     |  1.00th=[  253],  5.00th=[  314], 10.00th=[  343], 20.00th=[  379],
     | 30.00th=[  404], 40.00th=[  433], 50.00th=[  457], 60.00th=[  486],
     | 70.00th=[  519], 80.00th=[  570], 90.00th=[  676], 95.00th=[  799],
     | 99.00th=[ 1123], 99.50th=[ 1352], 99.90th=[ 2442], 99.95th=[ 3326],
     | 99.99th=[ 9634]
   bw (  KiB/s): min=134906, max=241992, per=100.00%, avg=204531.83, stdev=23423.10, samples=120
   iops        : min=33726, max=60498, avg=51132.94, stdev=5855.76, samples=120
  lat (usec)   : 250=0.89%, 500=64.23%, 750=28.60%, 1000=4.43%
  lat (msec)   : 2=1.70%, 4=0.11%, 10=0.03%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 1000=0.01%, 2000=0.01%, >=2000=0.01%
  cpu          : usr=7.50%, sys=25.97%, ctx=1329563, majf=0, minf=64
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=3068547,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=156MiB/s (164MB/s), 156MiB/s-156MiB/s (164MB/s-164MB/s), io=11.7GiB (12.6GB), run=76805-76805msec

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=512 -randrepeat=0 -rw=randread -filename=/dev/mapper/mpathc -runtime=60
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=512
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [r(1)][31.2%][eta 02m:21s]                           
test: (groupid=0, jobs=1): err= 0: pid=4141950: Thu Nov 12 16:13:25 2020
  read: IOPS=127k, BW=496MiB/s (520MB/s)(31.2GiB/64365msec)
    slat (usec): min=2, max=21922, avg= 5.86, stdev=20.07
    clat (usec): min=1804, max=10694k, avg=4027.47, stdev=63598.57
     lat (usec): min=1849, max=10694k, avg=4033.43, stdev=63598.59
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    5],
     | 99.00th=[    6], 99.50th=[    7], 99.90th=[    8], 99.95th=[   12],
     | 99.99th=[ 4396]
   bw (  KiB/s): min=331312, max=591160, per=100.00%, avg=544286.86, stdev=45323.01, samples=120
   iops        : min=82828, max=147790, avg=136071.70, stdev=11330.75, samples=120
  lat (msec)   : 2=0.01%, 4=90.80%, 10=9.14%, 20=0.03%, 50=0.02%
  lat (msec)   : 100=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%, >=2000=0.01%
  cpu          : usr=21.41%, sys=63.04%, ctx=46086, majf=0, minf=544
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=8167022,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=512

Run status group 0 (all jobs):
   READ: bw=496MiB/s (520MB/s), 496MiB/s-496MiB/s (520MB/s-520MB/s), io=31.2GiB (33.5GB), run=64365-64365msec

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=1024 -randrepeat=0 -rw=randread -filename=/dev/mapper/mpathc -runtime=60
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1024
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [r(1)][30.5%][eta 02m:28s]                           
test: (groupid=0, jobs=1): err= 0: pid=4153353: Thu Nov 12 16:14:44 2020
  read: IOPS=122k, BW=478MiB/s (502MB/s)(30.4GiB/65167msec)
    slat (usec): min=2, max=21191, avg= 5.97, stdev=19.81
    clat (msec): min=4, max=12550, avg= 8.35, stdev=81.79
     lat (msec): min=4, max=12550, avg= 8.36, stdev=81.79
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    6], 10.00th=[    7], 20.00th=[    7],
     | 30.00th=[    7], 40.00th=[    7], 50.00th=[    7], 60.00th=[    8],
     | 70.00th=[    8], 80.00th=[    8], 90.00th=[    9], 95.00th=[   10],
     | 99.00th=[   13], 99.50th=[   13], 99.90th=[   17], 99.95th=[   27],
     | 99.99th=[ 5201]
   bw (  KiB/s): min=330696, max=584616, per=100.00%, avg=531799.15, stdev=46840.89, samples=120
   iops        : min=82674, max=146154, avg=132949.79, stdev=11710.18, samples=120
  lat (msec)   : 10=96.24%, 20=3.68%, 50=0.05%, 250=0.01%, 500=0.01%
  lat (msec)   : 750=0.01%, 1000=0.01%, 2000=0.01%, >=2000=0.02%
  cpu          : usr=20.68%, sys=62.92%, ctx=44484, majf=0, minf=1056
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=7979219,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1024

Run status group 0 (all jobs):
   READ: bw=478MiB/s (502MB/s), 478MiB/s-478MiB/s (502MB/s-502MB/s), io=30.4GiB (32.7GB), run=65167-65167msec

drbd-diskless

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=1 -randrepeat=0 -rw=randwrite -filename=/dev/drbd3728 -runtime=60
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [w(1)][10.0%][w=8860KiB/s][w=2215 IOPS][eta 0Jobs: 1 (f=1): [w(1)][11.7%][w=1JobsJobs: 1 (f=1): [w(1)][100.0%][w=15.5MiB/s][w=3973 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2044833: Thu Nov 12 21:04:47 2020
  write: IOPS=3068, BW=11.0MiB/s (12.6MB/s)(719MiB/60001msec); 0 zone resets
    slat (usec): min=2, max=655, avg=17.91, stdev=14.50
    clat (usec): min=76, max=41088, avg=304.12, stdev=354.13
     lat (usec): min=81, max=41197, avg=322.50, stdev=356.59
    clat percentiles (usec):
     |  1.00th=[  125],  5.00th=[  163], 10.00th=[  180], 20.00th=[  200],
     | 30.00th=[  217], 40.00th=[  235], 50.00th=[  255], 60.00th=[  281],
     | 70.00th=[  314], 80.00th=[  359], 90.00th=[  453], 95.00th=[  553],
     | 99.00th=[  889], 99.50th=[ 1205], 99.90th=[ 4113], 99.95th=[ 6128],
     | 99.99th=[13960]
   bw (  KiB/s): min= 5984, max=19016, per=99.66%, avg=12231.23, stdev=2685.43, samples=119
   iops        : min= 1496, max= 4754, avg=3057.80, stdev=671.37, samples=119
  lat (usec)   : 100=0.17%, 250=47.38%, 500=45.13%, 750=5.58%, 1000=1.00%
  lat (msec)   : 2=0.51%, 4=0.12%, 10=0.09%, 20=0.02%, 50=0.01%
  cpu          : usr=3.89%, sys=9.37%, ctx=184170, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,184104,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=11.0MiB/s (12.6MB/s), 11.0MiB/s-11.0MiB/s (12.6MB/s-12.6MB/s), io=719MiB (754MB), run=60001-60001msec

Disk stats (read/write):
  drbd3728: ios=0/183674, merge=0/0, ticks=0/54576, in_queue=54576, util=98.60%

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=32 -randrepeat=0 -rw=randwrite -filename=/dev/drbd3728 -runtime=60
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=155MiB/s][w=39.6k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2050733: Thu Nov 12 21:06:39 2020
  write: IOPS=42.8k, BW=167MiB/s (175MB/s)(9.79GiB/60002msec); 0 zone resets
    slat (nsec): min=1818, max=450212, avg=3951.88, stdev=3564.79
    clat (usec): min=87, max=34604, avg=743.61, stdev=409.81
     lat (usec): min=90, max=34607, avg=747.63, stdev=409.87
    clat percentiles (usec):
     |  1.00th=[  212],  5.00th=[  474], 10.00th=[  545], 20.00th=[  603],
     | 30.00th=[  635], 40.00th=[  660], 50.00th=[  693], 60.00th=[  725],
     | 70.00th=[  766], 80.00th=[  832], 90.00th=[  963], 95.00th=[ 1123],
     | 99.00th=[ 1811], 99.50th=[ 2409], 99.90th=[ 5276], 99.95th=[ 6783],
     | 99.99th=[13042]
   bw (  KiB/s): min=85416, max=190376, per=100.00%, avg=171067.08, stdev=19169.74, samples=120
   iops        : min=21354, max=47594, avg=42766.77, stdev=4792.44, samples=120
  lat (usec)   : 100=0.01%, 250=1.50%, 500=4.60%, 750=60.29%, 1000=25.31%
  lat (msec)   : 2=7.53%, 4=0.59%, 10=0.15%, 20=0.02%, 50=0.01%
  cpu          : usr=8.34%, sys=20.92%, ctx=617803, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2566126,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=9.79GiB (10.5GB), run=60002-60002msec

Disk stats (read/write):
  drbd3728: ios=0/2561466, merge=0/0, ticks=0/1877916, in_queue=1877916, util=99.30%

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=512 -randrepeat=0 -rw=randwrite -filename=/dev/drbd3728 -runtime=60
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=512
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=264MiB/s][w=67.5k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2051621: Thu Nov 12 21:08:16 2020
  write: IOPS=63.2k, BW=247MiB/s (259MB/s)(14.5GiB/60009msec); 0 zone resets
    slat (nsec): min=1943, max=762965, avg=5426.41, stdev=9413.91
    clat (usec): min=1276, max=45337, avg=8090.49, stdev=1591.63
     lat (usec): min=1280, max=45341, avg=8096.00, stdev=1591.63
    clat percentiles (usec):
     |  1.00th=[ 3523],  5.00th=[ 6652], 10.00th=[ 7046], 20.00th=[ 7308],
     | 30.00th=[ 7570], 40.00th=[ 7767], 50.00th=[ 7963], 60.00th=[ 8160],
     | 70.00th=[ 8455], 80.00th=[ 8717], 90.00th=[ 9372], 95.00th=[10159],
     | 99.00th=[12518], 99.50th=[14353], 99.90th=[25297], 99.95th=[32375],
     | 99.99th=[39060]
   bw (  KiB/s): min=195776, max=273960, per=99.99%, avg=252914.79, stdev=12855.83, samples=120
   iops        : min=48944, max=68490, avg=63228.70, stdev=3213.95, samples=120
  lat (msec)   : 2=0.07%, 4=1.29%, 10=93.17%, 20=5.32%, 50=0.15%
  cpu          : usr=11.86%, sys=36.79%, ctx=455317, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,3794552,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=512

Run status group 0 (all jobs):
  WRITE: bw=247MiB/s (259MB/s), 247MiB/s-247MiB/s (259MB/s-259MB/s), io=14.5GiB (15.5GB), run=60009-60009msec

Disk stats (read/write):
  drbd3728: ios=0/3787378, merge=0/0, ticks=0/30461424, in_queue=30461424, util=99.59%


# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=1024 -randrepeat=0 -rw=randwrite -filename=/dev/drbd3728 -runtime=60
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1024
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=232MiB/s][w=59.5k IOPS][eta 00m:00s]:51s]
test: (groupid=0, jobs=1): err= 0: pid=2052891: Thu Nov 12 21:10:19 2020
  write: IOPS=65.1k, BW=254MiB/s (267MB/s)(14.9GiB/60014msec); 0 zone resets
    slat (nsec): min=1826, max=1577.5k, avg=4911.75, stdev=11318.83
    clat (usec): min=2991, max=79373, avg=15722.37, stdev=3237.22
     lat (usec): min=2996, max=79378, avg=15727.36, stdev=3237.44
    clat percentiles (usec):
     |  1.00th=[ 8160],  5.00th=[13042], 10.00th=[13566], 20.00th=[14222],
     | 30.00th=[14615], 40.00th=[14877], 50.00th=[15270], 60.00th=[15795],
     | 70.00th=[16319], 80.00th=[16909], 90.00th=[17957], 95.00th=[19792],
     | 99.00th=[25560], 99.50th=[30540], 99.90th=[55837], 99.95th=[63177],
     | 99.99th=[72877]
   bw (  KiB/s): min=169968, max=286984, per=99.99%, avg=260368.07, stdev=17813.04, samples=120
   iops        : min=42492, max=71746, avg=65092.03, stdev=4453.27, samples=120
  lat (msec)   : 4=0.01%, 10=1.81%, 20=93.75%, 50=4.31%, 100=0.12%
  cpu          : usr=11.37%, sys=35.35%, ctx=592279, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,3906999,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1024

Run status group 0 (all jobs):
  WRITE: bw=254MiB/s (267MB/s), 254MiB/s-254MiB/s (267MB/s-267MB/s), io=14.9GiB (16.0GB), run=60014-60014msec

Disk stats (read/write):
  drbd3728: ios=0/3899430, merge=0/0, ticks=0/61066004, in_queue=61066004, util=99.45%

# -----------------------------------------------------

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=1 -randrepeat=0 -rw=randread -filename=/dev/drbd3728 -runtime=60
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=10.6MiB/s][r=2709 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2056712: Thu Nov 12 21:11:30 2020
  read: IOPS=2862, BW=11.2MiB/s (11.7MB/s)(671MiB/60001msec)
    slat (usec): min=2, max=685, avg=26.15, stdev=14.02
    clat (usec): min=6, max=44054, avg=317.75, stdev=294.25
     lat (usec): min=111, max=44069, avg=344.55, stdev=295.96
    clat percentiles (usec):
     |  1.00th=[  157],  5.00th=[  190], 10.00th=[  206], 20.00th=[  229],
     | 30.00th=[  249], 40.00th=[  265], 50.00th=[  285], 60.00th=[  310],
     | 70.00th=[  338], 80.00th=[  375], 90.00th=[  433], 95.00th=[  486],
     | 99.00th=[  717], 99.50th=[ 1004], 99.90th=[ 3556], 99.95th=[ 5407],
     | 99.99th=[11863]
   bw (  KiB/s): min= 8632, max=14376, per=100.00%, avg=11457.95, stdev=1247.64, samples=119
   iops        : min= 2158, max= 3594, avg=2864.49, stdev=311.91, samples=119
  lat (usec)   : 10=0.01%, 250=31.20%, 500=64.68%, 750=3.22%, 1000=0.41%
  lat (msec)   : 2=0.26%, 4=0.16%, 10=0.07%, 20=0.02%, 50=0.01%
  cpu          : usr=5.23%, sys=13.68%, ctx=171799, majf=0, minf=35
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=171756,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=11.2MiB/s (11.7MB/s), 11.2MiB/s-11.2MiB/s (11.7MB/s-11.7MB/s), io=671MiB (704MB), run=60001-60001msec

Disk stats (read/write):
  drbd3728: ios=171408/0, merge=0/0, ticks=52012/0, in_queue=52012, util=98.98%

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=32 -randrepeat=0 -rw=randread -filename=/dev/drbd3728 -runtime=60
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=506MiB/s][r=130k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2057514: Thu Nov 12 21:12:40 2020
  read: IOPS=120k, BW=469MiB/s (492MB/s)(27.5GiB/60001msec)
    slat (nsec): min=1533, max=574803, avg=3509.49, stdev=3459.15
    clat (usec): min=89, max=63133, avg=262.24, stdev=424.19
     lat (usec): min=104, max=63140, avg=265.81, stdev=424.36
    clat percentiles (usec):
     |  1.00th=[  143],  5.00th=[  161], 10.00th=[  172], 20.00th=[  188],
     | 30.00th=[  202], 40.00th=[  217], 50.00th=[  231], 60.00th=[  247],
     | 70.00th=[  269], 80.00th=[  302], 90.00th=[  367], 95.00th=[  433],
     | 99.00th=[  578], 99.50th=[  685], 99.90th=[ 2040], 99.95th=[ 4817],
     | 99.99th=[21103]
   bw (  KiB/s): min=116264, max=535488, per=99.93%, avg=480342.61, stdev=70702.78, samples=119
   iops        : min=29066, max=133872, avg=120085.65, stdev=17675.60, samples=119
  lat (usec)   : 100=0.01%, 250=61.17%, 500=36.74%, 750=1.71%, 1000=0.17%
  lat (msec)   : 2=0.12%, 4=0.04%, 10=0.03%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=17.96%, sys=47.72%, ctx=1562799, majf=0, minf=66
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=7210169,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=469MiB/s (492MB/s), 469MiB/s-469MiB/s (492MB/s-492MB/s), io=27.5GiB (29.5GB), run=60001-60001msec

Disk stats (read/write):
  drbd3728: ios=7196444/0, merge=0/0, ticks=1831036/0, in_queue=1831036, util=98.61%

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=512 -randrepeat=0 -rw=randread -filename=/dev/drbd3728 -runtime=60
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=496MiB/s][r=127k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2058114: Thu Nov 12 21:13:46 2020
  read: IOPS=122k, BW=476MiB/s (499MB/s)(27.9GiB/60001msec)
    slat (nsec): min=1561, max=1492.8k, avg=3529.69, stdev=3356.56
    clat (usec): min=85, max=57238, avg=258.51, stdev=241.97
     lat (usec): min=104, max=57248, avg=262.12, stdev=242.03
    clat percentiles (usec):
     |  1.00th=[  143],  5.00th=[  161], 10.00th=[  174], 20.00th=[  190],
     | 30.00th=[  204], 40.00th=[  219], 50.00th=[  233], 60.00th=[  251],
     | 70.00th=[  273], 80.00th=[  306], 90.00th=[  367], 95.00th=[  429],
     | 99.00th=[  578], 99.50th=[  685], 99.90th=[ 1385], 99.95th=[ 2409],
     | 99.99th=[ 7635]
   bw (  KiB/s): min=314128, max=532640, per=100.00%, avg=487248.39, stdev=36313.96, samples=119
   iops        : min=78532, max=133160, avg=121812.12, stdev=9078.51, samples=119
  lat (usec)   : 100=0.01%, 250=59.98%, 500=37.88%, 750=1.78%, 1000=0.18%
  lat (msec)   : 2=0.12%, 4=0.04%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=19.05%, sys=48.87%, ctx=1546905, majf=0, minf=67
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=7309189,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=476MiB/s (499MB/s), 476MiB/s-476MiB/s (499MB/s-499MB/s), io=27.9GiB (29.9GB), run=60001-60001msec

Disk stats (read/write):
  drbd3728: ios=7296015/0, merge=0/0, ticks=1825428/0, in_queue=1825428, util=99.46%


# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=1024 -randrepeat=0 -rw=randread -filename=/dev/drbd3728 -runtime=60
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1024
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=552MiB/s][r=141k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2058871: Thu Nov 12 21:15:11 2020
  read: IOPS=141k, BW=550MiB/s (577MB/s)(32.2GiB/60022msec)
    slat (nsec): min=1640, max=752946, avg=5673.75, stdev=6957.85
    clat (usec): min=146, max=44935, avg=7263.16, stdev=3861.35
     lat (usec): min=150, max=44938, avg=7268.89, stdev=3861.39
    clat percentiles (usec):
     |  1.00th=[  515],  5.00th=[ 1680], 10.00th=[ 2507], 20.00th=[ 3752],
     | 30.00th=[ 4752], 40.00th=[ 5866], 50.00th=[ 6980], 60.00th=[ 8160],
     | 70.00th=[ 9241], 80.00th=[10552], 90.00th=[11994], 95.00th=[13304],
     | 99.00th=[18482], 99.50th=[19530], 99.90th=[21365], 99.95th=[30540],
     | 99.99th=[40109]
   bw (  KiB/s): min=428960, max=599568, per=100.00%, avg=563511.20, stdev=29656.54, samples=120
   iops        : min=107240, max=149892, avg=140877.81, stdev=7414.12, samples=120
  lat (usec)   : 250=0.01%, 500=0.92%, 750=0.96%, 1000=0.65%
  lat (msec)   : 2=4.30%, 4=15.43%, 10=53.30%, 20=24.14%, 50=0.30%
  cpu          : usr=18.87%, sys=78.93%, ctx=46134, majf=0, minf=1057
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=8453977,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1024

Run status group 0 (all jobs):
   READ: bw=550MiB/s (577MB/s), 550MiB/s-550MiB/s (577MB/s-577MB/s), io=32.2GiB (34.6GB), run=60022-60022msec

Disk stats (read/write):
  drbd3728: ios=8441031/0, merge=0/0, ticks=53187528/0, in_queue=53187524, util=99.86%

drbd-local

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=1 -randrepeat=0 -rw=randwrite -filename=/dev/drbd3728 -runtime=60
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=16.1MiB/s][r=0,w=4128 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=30987: Thu Nov 12 21:17:42 2020
  write: IOPS=3907, BW=15.3MiB/s (16.0MB/s)(916MiB/60001msec)
    slat (usec): min=2, max=578, avg= 6.16, stdev= 4.94
    clat (usec): min=25, max=21688, avg=247.98, stdev=143.86
     lat (usec): min=68, max=21693, avg=254.31, stdev=143.66
    clat percentiles (usec):
     |  1.00th=[   70],  5.00th=[  159], 10.00th=[  176], 20.00th=[  194],
     | 30.00th=[  212], 40.00th=[  231], 50.00th=[  243], 60.00th=[  255],
     | 70.00th=[  269], 80.00th=[  285], 90.00th=[  314], 95.00th=[  347],
     | 99.00th=[  545], 99.50th=[  668], 99.90th=[ 1336], 99.95th=[ 2474],
     | 99.99th=[ 5407]
   bw (  KiB/s): min= 9824, max=17720, per=99.93%, avg=15619.71, stdev=1380.86, samples=119
   iops        : min= 2456, max= 4430, avg=3904.90, stdev=345.23, samples=119
  lat (usec)   : 50=0.01%, 100=3.56%, 250=52.11%, 500=43.05%, 750=0.92%
  lat (usec)   : 1000=0.18%
  lat (msec)   : 2=0.12%, 4=0.04%, 10=0.02%, 20=0.01%, 50=0.01%
  cpu          : usr=1.59%, sys=3.70%, ctx=234539, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,234474,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=15.3MiB/s (16.0MB/s), 15.3MiB/s-15.3MiB/s (16.0MB/s-16.0MB/s), io=916MiB (960MB), run=60001-60001msec

Disk stats (read/write):
    drbd3728: ios=0/233971, merge=0/0, ticks=0/57576, in_queue=57444, util=95.80%, aggrios=0/589164, aggrmerge=0/0, aggrticks=0/19984, aggrin_queue=20124, aggrutil=31.31%
    dm-19: ios=0/589164, merge=0/0, ticks=0/19984, in_queue=20124, util=31.31%, aggrios=1/597560, aggrmerge=0/0, aggrticks=0/15892, aggrin_queue=15960, aggrutil=22.05%
    dm-5: ios=1/597560, merge=0/0, ticks=0/15892, in_queue=15960, util=22.05%, aggrios=5/298780, aggrmerge=0/0, aggrticks=0/7682, aggrin_queue=7720, aggrutil=21.19%
    dm-4: ios=1/597560, merge=0/0, ticks=0/15364, in_queue=15440, util=21.19%, aggrios=54/593073, aggrmerge=0/4487, aggrticks=8/14224, aggrin_queue=1136, aggrutil=0.13%
  nvme0n1: ios=54/593073, merge=0/4487, ticks=8/14224, in_queue=1136, util=0.13%
  dm-3: ios=9/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=32 -randrepeat=0 -rw=randwrite -filename=/dev/drbd3728 -runtime=60
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=176MiB/s][r=0,w=45.1k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=31926: Thu Nov 12 21:19:01 2020
  write: IOPS=43.8k, BW=171MiB/s (179MB/s)(10.0GiB/60001msec)
    slat (nsec): min=1259, max=3878.5k, avg=3779.92, stdev=5827.58
    clat (usec): min=34, max=26666, avg=726.06, stdev=309.85
     lat (usec): min=71, max=26668, avg=729.91, stdev=308.92
    clat percentiles (usec):
     |  1.00th=[  163],  5.00th=[  453], 10.00th=[  537], 20.00th=[  594],
     | 30.00th=[  635], 40.00th=[  676], 50.00th=[  709], 60.00th=[  742],
     | 70.00th=[  783], 80.00th=[  832], 90.00th=[  914], 95.00th=[ 1020],
     | 99.00th=[ 1500], 99.50th=[ 1893], 99.90th=[ 4113], 99.95th=[ 5276],
     | 99.99th=[ 9110]
   bw (  KiB/s): min=130680, max=193768, per=100.00%, avg=175207.40, stdev=11328.96, samples=120
   iops        : min=32670, max=48442, avg=43801.82, stdev=2832.18, samples=120
  lat (usec)   : 50=0.01%, 100=0.13%, 250=2.62%, 500=4.02%, 750=55.46%
  lat (usec)   : 1000=32.28%
  lat (msec)   : 2=5.05%, 4=0.33%, 10=0.10%, 20=0.01%, 50=0.01%
  cpu          : usr=6.96%, sys=19.58%, ctx=779584, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,2627695,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=171MiB/s (179MB/s), 171MiB/s-171MiB/s (179MB/s-179MB/s), io=10.0GiB (10.8GB), run=60001-60001msec

Disk stats (read/write):
    drbd3728: ios=0/2622211, merge=0/0, ticks=0/1878912, in_queue=1877568, util=99.72%, aggrios=0/4426644, aggrmerge=0/0, aggrticks=0/108136, aggrin_queue=108668, aggrutil=86.41%
    dm-19: ios=0/4426644, merge=0/0, ticks=0/108136, in_queue=108668, util=86.41%, aggrios=89/4434064, aggrmerge=0/0, aggrticks=8/87840, aggrin_queue=88360, aggrutil=75.13%
    dm-5: ios=89/4434064, merge=0/0, ticks=8/87840, in_queue=88360, util=75.13%, aggrios=680/2217032, aggrmerge=0/0, aggrticks=62/42422, aggrin_queue=42714, aggrutil=73.84%
    dm-4: ios=89/4434064, merge=0/0, ticks=8/84844, in_queue=85312, util=73.84%, aggrios=1381/4430324, aggrmerge=0/3740, aggrticks=128/83740, aggrin_queue=21876, aggrutil=28.85%
  nvme0n1: ios=1381/4430324, merge=0/3740, ticks=128/83740, in_queue=21876, util=28.85%
  dm-3: ios=1272/0, merge=0/0, ticks=116/0, in_queue=116, util=0.11%

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=512 -randrepeat=0 -rw=randwrite -filename=/dev/drbd3728 -runtime=60
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=512
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=250MiB/s][r=0,w=63.9k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2634: Thu Nov 12 21:20:37 2020
  write: IOPS=63.6k, BW=248MiB/s (260MB/s)(14.6GiB/60009msec)
    slat (nsec): min=1215, max=2460.9k, avg=4261.50, stdev=9978.47
    clat (usec): min=54, max=36269, avg=8046.25, stdev=2473.09
     lat (usec): min=78, max=36271, avg=8050.59, stdev=2471.27
    clat percentiles (usec):
     |  1.00th=[  441],  5.00th=[ 4948], 10.00th=[ 6718], 20.00th=[ 7177],
     | 30.00th=[ 7439], 40.00th=[ 7635], 50.00th=[ 7898], 60.00th=[ 8160],
     | 70.00th=[ 8455], 80.00th=[ 8979], 90.00th=[ 9896], 95.00th=[11469],
     | 99.00th=[16909], 99.50th=[18744], 99.90th=[24511], 99.95th=[28181],
     | 99.99th=[33424]
   bw (  KiB/s): min=175768, max=279288, per=100.00%, avg=254409.23, stdev=17253.55, samples=120
   iops        : min=43942, max=69822, avg=63602.27, stdev=4313.40, samples=120
  lat (usec)   : 100=0.01%, 250=0.28%, 500=0.98%, 750=0.99%, 1000=0.71%
  lat (msec)   : 2=1.18%, 4=0.58%, 10=85.93%, 20=8.97%, 50=0.36%
  cpu          : usr=8.49%, sys=31.94%, ctx=955586, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwt: total=0,3815938,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=512

Run status group 0 (all jobs):
  WRITE: bw=248MiB/s (260MB/s), 248MiB/s-248MiB/s (260MB/s-260MB/s), io=14.6GiB (15.6GB), run=60009-60009msec

Disk stats (read/write):
    drbd3728: ios=0/3808444, merge=0/0, ticks=0/30464608, in_queue=30489960, util=99.92%, aggrios=0/5780196, aggrmerge=0/0, aggrticks=0/147404, aggrin_queue=148128, aggrutil=95.21%
    dm-19: ios=0/5780196, merge=0/0, ticks=0/147404, in_queue=148128, util=95.21%, aggrios=131/5787974, aggrmerge=0/0, aggrticks=20/122684, aggrin_queue=123320, aggrutil=91.81%
    dm-5: ios=131/5787974, merge=0/0, ticks=20/122684, in_queue=123320, util=91.81%, aggrios=716/2893987, aggrmerge=0/0, aggrticks=110/59412, aggrin_queue=59898, aggrutil=91.44%
    dm-4: ios=131/5787974, merge=0/0, ticks=20/118824, in_queue=119592, util=91.44%, aggrios=1451/5783864, aggrmerge=1/4110, aggrticks=224/118028, aggrin_queue=27408, aggrutil=38.61%
  nvme0n1: ios=1451/5783864, merge=1/4110, ticks=224/118028, in_queue=27408, util=38.61%
  dm-3: ios=1301/0, merge=0/0, ticks=200/0, in_queue=204, util=0.15%

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=1024 -randrepeat=0 -rw=randwrite -filename=/dev/drbd3728 -runtime=60
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1024
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=253MiB/s][r=0,w=64.8k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=4660: Thu Nov 12 21:21:59 2020
  write: IOPS=65.6k, BW=256MiB/s (269MB/s)(15.0GiB/60016msec)
    slat (nsec): min=1192, max=1754.9k, avg=4234.41, stdev=9378.61
    clat (usec): min=24, max=47057, avg=15597.82, stdev=4411.81
     lat (usec): min=67, max=47059, avg=15602.13, stdev=4409.63
    clat percentiles (usec):
     |  1.00th=[  429],  5.00th=[ 6456], 10.00th=[13566], 20.00th=[14222],
     | 30.00th=[14746], 40.00th=[15139], 50.00th=[15664], 60.00th=[16188],
     | 70.00th=[16712], 80.00th=[17695], 90.00th=[19006], 95.00th=[21365],
     | 99.00th=[27919], 99.50th=[31589], 99.90th=[38011], 99.95th=[40633],
     | 99.99th=[45351]
   bw (  KiB/s): min=173048, max=296176, per=100.00%, avg=262514.30, stdev=16350.00, samples=120
   iops        : min=43262, max=74044, avg=65628.56, stdev=4087.49, samples=120
  lat (usec)   : 50=0.01%, 100=0.02%, 250=0.30%, 500=1.01%, 750=1.01%
  lat (usec)   : 1000=0.69%
  lat (msec)   : 2=1.15%, 4=0.48%, 10=1.14%, 20=86.90%, 50=7.32%
  cpu          : usr=9.19%, sys=32.56%, ctx=976741, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwt: total=0,3938411,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1024

Run status group 0 (all jobs):
  WRITE: bw=256MiB/s (269MB/s), 256MiB/s-256MiB/s (269MB/s-269MB/s), io=15.0GiB (16.1GB), run=60016-60016msec

Disk stats (read/write):
    drbd3728: ios=0/3930689, merge=0/0, ticks=0/61176484, in_queue=61217544, util=99.91%, aggrios=0/5661156, aggrmerge=0/0, aggrticks=0/146920, aggrin_queue=147716, aggrutil=95.05%
    dm-19: ios=0/5661156, merge=0/0, ticks=0/146920, in_queue=147716, util=95.05%, aggrios=54/5667990, aggrmerge=0/0, aggrticks=20/121840, aggrin_queue=122652, aggrutil=91.99%
    dm-5: ios=54/5667990, merge=0/0, ticks=20/121840, in_queue=122652, util=91.99%, aggrios=659/2833995, aggrmerge=0/0, aggrticks=68/58964, aggrin_queue=59396, aggrutil=91.47%
    dm-4: ios=54/5667990, merge=0/0, ticks=20/117928, in_queue=118676, util=91.47%, aggrios=1350/5664491, aggrmerge=0/3499, aggrticks=148/117484, aggrin_queue=26212, aggrutil=37.76%
  nvme0n1: ios=1350/5664491, merge=0/3499, ticks=148/117484, in_queue=26212, util=37.76%
  dm-3: ios=1264/0, merge=0/0, ticks=116/0, in_queue=116, util=0.09%

# -----------------------------------------------------

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=1 -randrepeat=0 -rw=randread -filename=/dev/drbd3728 -runtime=60
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.3MiB/s,w=0KiB/s][r=10.6k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=5748: Thu Nov 12 21:23:14 2020
   read: IOPS=9778, BW=38.2MiB/s (40.1MB/s)(2292MiB/60001msec)
    slat (usec): min=5, max=2059, avg=14.53, stdev=11.57
    clat (nsec): min=1402, max=19015k, avg=85923.27, stdev=42102.95
     lat (usec): min=59, max=19022, avg=100.62, stdev=45.00
    clat percentiles (usec):
     |  1.00th=[   64],  5.00th=[   74], 10.00th=[   75], 20.00th=[   76],
     | 30.00th=[   78], 40.00th=[   80], 50.00th=[   82], 60.00th=[   88],
     | 70.00th=[   91], 80.00th=[   94], 90.00th=[   99], 95.00th=[  106],
     | 99.00th=[  130], 99.50th=[  143], 99.90th=[  322], 99.95th=[  416],
     | 99.99th=[  619]
   bw (  KiB/s): min=24328, max=43184, per=100.00%, avg=39113.39, stdev=3403.15, samples=119
   iops        : min= 6082, max=10796, avg=9778.29, stdev=850.82, samples=119
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=91.02%, 250=8.86%, 500=0.09%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=3.62%, sys=17.61%, ctx=586986, majf=0, minf=30
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=586728,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=38.2MiB/s (40.1MB/s), 38.2MiB/s-38.2MiB/s (40.1MB/s-40.1MB/s), io=2292MiB (2403MB), run=60001-60001msec

Disk stats (read/write):
    drbd3728: ios=585732/0, merge=0/0, ticks=53004/0, in_queue=52716, util=87.91%, aggrios=586728/0, aggrmerge=0/0, aggrticks=52020/0, aggrin_queue=52168, aggrutil=86.74%
    dm-19: ios=586728/0, merge=0/0, ticks=52020/0, in_queue=52168, util=86.74%, aggrios=586867/9013, aggrmerge=0/0, aggrticks=48276/4744, aggrin_queue=53128, aggrutil=80.52%
    dm-5: ios=586867/9013, merge=0/0, ticks=48276/4744, in_queue=53128, util=80.52%, aggrios=293447/4506, aggrmerge=0/0, aggrticks=23842/2370, aggrin_queue=26248, aggrutil=79.49%
    dm-4: ios=586867/9013, merge=0/0, ticks=47684/4740, in_queue=52496, util=79.49%, aggrios=586927/3807, aggrmerge=0/5206, aggrticks=48576/260, aggrin_queue=108, aggrutil=0.18%
  nvme0n1: ios=586927/3807, merge=0/5206, ticks=48576/260, in_queue=108, util=0.18%
  dm-3: ios=28/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=32 -randrepeat=0 -rw=randread -filename=/dev/drbd3728 -runtime=60
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.1

Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=404MiB/s,w=0KiB/s][r=103k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=6558: Thu Nov 12 21:24:27 2020
   read: IOPS=107k, BW=418MiB/s (438MB/s)(24.5GiB/60001msec)
    slat (usec): min=2, max=888, avg= 7.90, stdev= 5.09
    clat (usec): min=56, max=3966, avg=290.36, stdev=54.34
     lat (usec): min=62, max=3972, avg=298.34, stdev=55.28
    clat percentiles (usec):
     |  1.00th=[  215],  5.00th=[  227], 10.00th=[  235], 20.00th=[  247],
     | 30.00th=[  258], 40.00th=[  269], 50.00th=[  281], 60.00th=[  293],
     | 70.00th=[  310], 80.00th=[  330], 90.00th=[  363], 95.00th=[  388],
     | 99.00th=[  449], 99.50th=[  482], 99.90th=[  668], 99.95th=[  709],
     | 99.99th=[  783]
   bw (  KiB/s): min=363320, max=450856, per=100.00%, avg=428399.89, stdev=14983.14, samples=119
   iops        : min=90830, max=112714, avg=107100.00, stdev=3745.82, samples=119
  lat (usec)   : 100=0.01%, 250=23.70%, 500=75.90%, 750=0.38%, 1000=0.02%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=13.37%, sys=86.61%, ctx=590, majf=0, minf=62
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=6420651,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=418MiB/s (438MB/s), 418MiB/s-418MiB/s (438MB/s-438MB/s), io=24.5GiB (26.3GB), run=60001-60001msec

Disk stats (read/write):
    drbd3728: ios=6410105/0, merge=0/0, ticks=570424/0, in_queue=568504, util=99.88%, aggrios=6420651/0, aggrmerge=0/0, aggrticks=566364/0, aggrin_queue=567268, aggrutil=99.81%
    dm-19: ios=6420651/0, merge=0/0, ticks=566364/0, in_queue=567268, util=99.81%, aggrios=6420837/7634, aggrmerge=0/0, aggrticks=541900/1280, aggrin_queue=544004, aggrutil=99.80%
    dm-5: ios=6420837/7634, merge=0/0, ticks=541900/1280, in_queue=544004, util=99.80%, aggrios=3211079/3828, aggrmerge=0/0, aggrticks=269226/638, aggrin_queue=270120, aggrutil=99.81%
    dm-4: ios=6420837/7634, merge=0/0, ticks=538356/1276, in_queue=540144, util=99.81%, aggrios=6422177/3913, aggrmerge=1/3743, aggrticks=534988/128, aggrin_queue=481132, aggrutil=99.82%
  nvme0n1: ios=6422177/3913, merge=1/3743, ticks=534988/128, in_queue=481132, util=99.82%
  dm-3: ios=1321/22, merge=0/0, ticks=96/0, in_queue=96, util=0.07%

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=512 -randrepeat=0 -rw=randread -filename=/dev/drbd3728 -runtime=60
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=512
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=426MiB/s,w=0KiB/s][r=109k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=9175: Thu Nov 12 21:25:31 2020
   read: IOPS=106k, BW=414MiB/s (434MB/s)(24.2GiB/60001msec)
    slat (usec): min=3, max=510, avg= 7.95, stdev= 5.03
    clat (usec): min=89, max=14428, avg=4827.09, stdev=473.24
     lat (usec): min=96, max=14593, avg=4835.13, stdev=473.89
    clat percentiles (usec):
     |  1.00th=[ 4178],  5.00th=[ 4293], 10.00th=[ 4359], 20.00th=[ 4490],
     | 30.00th=[ 4555], 40.00th=[ 4621], 50.00th=[ 4752], 60.00th=[ 4817],
     | 70.00th=[ 4948], 80.00th=[ 5080], 90.00th=[ 5342], 95.00th=[ 5800],
     | 99.00th=[ 6587], 99.50th=[ 6783], 99.90th=[ 7242], 99.95th=[ 7504],
     | 99.99th=[ 8029]
   bw (  KiB/s): min=365731, max=445760, per=100.00%, avg=423496.88, stdev=14529.90, samples=120
   iops        : min=91432, max=111444, avg=105874.27, stdev=3632.55, samples=120
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.03%, 10=99.96%, 20=0.01%
  cpu          : usr=13.52%, sys=86.49%, ctx=242, majf=0, minf=541
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwt: total=6351628,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=512

Run status group 0 (all jobs):
   READ: bw=414MiB/s (434MB/s), 414MiB/s-414MiB/s (434MB/s-434MB/s), io=24.2GiB (26.0GB), run=60001-60001msec

Disk stats (read/write):
    drbd3728: ios=6338873/0, merge=0/0, ticks=560608/0, in_queue=558804, util=99.83%, aggrios=6351628/0, aggrmerge=0/0, aggrticks=556976/0, aggrin_queue=557844, aggrutil=99.73%
    dm-19: ios=6351628/0, merge=0/0, ticks=556976/0, in_queue=557844, util=99.73%, aggrios=6351694/8381, aggrmerge=0/0, aggrticks=533420/2648, aggrin_queue=536704, aggrutil=99.70%
    dm-5: ios=6351694/8381, merge=0/0, ticks=533420/2648, in_queue=536704, util=99.70%, aggrios=3175869/4190, aggrmerge=0/0, aggrticks=265042/1322, aggrin_queue=266650, aggrutil=99.69%
    dm-4: ios=6351694/8381, merge=0/0, ticks=530084/2644, in_queue=533300, util=99.69%, aggrios=6351770/4096, aggrmerge=0/4285, aggrticks=526540/156, aggrin_queue=472176, aggrutil=99.68%
  nvme0n1: ios=6351770/4096, merge=0/4285, ticks=526540/156, in_queue=472176, util=99.68%
  dm-3: ios=44/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=1024 -randrepeat=0 -rw=randread -filename=/dev/drbd3728 -runtime=60
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1024
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=404MiB/s,w=0KiB/s][r=103k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=10148: Thu Nov 12 21:26:33 2020
   read: IOPS=103k, BW=403MiB/s (423MB/s)(23.6GiB/60001msec)
    slat (usec): min=3, max=518, avg= 8.18, stdev= 5.09
    clat (usec): min=70, max=16192, avg=9905.72, stdev=951.85
     lat (usec): min=82, max=16205, avg=9913.97, stdev=952.52
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[ 8848], 10.00th=[ 8979], 20.00th=[ 9241],
     | 30.00th=[ 9372], 40.00th=[ 9503], 50.00th=[ 9634], 60.00th=[ 9896],
     | 70.00th=[10028], 80.00th=[10421], 90.00th=[11076], 95.00th=[11994],
     | 99.00th=[13304], 99.50th=[13698], 99.90th=[15008], 99.95th=[15401],
     | 99.99th=[15795]
   bw (  KiB/s): min=337352, max=438528, per=100.00%, avg=413108.18, stdev=16881.24, samples=120
   iops        : min=84338, max=109632, avg=103277.10, stdev=4220.39, samples=120
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=66.50%, 20=33.49%
  cpu          : usr=14.23%, sys=85.78%, ctx=170, majf=0, minf=1053
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwt: total=6196367,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1024

Run status group 0 (all jobs):
   READ: bw=403MiB/s (423MB/s), 403MiB/s-403MiB/s (423MB/s-423MB/s), io=23.6GiB (25.4GB), run=60001-60001msec

Disk stats (read/write):
    drbd3728: ios=6184417/0, merge=0/0, ticks=549012/0, in_queue=547224, util=99.86%, aggrios=6196367/0, aggrmerge=0/0, aggrticks=545504/0, aggrin_queue=546108, aggrutil=99.75%
    dm-19: ios=6196367/0, merge=0/0, ticks=545504/0, in_queue=546108, util=99.75%, aggrios=6196468/7588, aggrmerge=0/0, aggrticks=522368/1304, aggrin_queue=524244, aggrutil=99.75%
    dm-5: ios=6196468/7588, merge=0/0, ticks=522368/1304, in_queue=524244, util=99.75%, aggrios=3098247/3794, aggrmerge=0/0, aggrticks=259410/648, aggrin_queue=260332, aggrutil=99.76%
    dm-4: ios=6196468/7588, merge=0/0, ticks=518820/1296, in_queue=520664, util=99.76%, aggrios=6196514/3779, aggrmerge=0/3809, aggrticks=515364/140, aggrin_queue=461528, aggrutil=99.76%
  nvme0n1: ios=6196514/3779, merge=0/3809, ticks=515364/140, in_queue=461528, util=99.76%
  dm-3: ios=26/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

local

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=1 -randrepeat=0 -rw=randwrite -filename=/dev/data/testzzz_00000 -runtime=60
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=129MiB/s][r=0,w=33.1k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=12729: Thu Nov 12 21:29:02 2020
  write: IOPS=32.1k, BW=126MiB/s (132MB/s)(7532MiB/60001msec)
    slat (usec): min=3, max=1168, avg= 7.64, stdev= 4.08
    clat (nsec): min=606, max=15939k, avg=22502.97, stdev=16572.05
     lat (usec): min=16, max=15952, avg=30.23, stdev=17.32
    clat percentiles (usec):
     |  1.00th=[   18],  5.00th=[   19], 10.00th=[   21], 20.00th=[   21],
     | 30.00th=[   21], 40.00th=[   22], 50.00th=[   22], 60.00th=[   22],
     | 70.00th=[   23], 80.00th=[   24], 90.00th=[   26], 95.00th=[   28],
     | 99.00th=[   38], 99.50th=[   48], 99.90th=[   90], 99.95th=[  124],
     | 99.99th=[  359]
   bw (  KiB/s): min=95800, max=137796, per=99.98%, avg=128523.64, stdev=6492.42, samples=119
   iops        : min=23950, max=34449, avg=32130.89, stdev=1623.10, samples=119
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=9.30%, 50=90.24%
  lat (usec)   : 100=0.36%, 250=0.06%, 500=0.02%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=7.16%, sys=29.88%, ctx=1929001, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,1928234,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=7532MiB (7898MB), run=60001-60001msec

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=32 -randrepeat=0 -rw=randwrite -filename=/dev/data/testzzz_00000 -runtime=60
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=467MiB/s][r=0,w=120k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=14807: Thu Nov 12 21:30:06 2020
  write: IOPS=118k, BW=461MiB/s (484MB/s)(27.0GiB/60001msec)
    slat (usec): min=2, max=835, avg= 6.98, stdev= 4.15
    clat (usec): min=9, max=3295, avg=263.21, stdev=49.40
     lat (usec): min=17, max=3330, avg=270.26, stdev=50.35
    clat percentiles (usec):
     |  1.00th=[  212],  5.00th=[  215], 10.00th=[  219], 20.00th=[  227],
     | 30.00th=[  231], 40.00th=[  239], 50.00th=[  251], 60.00th=[  262],
     | 70.00th=[  277], 80.00th=[  297], 90.00th=[  330], 95.00th=[  355],
     | 99.00th=[  404], 99.50th=[  433], 99.90th=[  652], 99.95th=[  693],
     | 99.99th=[  873]
   bw (  KiB/s): min=386576, max=500024, per=99.98%, avg=472127.33, stdev=19777.81, samples=119
   iops        : min=96644, max=125006, avg=118031.84, stdev=4944.48, samples=119
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=50.08%
  lat (usec)   : 500=49.62%, 750=0.26%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=17.97%, sys=81.92%, ctx=3198, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,7083625,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=461MiB/s (484MB/s), 461MiB/s-461MiB/s (484MB/s-484MB/s), io=27.0GiB (29.0GB), run=60001-60001msec

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=512 -randrepeat=0 -rw=randwrite -filename=/dev/data/testzzz_00000 -runtime=60
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=512
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=447MiB/s][r=0,w=114k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=16077: Thu Nov 12 21:31:07 2020
  write: IOPS=115k, BW=449MiB/s (471MB/s)(26.3GiB/60001msec)
    slat (nsec): min=1500, max=9222.2k, avg=7067.87, stdev=5674.75
    clat (usec): min=10, max=48284, avg=4446.25, stdev=615.74
     lat (usec): min=17, max=48286, avg=4453.38, stdev=616.25
    clat percentiles (usec):
     |  1.00th=[ 3752],  5.00th=[ 3851], 10.00th=[ 3949], 20.00th=[ 4047],
     | 30.00th=[ 4146], 40.00th=[ 4228], 50.00th=[ 4359], 60.00th=[ 4424],
     | 70.00th=[ 4555], 80.00th=[ 4752], 90.00th=[ 5080], 95.00th=[ 5538],
     | 99.00th=[ 6063], 99.50th=[ 6194], 99.90th=[ 6718], 99.95th=[ 8291],
     | 99.99th=[26346]
   bw (  KiB/s): min=392592, max=496880, per=100.00%, avg=459788.54, stdev=19931.06, samples=119
   iops        : min=98148, max=124220, avg=114947.13, stdev=4982.78, samples=119
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.02%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=15.18%, 10=84.71%, 20=0.02%, 50=0.01%
  cpu          : usr=19.17%, sys=80.72%, ctx=1811, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwt: total=0,6896627,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=512

Run status group 0 (all jobs):
  WRITE: bw=449MiB/s (471MB/s), 449MiB/s-449MiB/s (471MB/s-471MB/s), io=26.3GiB (28.2GB), run=60001-60001msec

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=1024 -randrepeat=0 -rw=randwrite -filename=/dev/data/testzzz_00000 -runtime=60
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1024
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=465MiB/s][r=0,w=119k IOPS][eta 00m:00s]15s]
test: (groupid=0, jobs=1): err= 0: pid=17956: Thu Nov 12 21:32:08 2020
  write: IOPS=116k, BW=452MiB/s (474MB/s)(26.5GiB/60001msec)
    slat (nsec): min=1558, max=674165, avg=7098.59, stdev=4331.38
    clat (usec): min=12, max=89893, avg=8831.70, stdev=1069.02
     lat (usec): min=17, max=89897, avg=8838.86, stdev=1069.37
    clat percentiles (usec):
     |  1.00th=[ 7635],  5.00th=[ 7898], 10.00th=[ 8029], 20.00th=[ 8225],
     | 30.00th=[ 8356], 40.00th=[ 8455], 50.00th=[ 8717], 60.00th=[ 8848],
     | 70.00th=[ 8979], 80.00th=[ 9372], 90.00th=[ 9765], 95.00th=[10421],
     | 99.00th=[11731], 99.50th=[12125], 99.90th=[12911], 99.95th=[13304],
     | 99.99th=[49021]
   bw (  KiB/s): min=405504, max=489176, per=99.97%, avg=463188.86, stdev=15795.44, samples=119
   iops        : min=101376, max=122294, avg=115797.24, stdev=3948.89, samples=119
  lat (usec)   : 20=0.01%, 50=0.02%, 100=0.01%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.02%, 10=91.84%, 20=8.04%, 50=0.02%
  lat (msec)   : 100=0.01%
  cpu          : usr=18.42%, sys=81.53%, ctx=1588, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwt: total=0,6950189,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1024

Run status group 0 (all jobs):
  WRITE: bw=452MiB/s (474MB/s), 452MiB/s-452MiB/s (474MB/s-474MB/s), io=26.5GiB (28.5GB), run=60001-60001msec

# -----------------------------------------------------

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=1 -randrepeat=0 -rw=randread -filename=/dev/data/testzzz_00000 -runtime=60
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=40.5MiB/s,w=0KiB/s][r=10.4k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=18814: Thu Nov 12 21:33:10 2020
   read: IOPS=9777, BW=38.2MiB/s (40.0MB/s)(2292MiB/60001msec)
    slat (usec): min=3, max=2067, avg=14.08, stdev=11.19
    clat (nsec): min=1892, max=11283k, avg=86108.70, stdev=34153.24
     lat (usec): min=56, max=11296, avg=100.38, stdev=37.47
    clat percentiles (usec):
     |  1.00th=[   64],  5.00th=[   74], 10.00th=[   75], 20.00th=[   76],
     | 30.00th=[   78], 40.00th=[   80], 50.00th=[   82], 60.00th=[   88],
     | 70.00th=[   91], 80.00th=[   94], 90.00th=[   99], 95.00th=[  106],
     | 99.00th=[  135], 99.50th=[  157], 99.90th=[  347], 99.95th=[  429],
     | 99.99th=[  783]
   bw (  KiB/s): min=26080, max=43904, per=99.90%, avg=39071.71, stdev=3706.51, samples=119
   iops        : min= 6520, max=10976, avg=9767.90, stdev=926.64, samples=119
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=90.83%, 250=9.02%, 500=0.12%, 750=0.02%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=3.89%, sys=17.43%, ctx=587013, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=586653,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=38.2MiB/s (40.0MB/s), 38.2MiB/s-38.2MiB/s (40.0MB/s-40.0MB/s), io=2292MiB (2403MB), run=60001-60001msec

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=32 -randrepeat=0 -rw=randread -filename=/dev/data/testzzz_00000 -runtime=60
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=461MiB/s,w=0KiB/s][r=118k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=19566: Thu Nov 12 21:34:11 2020
   read: IOPS=120k, BW=469MiB/s (491MB/s)(27.5GiB/60001msec)
    slat (usec): min=3, max=522, avg= 6.96, stdev= 4.59
    clat (usec): min=69, max=2693, avg=259.09, stdev=52.67
     lat (usec): min=73, max=2705, avg=266.11, stdev=53.52
    clat percentiles (usec):
     |  1.00th=[  188],  5.00th=[  200], 10.00th=[  208], 20.00th=[  219],
     | 30.00th=[  227], 40.00th=[  239], 50.00th=[  249], 60.00th=[  262],
     | 70.00th=[  277], 80.00th=[  297], 90.00th=[  326], 95.00th=[  351],
     | 99.00th=[  408], 99.50th=[  445], 99.90th=[  660], 99.95th=[  709],
     | 99.99th=[  955]
   bw (  KiB/s): min=406544, max=506344, per=100.00%, avg=480032.82, stdev=17464.33, samples=119
   iops        : min=101636, max=126586, avg=120008.20, stdev=4366.08, samples=119
  lat (usec)   : 100=0.01%, 250=50.88%, 500=48.78%, 750=0.31%, 1000=0.02%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=14.90%, sys=85.06%, ctx=1203, majf=0, minf=60
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=7197330,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=469MiB/s (491MB/s), 469MiB/s-469MiB/s (491MB/s-491MB/s), io=27.5GiB (29.5GB), run=60001-60001msec

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=512 -randrepeat=0 -rw=randread -filename=/dev/data/testzzz_00000 -runtime=60
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=512
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=438MiB/s,w=0KiB/s][r=112k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=21524: Thu Nov 12 21:35:12 2020
   read: IOPS=118k, BW=462MiB/s (484MB/s)(27.0GiB/60001msec)
    slat (nsec): min=1745, max=602280, avg=7057.87, stdev=4560.24
    clat (usec): min=51, max=40835, avg=4325.61, stdev=510.56
     lat (usec): min=55, max=40838, avg=4332.73, stdev=511.00
    clat percentiles (usec):
     |  1.00th=[ 3687],  5.00th=[ 3818], 10.00th=[ 3884], 20.00th=[ 3982],
     | 30.00th=[ 4080], 40.00th=[ 4146], 50.00th=[ 4228], 60.00th=[ 4359],
     | 70.00th=[ 4424], 80.00th=[ 4621], 90.00th=[ 4883], 95.00th=[ 5211],
     | 99.00th=[ 5800], 99.50th=[ 5932], 99.90th=[ 6325], 99.95th=[ 6521],
     | 99.99th=[20841]
   bw (  KiB/s): min=419424, max=504312, per=100.00%, avg=472812.06, stdev=16526.78, samples=119
   iops        : min=104856, max=126078, avg=118203.03, stdev=4131.67, samples=119
  lat (usec)   : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=21.58%, 10=78.35%, 20=0.01%, 50=0.01%
  cpu          : usr=14.12%, sys=85.84%, ctx=1262, majf=0, minf=540
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwt: total=7088936,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=512

Run status group 0 (all jobs):
   READ: bw=462MiB/s (484MB/s), 462MiB/s-462MiB/s (484MB/s-484MB/s), io=27.0GiB (29.0GB), run=60001-60001msec

# fio -ioengine=libaio -name=test -bs=4k -direct=1 -numjobs=1 -iodepth=1024 -randrepeat=0 -rw=randread -filename=/dev/data/testzzz_00000 -runtime=60
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1024
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=448MiB/s,w=0KiB/s][r=115k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=22877: Thu Nov 12 21:36:14 2020
   read: IOPS=118k, BW=462MiB/s (484MB/s)(27.1GiB/60001msec)
    slat (usec): min=2, max=1258, avg= 7.03, stdev= 4.62
    clat (usec): min=55, max=77740, avg=8654.69, stdev=971.49
     lat (usec): min=64, max=77750, avg=8661.78, stdev=971.93
    clat percentiles (usec):
     |  1.00th=[ 7570],  5.00th=[ 7767], 10.00th=[ 7898], 20.00th=[ 8094],
     | 30.00th=[ 8225], 40.00th=[ 8356], 50.00th=[ 8455], 60.00th=[ 8586],
     | 70.00th=[ 8848], 80.00th=[ 9110], 90.00th=[ 9503], 95.00th=[10159],
     | 99.00th=[11600], 99.50th=[11994], 99.90th=[12911], 99.95th=[13698],
     | 99.99th=[43254]
   bw (  KiB/s): min=393008, max=496560, per=99.99%, avg=472681.47, stdev=17830.98, samples=119
   iops        : min=98252, max=124140, avg=118170.35, stdev=4457.84, samples=119
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=94.10%, 20=5.83%, 50=0.03%
  lat (msec)   : 100=0.01%
  cpu          : usr=14.73%, sys=85.25%, ctx=744, majf=0, minf=1052
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwt: total=7091259,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1024

Run status group 0 (all jobs):
   READ: bw=462MiB/s (484MB/s), 462MiB/s-462MiB/s (484MB/s-484MB/s), io=27.1GiB (29.0GB), run=60001-60001msec