cc20dev0.naist.jp.txt

[keichi@cc20dev0 keichi]$ ./oci_fio_benchmark.sh
seqread-4k: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.26-19-ge7e53
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=25.1MiB/s][r=6432 IOPS][eta 00m:00s]
seqread-4k: (groupid=0, jobs=1): err= 0: pid=247802: Tue Mar 30 17:39:00 2021
  read: IOPS=6434, BW=25.1MiB/s (26.4MB/s)(1508MiB/60001msec)
    clat (usec): min=137, max=14343, avg=154.45, stdev=27.89
     lat (usec): min=138, max=14344, avg=154.63, stdev=27.89
    clat percentiles (usec):
     |  1.00th=[  143],  5.00th=[  145], 10.00th=[  145], 20.00th=[  147],
     | 30.00th=[  147], 40.00th=[  149], 50.00th=[  149], 60.00th=[  151],
     | 70.00th=[  153], 80.00th=[  157], 90.00th=[  172], 95.00th=[  192],
     | 99.00th=[  225], 99.50th=[  229], 99.90th=[  239], 99.95th=[  245],
     | 99.99th=[  265]
   bw (  KiB/s): min=24846, max=26056, per=100.00%, avg=25751.51, stdev=247.90, samples=119
   iops        : min= 6211, max= 6514, avg=6437.87, stdev=61.99, samples=119
  lat (usec)   : 250=99.97%, 500=0.03%, 750=0.01%
  lat (msec)   : 20=0.01%
  cpu          : usr=1.38%, sys=5.45%, ctx=386203, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=386063,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=25.1MiB/s (26.4MB/s), 25.1MiB/s-25.1MiB/s (26.4MB/s-26.4MB/s), io=1508MiB (1581MB), run=60001-60001msec
seqread-1m: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.26-19-ge7e53
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=556MiB/s][r=555 IOPS][eta 00m:00s]
seqread-1m: (groupid=0, jobs=1): err= 0: pid=247985: Tue Mar 30 17:39:07 2021
  read: IOPS=555, BW=555MiB/s (582MB/s)(4096MiB/7378msec)
    clat (usec): min=1555, max=24188, avg=1795.51, stdev=397.35
     lat (usec): min=1555, max=24189, avg=1796.14, stdev=397.35
    clat percentiles (usec):
     |  1.00th=[ 1647],  5.00th=[ 1680], 10.00th=[ 1713], 20.00th=[ 1745],
     | 30.00th=[ 1762], 40.00th=[ 1778], 50.00th=[ 1795], 60.00th=[ 1811],
     | 70.00th=[ 1827], 80.00th=[ 1827], 90.00th=[ 1844], 95.00th=[ 1860],
     | 99.00th=[ 1909], 99.50th=[ 1926], 99.90th=[ 2040], 99.95th=[ 3556],
     | 99.99th=[24249]
   bw (  KiB/s): min=538624, max=581632, per=100.00%, avg=568758.86, stdev=9278.93, samples=14
   iops        : min=  526, max=  568, avg=555.43, stdev= 9.06, samples=14
  lat (msec)   : 2=99.85%, 4=0.10%, 20=0.02%, 50=0.02%
  cpu          : usr=0.43%, sys=9.85%, ctx=4099, majf=0, minf=266
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=555MiB/s (582MB/s), 555MiB/s-555MiB/s (582MB/s-582MB/s), io=4096MiB (4295MB), run=7378-7378msec
randread-4k: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.26-19-ge7e53
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=5413KiB/s][r=1353 IOPS][eta 00m:00s]
randread-4k: (groupid=0, jobs=1): err= 0: pid=252138: Tue Mar 30 17:40:08 2021
  read: IOPS=976, BW=3907KiB/s (4000kB/s)(229MiB/60001msec)
    clat (usec): min=140, max=69037, avg=1022.02, stdev=2434.67
     lat (usec): min=140, max=69038, avg=1022.24, stdev=2434.70
    clat percentiles (usec):
     |  1.00th=[  145],  5.00th=[  151], 10.00th=[  392], 20.00th=[  404],
     | 30.00th=[  412], 40.00th=[  420], 50.00th=[  429], 60.00th=[  441],
     | 70.00th=[  453], 80.00th=[  478], 90.00th=[  619], 95.00th=[ 6325],
     | 99.00th=[12649], 99.50th=[15008], 99.90th=[19006], 99.95th=[22414],
     | 99.99th=[36439]
   bw (  KiB/s): min=  504, max=25696, per=99.65%, avg=3893.78, stdev=2678.10, samples=119
   iops        : min=  126, max= 6424, avg=973.45, stdev=669.52, samples=119
  lat (usec)   : 250=8.17%, 500=76.27%, 750=7.66%, 1000=0.29%
  lat (msec)   : 2=0.44%, 4=0.68%, 10=4.11%, 20=2.31%, 50=0.07%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.43%, sys=1.45%, ctx=58609, majf=0, minf=52
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=58599,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=3907KiB/s (4000kB/s), 3907KiB/s-3907KiB/s (4000kB/s-4000kB/s), io=229MiB (240MB), run=60001-60001msec
randread-1m: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.26-19-ge7e53
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=198MiB/s][r=198 IOPS][eta 00m:00s]
randread-1m: (groupid=0, jobs=1): err= 0: pid=305984: Tue Mar 30 17:40:24 2021
  read: IOPS=258, BW=258MiB/s (271MB/s)(4096MiB/15866msec)
    clat (usec): min=1554, max=105781, avg=3866.93, stdev=4804.98
     lat (usec): min=1555, max=105782, avg=3867.56, stdev=4804.99
    clat percentiles (usec):
     |  1.00th=[  1598],  5.00th=[  1663], 10.00th=[  1696], 20.00th=[  1745],
     | 30.00th=[  1778], 40.00th=[  1811], 50.00th=[  1827], 60.00th=[  1876],
     | 70.00th=[  2474], 80.00th=[  2737], 90.00th=[ 11731], 95.00th=[ 13698],
     | 99.00th=[ 20055], 99.50th=[ 21890], 99.90th=[ 30278], 99.95th=[ 51643],
     | 99.99th=[105382]
   bw (  KiB/s): min=59392, max=589824, per=100.00%, avg=264456.26, stdev=198469.23, samples=31
   iops        : min=   58, max=  576, avg=258.26, stdev=193.82, samples=31
  lat (msec)   : 2=61.13%, 4=21.36%, 10=4.39%, 20=12.04%, 50=1.00%
  lat (msec)   : 100=0.05%, 250=0.02%
  cpu          : usr=0.27%, sys=4.53%, ctx=4098, majf=0, minf=267
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=258MiB/s (271MB/s), 258MiB/s-258MiB/s (271MB/s-271MB/s), io=4096MiB (4295MB), run=15866-15866msec
seqwrite-4k: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.26-19-ge7e53
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=3524KiB/s][w=881 IOPS][eta 00m:00s]
seqwrite-4k: (groupid=0, jobs=1): err= 0: pid=310158: Tue Mar 30 17:41:24 2021
  write: IOPS=909, BW=3640KiB/s (3727kB/s)(213MiB/60001msec); 0 zone resets
    clat (usec): min=600, max=5167, avg=1094.55, stdev=169.78
     lat (usec): min=600, max=5167, avg=1095.29, stdev=169.84
    clat percentiles (usec):
     |  1.00th=[  857],  5.00th=[  898], 10.00th=[  930], 20.00th=[  979],
     | 30.00th=[ 1012], 40.00th=[ 1057], 50.00th=[ 1090], 60.00th=[ 1123],
     | 70.00th=[ 1156], 80.00th=[ 1205], 90.00th=[ 1254], 95.00th=[ 1287],
     | 99.00th=[ 1385], 99.50th=[ 1549], 99.90th=[ 3589], 99.95th=[ 4047],
     | 99.99th=[ 4228]
   bw (  KiB/s): min= 3264, max= 4128, per=100.00%, avg=3642.35, stdev=256.24, samples=119
   iops        : min=  816, max= 1032, avg=910.59, stdev=64.06, samples=119
  lat (usec)   : 750=0.01%, 1000=26.73%
  lat (msec)   : 2=73.03%, 4=0.18%, 10=0.06%
  cpu          : usr=0.81%, sys=2.50%, ctx=54601, majf=0, minf=16
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,54596,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=3640KiB/s (3727kB/s), 3640KiB/s-3640KiB/s (3727kB/s-3727kB/s), io=213MiB (224MB), run=60001-60001msec
seqwrite-1m: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.26-19-ge7e53
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=309MiB/s][w=309 IOPS][eta 00m:00s]
seqwrite-1m: (groupid=0, jobs=1): err= 0: pid=364925: Tue Mar 30 17:41:38 2021
  write: IOPS=307, BW=308MiB/s (323MB/s)(4096MiB/13300msec); 0 zone resets
    clat (usec): min=2666, max=21136, avg=3202.80, stdev=417.16
     lat (usec): min=2692, max=21176, avg=3241.69, stdev=417.22
    clat percentiles (usec):
     |  1.00th=[ 2802],  5.00th=[ 2900], 10.00th=[ 2933], 20.00th=[ 3032],
     | 30.00th=[ 3130], 40.00th=[ 3163], 50.00th=[ 3195], 60.00th=[ 3228],
     | 70.00th=[ 3261], 80.00th=[ 3294], 90.00th=[ 3326], 95.00th=[ 3392],
     | 99.00th=[ 4621], 99.50th=[ 5276], 99.90th=[ 6587], 99.95th=[ 7635],
     | 99.99th=[21103]
   bw (  KiB/s): min=301056, max=325632, per=100.00%, avg=315367.46, stdev=5919.35, samples=26
   iops        : min=  294, max=  318, avg=307.96, stdev= 5.77, samples=26
  lat (msec)   : 4=98.56%, 10=1.42%, 50=0.02%
  cpu          : usr=2.18%, sys=4.97%, ctx=4100, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4096,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=308MiB/s (323MB/s), 308MiB/s-308MiB/s (323MB/s-323MB/s), io=4096MiB (4295MB), run=13300-13300msec
randwrite-4k: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.26-19-ge7e53
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=3323KiB/s][w=830 IOPS][eta 00m:00s]
randwrite-4k: (groupid=0, jobs=1): err= 0: pid=369095: Tue Mar 30 17:42:38 2021
  write: IOPS=897, BW=3591KiB/s (3677kB/s)(210MiB/60001msec); 0 zone resets
    clat (usec): min=767, max=5143, avg=1108.56, stdev=171.87
     lat (usec): min=767, max=5144, avg=1109.35, stdev=171.91
    clat percentiles (usec):
     |  1.00th=[  824],  5.00th=[  865], 10.00th=[  930], 20.00th=[ 1004],
     | 30.00th=[ 1045], 40.00th=[ 1090], 50.00th=[ 1123], 60.00th=[ 1139],
     | 70.00th=[ 1172], 80.00th=[ 1205], 90.00th=[ 1254], 95.00th=[ 1270],
     | 99.00th=[ 1352], 99.50th=[ 1483], 99.90th=[ 3687], 99.95th=[ 4047],
     | 99.99th=[ 4817]
   bw (  KiB/s): min= 3312, max= 4768, per=100.00%, avg=3594.76, stdev=235.97, samples=119
   iops        : min=  828, max= 1192, avg=898.69, stdev=58.99, samples=119
  lat (usec)   : 1000=19.38%
  lat (msec)   : 2=80.37%, 4=0.19%, 10=0.06%
  cpu          : usr=0.87%, sys=2.06%, ctx=53869, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,53864,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=3591KiB/s (3677kB/s), 3591KiB/s-3591KiB/s (3677kB/s-3677kB/s), io=210MiB (221MB), run=60001-60001msec
randwrite-1m: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.26-19-ge7e53
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=292MiB/s][w=292 IOPS][eta 00m:00s]
randwrite-1m: (groupid=0, jobs=1): err= 0: pid=423131: Tue Mar 30 17:42:52 2021
  write: IOPS=298, BW=299MiB/s (313MB/s)(4096MiB/13719msec); 0 zone resets
    clat (usec): min=2977, max=18485, avg=3304.89, stdev=421.16
     lat (usec): min=3001, max=18508, avg=3343.33, stdev=420.91
    clat percentiles (usec):
     |  1.00th=[ 3032],  5.00th=[ 3097], 10.00th=[ 3130], 20.00th=[ 3163],
     | 30.00th=[ 3195], 40.00th=[ 3228], 50.00th=[ 3228], 60.00th=[ 3261],
     | 70.00th=[ 3294], 80.00th=[ 3359], 90.00th=[ 3523], 95.00th=[ 3621],
     | 99.00th=[ 4555], 99.50th=[ 5669], 99.90th=[ 8455], 99.95th=[10552],
     | 99.99th=[18482]
   bw (  KiB/s): min=286720, max=317440, per=100.00%, avg=305834.67, stdev=7117.18, samples=27
   iops        : min=  280, max=  310, avg=298.67, stdev= 6.95, samples=27
  lat (msec)   : 4=98.49%, 10=1.44%, 20=0.07%
  cpu          : usr=1.33%, sys=5.51%, ctx=4100, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4096,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=299MiB/s (313MB/s), 299MiB/s-299MiB/s (313MB/s-313MB/s), io=4096MiB (4295MB), run=13719-13719msec

cc20test1.naist.jp.txt

keichi@cc20test1:/work/keichi$ ./oci_fio_benchmark.sh
seqread-4k: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.26-19-ge7e53
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=444KiB/s][r=111 IOPS][eta 00m:00s]
seqread-4k: (groupid=0, jobs=1): err= 0: pid=6553: Tue Mar 30 17:30:10 2021
  read: IOPS=111, BW=444KiB/s (455kB/s)(26.0MiB/60008msec)
    clat (usec): min=8810, max=35878, avg=9005.58, stdev=344.92
     lat (usec): min=8810, max=35879, avg=9005.82, stdev=344.93
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 8848], 10.00th=[ 8979], 20.00th=[ 8979],
     | 30.00th=[ 8979], 40.00th=[ 8979], 50.00th=[ 8979], 60.00th=[ 8979],
     | 70.00th=[ 8979], 80.00th=[ 9110], 90.00th=[ 9110], 95.00th=[ 9110],
     | 99.00th=[ 9241], 99.50th=[10028], 99.90th=[10159], 99.95th=[10290],
     | 99.99th=[35914]
   bw (  KiB/s): min=  408, max=  448, per=100.00%, avg=444.03, stdev= 5.19, samples=119
   iops        : min=  102, max=  112, avg=111.01, stdev= 1.30, samples=119
  lat (msec)   : 10=99.55%, 20=0.44%, 50=0.02%
  cpu          : usr=0.03%, sys=0.15%, ctx=9633, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=6661,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=444KiB/s (455kB/s), 444KiB/s-444KiB/s (455kB/s-455kB/s), io=26.0MiB (27.3MB), run=60008-60008msec
seqread-1m: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.26-19-ge7e53
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=55.0MiB/s][r=55 IOPS][eta 00m:00s]
seqread-1m: (groupid=0, jobs=1): err= 0: pid=6557: Tue Mar 30 17:31:10 2021
  read: IOPS=54, BW=54.9MiB/s (57.6MB/s)(3295MiB/60004msec)
    clat (usec): min=17981, max=40352, avg=18205.48, stdev=396.20
     lat (usec): min=17981, max=40353, avg=18205.77, stdev=396.20
    clat percentiles (usec):
     |  1.00th=[17957],  5.00th=[17957], 10.00th=[18220], 20.00th=[18220],
     | 30.00th=[18220], 40.00th=[18220], 50.00th=[18220], 60.00th=[18220],
     | 70.00th=[18220], 80.00th=[18220], 90.00th=[18220], 95.00th=[18220],
     | 99.00th=[18482], 99.50th=[18482], 99.90th=[19006], 99.95th=[20317],
     | 99.99th=[40109]
   bw (  KiB/s): min=51200, max=57344, per=100.00%, avg=56242.55, stdev=1124.58, samples=119
   iops        : min=   50, max=   56, avg=54.92, stdev= 1.10, samples=119
  lat (msec)   : 20=99.91%, 50=0.09%
  cpu          : usr=0.03%, sys=0.28%, ctx=5203, majf=0, minf=266
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3295,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=54.9MiB/s (57.6MB/s), 54.9MiB/s-54.9MiB/s (57.6MB/s-57.6MB/s), io=3295MiB (3455MB), run=60004-60004msec
randread-4k: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.26-19-ge7e53
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=320KiB/s][r=80 IOPS][eta 00m:00s]
randread-4k: (groupid=0, jobs=1): err= 0: pid=6560: Tue Mar 30 17:32:10 2021
  read: IOPS=79, BW=319KiB/s (327kB/s)(18.7MiB/60003msec)
    clat (usec): min=8990, max=62566, avg=12533.00, stdev=5115.62
     lat (usec): min=8990, max=62567, avg=12533.26, stdev=5115.62
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[ 9110], 10.00th=[ 9110], 20.00th=[ 9110],
     | 30.00th=[ 9241], 40.00th=[ 9241], 50.00th=[ 9241], 60.00th=[ 9372],
     | 70.00th=[14222], 80.00th=[17433], 90.00th=[20055], 95.00th=[22676],
     | 99.00th=[26608], 99.50th=[27657], 99.90th=[48497], 99.95th=[53740],
     | 99.99th=[62653]
   bw (  KiB/s): min=  216, max=  392, per=99.67%, avg=318.92, stdev=41.89, samples=119
   iops        : min=   54, max=   98, avg=79.73, stdev=10.47, samples=119
  lat (msec)   : 10=62.54%, 20=26.83%, 50=10.55%, 100=0.08%
  cpu          : usr=0.03%, sys=0.12%, ctx=7500, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4786,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=319KiB/s (327kB/s), 319KiB/s-319KiB/s (327kB/s-327kB/s), io=18.7MiB (19.6MB), run=60003-60003msec
randread-1m: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.26-19-ge7e53
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=45.0MiB/s][r=45 IOPS][eta 00m:00s]
randread-1m: (groupid=0, jobs=1): err= 0: pid=6565: Tue Mar 30 17:33:11 2021
  read: IOPS=41, BW=41.0MiB/s (43.0MB/s)(2462MiB/60013msec)
    clat (usec): min=18681, max=68657, avg=24369.36, stdev=6403.77
     lat (usec): min=18681, max=68658, avg=24369.61, stdev=6403.77
    clat percentiles (usec):
     |  1.00th=[18744],  5.00th=[18744], 10.00th=[18744], 20.00th=[19006],
     | 30.00th=[19006], 40.00th=[19006], 50.00th=[19268], 60.00th=[26346],
     | 70.00th=[28705], 80.00th=[30540], 90.00th=[33817], 95.00th=[35914],
     | 99.00th=[39060], 99.50th=[42730], 99.90th=[46924], 99.95th=[50070],
     | 99.99th=[68682]
   bw (  KiB/s): min=26624, max=49152, per=99.96%, avg=41992.61, stdev=3899.27, samples=119
   iops        : min=   26, max=   48, avg=41.01, stdev= 3.81, samples=119
  lat (msec)   : 20=51.79%, 50=48.17%, 100=0.04%
  cpu          : usr=0.02%, sys=0.19%, ctx=4416, majf=0, minf=268
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2462,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=41.0MiB/s (43.0MB/s), 41.0MiB/s-41.0MiB/s (43.0MB/s-43.0MB/s), io=2462MiB (2582MB), run=60013-60013msec
seqwrite-4k: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.26-19-ge7e53
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=408KiB/s][w=102 IOPS][eta 00m:00s]
seqwrite-4k: (groupid=0, jobs=1): err= 0: pid=6569: Tue Mar 30 17:34:11 2021
  write: IOPS=97, BW=389KiB/s (399kB/s)(22.8MiB/60005msec); 0 zone resets
    clat (usec): min=9590, max=14087, avg=10272.72, stdev=580.12
     lat (usec): min=9590, max=14088, avg=10273.10, stdev=580.12
    clat percentiles (usec):
     |  1.00th=[ 9634],  5.00th=[ 9634], 10.00th=[ 9634], 20.00th=[ 9765],
     | 30.00th=[ 9765], 40.00th=[ 9765], 50.00th=[ 9896], 60.00th=[10683],
     | 70.00th=[10814], 80.00th=[10814], 90.00th=[10945], 95.00th=[10945],
     | 99.00th=[11469], 99.50th=[11994], 99.90th=[13698], 99.95th=[13829],
     | 99.99th=[14091]
   bw (  KiB/s): min=  352, max=  416, per=99.94%, avg=389.11, stdev=20.53, samples=119
   iops        : min=   88, max=  104, avg=97.28, stdev= 5.13, samples=119
  lat (msec)   : 10=51.52%, 20=48.48%
  cpu          : usr=0.03%, sys=0.25%, ctx=7622, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5839,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=389KiB/s (399kB/s), 389KiB/s-389KiB/s (399kB/s-399kB/s), io=22.8MiB (23.9MB), run=60005-60005msec
seqwrite-1m: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.26-19-ge7e53
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=26.0MiB/s][w=26 IOPS][eta 00m:00s]
seqwrite-1m: (groupid=0, jobs=1): err= 0: pid=6572: Tue Mar 30 17:35:11 2021
  write: IOPS=29, BW=29.3MiB/s (30.7MB/s)(1759MiB/60022msec); 0 zone resets
    clat (usec): min=29898, max=67087, avg=34088.25, stdev=3891.65
     lat (usec): min=29925, max=67096, avg=34115.10, stdev=3891.50
    clat percentiles (usec):
     |  1.00th=[30278],  5.00th=[30540], 10.00th=[30802], 20.00th=[31065],
     | 30.00th=[31327], 40.00th=[31589], 50.00th=[31589], 60.00th=[31851],
     | 70.00th=[38011], 80.00th=[38536], 90.00th=[39060], 95.00th=[39584],
     | 99.00th=[39584], 99.50th=[47449], 99.90th=[56886], 99.95th=[66847],
     | 99.99th=[66847]
   bw (  KiB/s): min=18432, max=34816, per=99.98%, avg=30003.20, stdev=3011.45, samples=120
   iops        : min=   18, max=   34, avg=29.30, stdev= 2.94, samples=120
  lat (msec)   : 50=99.72%, 100=0.28%
  cpu          : usr=0.09%, sys=0.16%, ctx=1868, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1759,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=29.3MiB/s (30.7MB/s), 29.3MiB/s-29.3MiB/s (30.7MB/s-30.7MB/s), io=1759MiB (1844MB), run=60022-60022msec
randwrite-4k: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.26-19-ge7e53
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=412KiB/s][w=103 IOPS][eta 00m:00s]
randwrite-4k: (groupid=0, jobs=1): err= 0: pid=6577: Tue Mar 30 17:36:12 2021
  write: IOPS=102, BW=411KiB/s (421kB/s)(24.1MiB/60005msec); 0 zone resets
    clat (usec): min=9543, max=13503, avg=9719.51, stdev=194.07
     lat (usec): min=9544, max=13503, avg=9719.89, stdev=194.08
    clat percentiles (usec):
     |  1.00th=[ 9634],  5.00th=[ 9634], 10.00th=[ 9634], 20.00th=[ 9634],
     | 30.00th=[ 9634], 40.00th=[ 9634], 50.00th=[ 9634], 60.00th=[ 9765],
     | 70.00th=[ 9765], 80.00th=[ 9765], 90.00th=[ 9765], 95.00th=[ 9896],
     | 99.00th=[10683], 99.50th=[10945], 99.90th=[12387], 99.95th=[12649],
     | 99.99th=[13566]
   bw (  KiB/s): min=  400, max=  416, per=99.91%, avg=411.43, stdev= 4.11, samples=119
   iops        : min=  100, max=  104, avg=102.86, stdev= 1.03, samples=119
  lat (msec)   : 10=97.96%, 20=2.04%
  cpu          : usr=0.03%, sys=0.24%, ctx=8457, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,6171,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=411KiB/s (421kB/s), 411KiB/s-411KiB/s (421kB/s-421kB/s), io=24.1MiB (25.3MB), run=60005-60005msec
randwrite-1m: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.26-19-ge7e53
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=32.0MiB/s][w=32 IOPS][eta 00m:00s]
randwrite-1m: (groupid=0, jobs=1): err= 0: pid=6580: Tue Mar 30 17:37:12 2021
  write: IOPS=29, BW=29.4MiB/s (30.8MB/s)(1765MiB/60030msec); 0 zone resets
    clat (usec): min=29978, max=41382, avg=33973.75, stdev=3560.62
     lat (usec): min=30005, max=41416, avg=34001.66, stdev=3560.71
    clat percentiles (usec):
     |  1.00th=[30278],  5.00th=[30540], 10.00th=[30802], 20.00th=[31065],
     | 30.00th=[31327], 40.00th=[31589], 50.00th=[31851], 60.00th=[32113],
     | 70.00th=[38536], 80.00th=[38536], 90.00th=[39060], 95.00th=[39584],
     | 99.00th=[39584], 99.50th=[40109], 99.90th=[40633], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min=24576, max=34816, per=99.99%, avg=30105.60, stdev=2766.85, samples=120
   iops        : min=   24, max=   34, avg=29.40, stdev= 2.70, samples=120
  lat (msec)   : 50=100.00%
  cpu          : usr=0.09%, sys=0.19%, ctx=1870, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1765,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=29.4MiB/s (30.8MB/s), 29.4MiB/s-29.4MiB/s (30.8MB/s-30.8MB/s), io=1765MiB (1851MB), run=60030-60030msec