Skip to content

Instantly share code, notes, and snippets.

@niwinz
Last active November 2, 2023 15:32
Show Gist options
  • Save niwinz/138c9ce49f484b3b66ee5f061d7f999b to your computer and use it in GitHub Desktop.
Save niwinz/138c9ce49f484b3b66ee5f061d7f999b to your computer and use it in GitHub Desktop.
EBS Benchmark (gp3 vs gp2, and st1 just for comparison).

EBS Volume Type Benchmark

Tested volumes:

  • /dev/nvme1n1 (type: st1, size: 125gb, defaults)
  • /dev/nvme2n1 (type: gp3, size: 15gb, iops:3000, tp: 125MB/s)
  • /dev/nvme3n1 (type: gp2, size: 15gb, defaults (iops:100 burstable until 3000))

All volumes created from scrach, no gp2->gp3 conversion.

EC2 instance: c6g.medium (ARM64)

IOPS RANDREAD

Command:

fio --filename=device --direct=1 --rw=randread --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --readonly

GP3

iops-test-job: (groupid=0, jobs=4): err= 0: pid=4661: Tue Dec  8 10:49:36 2020
  read: IOPS=3024, BW=11.8MiB/s (12.4MB/s)(1418MiB/120022msec)
    slat (nsec): min=1583, max=28615k, avg=1320862.80, stdev=3295638.45
    clat (msec): min=11, max=456, avg=337.16, stdev=32.67
     lat (msec): min=15, max=456, avg=338.48, stdev=32.93
    clat percentiles (msec):
     |  1.00th=[  239],  5.00th=[  313], 10.00th=[  317], 20.00th=[  326],
     | 30.00th=[  330], 40.00th=[  338], 50.00th=[  338], 60.00th=[  347],
     | 70.00th=[  347], 80.00th=[  355], 90.00th=[  363], 95.00th=[  368],
     | 99.00th=[  384], 99.50th=[  388], 99.90th=[  405], 99.95th=[  414],
     | 99.99th=[  426]
   bw (  KiB/s): min= 2512, max= 7656, per=24.93%, avg=3016.63, stdev=289.06, samples=960
   iops        : min=  628, max= 1914, avg=754.14, stdev=72.27, samples=960
  lat (msec)   : 20=0.02%, 50=0.66%, 100=0.09%, 250=0.26%, 500=98.98%
  cpu          : usr=0.08%, sys=0.21%, ctx=64783, majf=0, minf=1061
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=363064,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=11.8MiB/s (12.4MB/s), 11.8MiB/s-11.8MiB/s (12.4MB/s-12.4MB/s), io=1418MiB (1487MB), run=120022-120022msec

Disk stats (read/write):
  nvme2n1: ios=362690/0, merge=8/0, ticks=7101384/0, in_queue=3716744, util=99.97%

GP2

iops-test-job: (groupid=0, jobs=4): err= 0: pid=4814: Tue Dec  8 10:53:21 2020
  read: IOPS=3024, BW=11.8MiB/s (12.4MB/s)(1418MiB/120021msec)
    slat (nsec): min=1551, max=10056k, avg=1321031.72, stdev=3245633.28
    clat (msec): min=12, max=429, avg=337.15, stdev=28.93
     lat (msec): min=21, max=429, avg=338.47, stdev=29.22
    clat percentiles (msec):
     |  1.00th=[  234],  5.00th=[  338], 10.00th=[  338], 20.00th=[  338],
     | 30.00th=[  338], 40.00th=[  338], 50.00th=[  338], 60.00th=[  338],
     | 70.00th=[  347], 80.00th=[  347], 90.00th=[  347], 95.00th=[  347],
     | 99.00th=[  363], 99.50th=[  376], 99.90th=[  384], 99.95th=[  393],
     | 99.99th=[  401]
   bw (  KiB/s): min= 2960, max= 7424, per=24.93%, avg=3016.29, stdev=265.77, samples=960
   iops        : min=  740, max= 1856, avg=754.04, stdev=66.45, samples=960
  lat (msec)   : 20=0.01%, 50=0.68%, 100=0.09%, 250=0.26%, 500=98.97%
  cpu          : usr=0.13%, sys=0.13%, ctx=52170, majf=0, minf=1064
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=363063,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=11.8MiB/s (12.4MB/s), 11.8MiB/s-11.8MiB/s (12.4MB/s-12.4MB/s), io=1418MiB (1487MB), run=120021-120021msec

Disk stats (read/write):
  nvme3n1: ios=362693/0, merge=5/0, ticks=7067684/0, in_queue=3716776, util=99.96%

ST1

iops-test-job: (groupid=0, jobs=4): err= 0: pid=4821: Tue Dec  8 10:56:57 2020
  read: IOPS=33, BW=133KiB/s (136kB/s)(15.8MiB/121878msec)
    slat (nsec): min=1673, max=862315k, avg=118776383.50, stdev=294236412.63
    clat (msec): min=1090, max=33871, avg=27379.07, stdev=7467.12
     lat (msec): min=1939, max=33940, avg=27497.85, stdev=7470.28
    clat percentiles (msec):
     |  1.00th=[ 3037],  5.00th=[ 7886], 10.00th=[14026], 20.00th=[17113],
     | 30.00th=[17113], 40.00th=[17113], 50.00th=[17113], 60.00th=[17113],
     | 70.00th=[17113], 80.00th=[17113], 90.00th=[17113], 95.00th=[17113],
     | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113],
     | 99.99th=[17113]
   bw (  KiB/s): min=    8, max=   56, per=41.64%, avg=55.38, stdev= 4.85, samples=438
   iops        : min=    2, max=   14, avg=13.81, stdev= 1.23, samples=438
  cpu          : usr=0.00%, sys=0.00%, ctx=620, majf=0, minf=1061
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=3.2%, >=64=93.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=4055,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=133KiB/s (136kB/s), 133KiB/s-133KiB/s (136kB/s-136kB/s), io=15.8MiB (16.6MB), run=121878-121878msec

Disk stats (read/write):
  nvme1n1: ios=4047/0, merge=0/0, ticks=7122841/0, in_queue=3763204, util=99.98%

IOPS SEQREAD

Command:

fio --filename=device --direct=1 --rw=read --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --readonly

GP3

iops-test-job: (groupid=0, jobs=4): err= 0: pid=4829: Tue Dec  8 11:00:48 2020
  read: IOPS=32.3k, BW=126MiB/s (132MB/s)(14.8GiB/120032msec)
    slat (nsec): min=886, max=1143.3k, avg=1122.71, stdev=2157.57
    clat (usec): min=55, max=61412, avg=31682.93, stdev=3278.13
     lat (usec): min=208, max=61414, avg=31684.12, stdev=3278.02
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[28967], 10.00th=[29754], 20.00th=[30540],
     | 30.00th=[31065], 40.00th=[31589], 50.00th=[31851], 60.00th=[32113],
     | 70.00th=[32637], 80.00th=[33424], 90.00th=[34341], 95.00th=[34866],
     | 99.00th=[36439], 99.50th=[36963], 99.90th=[38011], 99.95th=[39060],
     | 99.99th=[46400]
   bw (  KiB/s): min=30760, max=100344, per=25.00%, avg=32313.87, stdev=4162.99, samples=960
   iops        : min= 7690, max=25086, avg=8078.44, stdev=1040.75, samples=960
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.31%, 4=0.46%, 10=0.13%, 20=0.01%, 50=99.08%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.60%, sys=0.92%, ctx=133721, majf=0, minf=1066
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=3879094,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=14.8GiB (15.9GB), run=120032-120032msec

Disk stats (read/write):
  nvme2n1: ios=127820/0, merge=3747085/0, ticks=3998439/0, in_queue=3716864, util=99.98%

GP2

iops-test-job: (groupid=0, jobs=4): err= 0: pid=4835: Tue Dec  8 11:03:33 2020
  read: IOPS=33.1k, BW=129MiB/s (136MB/s)(15.1GiB/120031msec)
    slat (nsec): min=894, max=932759, avg=1100.08, stdev=2211.40
    clat (usec): min=1132, max=130793, avg=30951.44, stdev=3393.65
     lat (usec): min=1143, max=130794, avg=30952.61, stdev=3393.52
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[   32], 10.00th=[   32], 20.00th=[   32],
     | 30.00th=[   32], 40.00th=[   32], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   32], 80.00th=[   32], 90.00th=[   32], 95.00th=[   32],
     | 99.00th=[   32], 99.50th=[   32], 99.90th=[   32], 99.95th=[   91],
     | 99.99th=[  123]
   bw (  KiB/s): min=31776, max=103656, per=25.00%, avg=33078.53, stdev=4248.62, samples=960
   iops        : min= 7944, max=25914, avg=8269.61, stdev=1062.16, samples=960
  lat (msec)   : 2=0.02%, 4=0.99%, 10=0.02%, 20=0.02%, 50=98.88%
  lat (msec)   : 100=0.03%, 250=0.03%
  cpu          : usr=0.58%, sys=0.96%, ctx=127979, majf=0, minf=1069
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=3970844,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=129MiB/s (136MB/s), 129MiB/s-129MiB/s (136MB/s-136MB/s), io=15.1GiB (16.3GB), run=120031-120031msec

Disk stats (read/write):
  nvme3n1: ios=128104/0, merge=3839219/0, ticks=3955260/0, in_queue=3716512, util=99.98%

ST1

iops-test-job: (groupid=0, jobs=4): err= 0: pid=4842: Tue Dec  8 11:06:10 2020
  read: IOPS=4667, BW=18.2MiB/s (19.1MB/s)(2191MiB/120155msec)
    slat (nsec): min=878, max=1029.5k, avg=1156.79, stdev=2561.85
    clat (usec): min=632, max=727669, avg=219374.82, stdev=116999.26
     lat (usec): min=634, max=727670, avg=219376.05, stdev=116999.24
    clat percentiles (msec):
     |  1.00th=[  108],  5.00th=[  115], 10.00th=[  118], 20.00th=[  122],
     | 30.00th=[  126], 40.00th=[  146], 50.00th=[  178], 60.00th=[  213],
     | 70.00th=[  266], 80.00th=[  326], 90.00th=[  397], 95.00th=[  456],
     | 99.00th=[  550], 99.50th=[  575], 99.90th=[  651], 99.95th=[  667],
     | 99.99th=[  726]
   bw (  KiB/s): min=  824, max= 8176, per=24.99%, avg=4664.47, stdev=1583.94, samples=960
   iops        : min=  206, max= 2044, avg=1166.09, stdev=395.99, samples=960
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.06%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.34%, 250=68.27%, 500=28.47%, 750=2.82%
  cpu          : usr=0.08%, sys=0.15%, ctx=20417, majf=0, minf=1071
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=560810,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=18.2MiB/s (19.1MB/s), 18.2MiB/s-18.2MiB/s (19.1MB/s-19.1MB/s), io=2191MiB (2297MB), run=120155-120155msec

Disk stats (read/write):
  nvme1n1: ios=18631/0, merge=541660/0, ticks=4024160/0, in_queue=3721936, util=99.98%

IOPS RANDRW

Command:

fio --filename=device --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 

GP3

iops-test-job: (groupid=0, jobs=4): err= 0: pid=4973: Tue Dec  8 12:05:32 2020
  read: IOPS=1512, BW=6051KiB/s (6196kB/s)(709MiB/120023msec)
    slat (nsec): min=1493, max=38620k, avg=1332134.10, stdev=3313125.49
    clat (msec): min=13, max=391, avg=332.73, stdev=28.43
     lat (msec): min=21, max=391, avg=334.06, stdev=28.70
    clat percentiles (msec):
     |  1.00th=[  236],  5.00th=[  321], 10.00th=[  326], 20.00th=[  326],
     | 30.00th=[  334], 40.00th=[  334], 50.00th=[  338], 60.00th=[  338],
     | 70.00th=[  338], 80.00th=[  342], 90.00th=[  347], 95.00th=[  347],
     | 99.00th=[  355], 99.50th=[  363], 99.90th=[  372], 99.95th=[  372],
     | 99.99th=[  380]
   bw (  KiB/s): min= 1224, max= 3560, per=24.94%, avg=1508.72, stdev=152.24, samples=960
   iops        : min=  306, max=  890, avg=377.17, stdev=38.06, samples=960
  write: IOPS=1512, BW=6049KiB/s (6194kB/s)(709MiB/120023msec); 0 zone resets
    slat (nsec): min=1551, max=37140k, avg=1309680.21, stdev=3287985.96
    clat (msec): min=12, max=424, avg=341.56, stdev=31.59
     lat (msec): min=21, max=433, avg=342.87, stdev=31.85
    clat percentiles (msec):
     |  1.00th=[  213],  5.00th=[  326], 10.00th=[  326], 20.00th=[  334],
     | 30.00th=[  338], 40.00th=[  338], 50.00th=[  347], 60.00th=[  347],
     | 70.00th=[  351], 80.00th=[  355], 90.00th=[  363], 95.00th=[  368],
     | 99.00th=[  384], 99.50th=[  384], 99.90th=[  397], 99.95th=[  401],
     | 99.99th=[  414]
   bw (  KiB/s): min= 1264, max= 3752, per=24.93%, avg=1508.03, stdev=159.78, samples=960
   iops        : min=  316, max=  938, avg=377.00, stdev=39.95, samples=960
  lat (msec)   : 20=0.01%, 50=0.70%, 100=0.09%, 250=0.26%, 500=98.95%
  cpu          : usr=0.10%, sys=0.19%, ctx=56486, majf=0, minf=43
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=181557,181511,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=6051KiB/s (6196kB/s), 6051KiB/s-6051KiB/s (6196kB/s-6196kB/s), io=709MiB (744MB), run=120023-120023msec
  WRITE: bw=6049KiB/s (6194kB/s), 6049KiB/s-6049KiB/s (6194kB/s-6194kB/s), io=709MiB (743MB), run=120023-120023msec

Disk stats (read/write):
  nvme2n1: ios=181429/181305, merge=0/4, ticks=2703582/4375786, in_queue=3716896, util=100.00%

GP2


iops-test-job: (groupid=0, jobs=4): err= 0: pid=4981: Tue Dec  8 12:07:59 2020
  read: IOPS=1512, BW=6051KiB/s (6196kB/s)(709MiB/120023msec)
    slat (nsec): min=1461, max=24547k, avg=1326464.65, stdev=3252327.87
    clat (msec): min=17, max=382, avg=332.96, stdev=27.30
     lat (msec): min=21, max=382, avg=334.28, stdev=27.58
    clat percentiles (msec):
     |  1.00th=[  224],  5.00th=[  326], 10.00th=[  326], 20.00th=[  338],
     | 30.00th=[  338], 40.00th=[  338], 50.00th=[  338], 60.00th=[  338],
     | 70.00th=[  338], 80.00th=[  338], 90.00th=[  338], 95.00th=[  347],
     | 99.00th=[  347], 99.50th=[  355], 99.90th=[  355], 99.95th=[  355],
     | 99.99th=[  363]
   bw (  KiB/s): min= 1248, max= 3648, per=24.94%, avg=1508.61, stdev=150.10, samples=960
   iops        : min=  312, max=  912, avg=377.15, stdev=37.53, samples=960
  write: IOPS=1512, BW=6049KiB/s (6194kB/s)(709MiB/120023msec); 0 zone resets
    slat (nsec): min=1501, max=25373k, avg=1315415.85, stdev=3239952.20
    clat (msec): min=12, max=438, avg=341.32, stdev=30.52
     lat (msec): min=21, max=438, avg=342.63, stdev=30.78
    clat percentiles (msec):
     |  1.00th=[  201],  5.00th=[  326], 10.00th=[  338], 20.00th=[  338],
     | 30.00th=[  338], 40.00th=[  338], 50.00th=[  347], 60.00th=[  347],
     | 70.00th=[  347], 80.00th=[  355], 90.00th=[  363], 95.00th=[  363],
     | 99.00th=[  376], 99.50th=[  384], 99.90th=[  393], 99.95th=[  393],
     | 99.99th=[  405]
   bw (  KiB/s): min= 1280, max= 3776, per=24.93%, avg=1508.07, stdev=157.66, samples=960
   iops        : min=  320, max=  944, avg=377.01, stdev=39.42, samples=960
  lat (msec)   : 20=0.01%, 50=0.71%, 100=0.09%, 250=0.26%, 500=98.93%
  cpu          : usr=0.09%, sys=0.19%, ctx=52286, majf=0, minf=41
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=181555,181512,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=6051KiB/s (6196kB/s), 6051KiB/s-6051KiB/s (6196kB/s-6196kB/s), io=709MiB (744MB), run=120023-120023msec
  WRITE: bw=6049KiB/s (6194kB/s), 6049KiB/s-6049KiB/s (6194kB/s-6194kB/s), io=709MiB (743MB), run=120023-120023msec

Disk stats (read/write):
  nvme3n1: ios=181420/181317, merge=0/2, ticks=2687327/4382966, in_queue=3716784, util=99.98%

ST1

iops-test-job: (groupid=0, jobs=4): err= 0: pid=4989: Tue Dec  8 12:10:31 2020
  read: IOPS=15, BW=63.0KiB/s (65.5kB/s)(7800KiB/121908msec)
    slat (nsec): min=1583, max=1484.3M, avg=118236233.82, stdev=294966037.99
    clat (msec): min=1088, max=33091, avg=27159.57, stdev=7317.49
     lat (msec): min=1939, max=33091, avg=27277.81, stdev=7323.22
    clat percentiles (msec):
     |  1.00th=[ 2802],  5.00th=[ 7752], 10.00th=[14563], 20.00th=[17113],
     | 30.00th=[17113], 40.00th=[17113], 50.00th=[17113], 60.00th=[17113],
     | 70.00th=[17113], 80.00th=[17113], 90.00th=[17113], 95.00th=[17113],
     | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113],
     | 99.99th=[17113]
   bw (  KiB/s): min=    8, max=   56, per=44.64%, avg=28.12, stdev=10.94, samples=418
   iops        : min=    2, max=   14, avg= 7.00, stdev= 2.74, samples=418
  write: IOPS=16, BW=67.8KiB/s (69.4kB/s)(8264KiB/121908msec); 0 zone resets
    slat (usec): min=2, max=860490, avg=121581.24, stdev=296881.20
    clat (msec): min=1545, max=35727, avg=27575.66, stdev=7685.45
     lat (msec): min=1939, max=35727, avg=27697.24, stdev=7685.57
    clat percentiles (msec):
     |  1.00th=[ 2970],  5.00th=[ 7886], 10.00th=[13489], 20.00th=[17113],
     | 30.00th=[17113], 40.00th=[17113], 50.00th=[17113], 60.00th=[17113],
     | 70.00th=[17113], 80.00th=[17113], 90.00th=[17113], 95.00th=[17113],
     | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113],
     | 99.99th=[17113]
   bw (  KiB/s): min=    8, max=   56, per=42.75%, avg=28.64, stdev=11.52, samples=425
   iops        : min=    2, max=   14, avg= 7.12, stdev= 2.90, samples=425
  cpu          : usr=0.00%, sys=0.00%, ctx=626, majf=0, minf=41
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=3.2%, >=64=93.7%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=1950,2066,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=63.0KiB/s (65.5kB/s), 63.0KiB/s-63.0KiB/s (65.5kB/s-65.5kB/s), io=7800KiB (7987kB), run=121908-121908msec
  WRITE: bw=67.8KiB/s (69.4kB/s), 67.8KiB/s-67.8KiB/s (69.4kB/s-69.4kB/s), io=8264KiB (8462kB), run=121908-121908msec

Disk stats (read/write):
  nvme1n1: ios=2001/2065, merge=0/0, ticks=2678905/4457067, in_queue=3765564, util=100.00%

THROUGHPUT RANDREAD

Command:

fio --filename=device --direct=1 --rw=randread --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --readonly

GP3

throughput-test-job: (groupid=0, jobs=4): err= 0: pid=4848: Tue Dec  8 11:11:19 2020
  read: IOPS=2019, BW=126MiB/s (132MB/s)(14.8GiB/120032msec)
    slat (nsec): min=1698, max=39413k, avg=1978836.02, stdev=4887117.00
    clat (usec): min=1538, max=238269, avg=124756.76, stdev=15294.44
     lat (usec): min=1540, max=248438, avg=126735.70, stdev=16157.69
    clat percentiles (msec):
     |  1.00th=[   88],  5.00th=[  111], 10.00th=[  112], 20.00th=[  113],
     | 30.00th=[  125], 40.00th=[  126], 50.00th=[  126], 60.00th=[  127],
     | 70.00th=[  127], 80.00th=[  129], 90.00th=[  140], 95.00th=[  142],
     | 99.00th=[  167], 99.50th=[  169], 99.90th=[  197], 99.95th=[  199],
     | 99.99th=[  218]
   bw (  KiB/s): min=29952, max=97280, per=24.98%, avg=32290.26, stdev=3782.80, samples=960
   iops        : min=  468, max= 1520, avg=504.53, stdev=59.11, samples=960
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.02%, 20=0.80%, 50=0.08%
  lat (msec)   : 100=0.29%, 250=98.80%
  cpu          : usr=0.05%, sys=0.17%, ctx=37717, majf=0, minf=2085
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=242441,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=14.8GiB (15.9GB), run=120032-120032msec

Disk stats (read/write):
  nvme2n1: ios=242102/0, merge=67/0, ticks=7077244/0, in_queue=3716776, util=99.96%

GP2

throughput-test-job: (groupid=0, jobs=4): err= 0: pid=4868: Tue Dec  8 11:15:42 2020
  read: IOPS=2067, BW=129MiB/s (135MB/s)(15.1GiB/120031msec)
    slat (nsec): min=1773, max=56015k, avg=1933148.44, stdev=4753362.85
    clat (msec): min=6, max=232, avg=121.88, stdev=14.60
     lat (msec): min=6, max=245, avg=123.81, stdev=15.45
    clat percentiles (msec):
     |  1.00th=[   93],  5.00th=[  110], 10.00th=[  110], 20.00th=[  110],
     | 30.00th=[  124], 40.00th=[  124], 50.00th=[  124], 60.00th=[  124],
     | 70.00th=[  124], 80.00th=[  124], 90.00th=[  138], 95.00th=[  138],
     | 99.00th=[  165], 99.50th=[  165], 99.90th=[  192], 99.95th=[  192],
     | 99.99th=[  220]
   bw (  KiB/s): min=32191, max=97920, per=24.98%, avg=33050.74, stdev=3866.76, samples=960
   iops        : min=  502, max= 1530, avg=516.36, stdev=60.42, samples=960
  lat (msec)   : 10=0.02%, 20=0.84%, 50=0.05%, 100=0.11%, 250=98.98%
  cpu          : usr=0.05%, sys=0.17%, ctx=35666, majf=0, minf=2093
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=248169,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=15.1GiB (16.3GB), run=120031-120031msec

Disk stats (read/write):
  nvme3n1: ios=247840/0, merge=58/0, ticks=7068356/0, in_queue=3716776, util=99.97%

ST1

throughput-test-job: (groupid=0, jobs=4): err= 0: pid=4874: Tue Dec  8 11:18:05 2020
  read: IOPS=33, BW=2129KiB/s (2180kB/s)(253MiB/121857msec)
    slat (nsec): min=1854, max=862349k, avg=118777048.51, stdev=294241889.92
    clat (msec): min=1119, max=10195, avg=7438.29, stdev=1105.93
     lat (msec): min=1708, max=11017, avg=7557.07, stdev=1144.82
    clat percentiles (msec):
     |  1.00th=[ 2567],  5.00th=[ 5738], 10.00th=[ 6812], 20.00th=[ 6812],
     | 30.00th=[ 6812], 40.00th=[ 7617], 50.00th=[ 7617], 60.00th=[ 7617],
     | 70.00th=[ 7617], 80.00th=[ 8490], 90.00th=[ 8490], 95.00th=[ 8490],
     | 99.00th=[ 9329], 99.50th=[ 9329], 99.90th=[10134], 99.95th=[10134],
     | 99.99th=[10134]
   bw (  KiB/s): min=  128, max=  896, per=41.79%, avg=889.62, stdev=66.58, samples=547
   iops        : min=    2, max=   14, avg=13.87, stdev= 1.06, samples=547
  cpu          : usr=0.00%, sys=0.00%, ctx=635, majf=0, minf=2089
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=3.2%, >=64=93.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=4054,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=2129KiB/s (2180kB/s), 2129KiB/s-2129KiB/s (2180kB/s-2180kB/s), io=253MiB (266MB), run=121857-121857msec

Disk stats (read/write):
  nvme1n1: ios=4047/0, merge=0/0, ticks=7122982/0, in_queue=3762548, util=99.97%

THROUGHPUT RANDRW

Command:

fio --filename=device --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 

GP3

throughput-test-job: (groupid=0, jobs=4): err= 0: pid=4926: Tue Dec  8 11:52:46 2020
  read: IOPS=1006, BW=62.9MiB/s (65.0MB/s)(7555MiB/120046msec)
    slat (nsec): min=1821, max=39453k, avg=1986510.31, stdev=4889550.93
    clat (msec): min=3, max=616, avg=118.01, stdev=12.81
     lat (msec): min=3, max=616, avg=120.00, stdev=13.73
    clat percentiles (msec):
     |  1.00th=[   86],  5.00th=[  111], 10.00th=[  111], 20.00th=[  112],
     | 30.00th=[  112], 40.00th=[  113], 50.00th=[  114], 60.00th=[  126],
     | 70.00th=[  126], 80.00th=[  127], 90.00th=[  127], 95.00th=[  128],
     | 99.00th=[  140], 99.50th=[  142], 99.90th=[  155], 99.95th=[  155],
     | 99.99th=[  169]
   bw (  KiB/s): min=12544, max=46336, per=24.98%, avg=16099.06, stdev=2081.48, samples=960
   iops        : min=  196, max=  724, avg=251.53, stdev=32.52, samples=960
  write: IOPS=1012, BW=63.3MiB/s (66.4MB/s)(7598MiB/120046msec); 0 zone resets
    slat (usec): min=2, max=39904, avg=1971.03, stdev=4875.74
    clat (msec): min=2, max=237, avg=131.47, stdev=20.49
     lat (msec): min=2, max=249, avg=133.44, stdev=21.13
    clat percentiles (msec):
     |  1.00th=[   80],  5.00th=[  111], 10.00th=[  112], 20.00th=[  114],
     | 30.00th=[  126], 40.00th=[  126], 50.00th=[  127], 60.00th=[  130],
     | 70.00th=[  140], 80.00th=[  142], 90.00th=[  155], 95.00th=[  167],
     | 99.00th=[  182], 99.50th=[  186], 99.90th=[  199], 99.95th=[  209],
     | 99.99th=[  224]
   bw (  KiB/s): min=12672, max=52352, per=24.98%, avg=16190.79, stdev=2205.70, samples=960
   iops        : min=  198, max=  818, avg=252.96, stdev=34.46, samples=960
  lat (msec)   : 4=0.01%, 10=0.01%, 20=0.63%, 50=0.28%, 100=0.61%
  lat (msec)   : 250=98.46%, 750=0.01%
  cpu          : usr=0.09%, sys=0.17%, ctx=37373, majf=0, minf=45
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=120876,121574,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=62.9MiB/s (65.0MB/s), 62.9MiB/s-62.9MiB/s (65.0MB/s-65.0MB/s), io=7555MiB (7922MB), run=120046-120046msec
  WRITE: bw=63.3MiB/s (66.4MB/s), 63.3MiB/s-63.3MiB/s (66.4MB/s-66.4MB/s), io=7598MiB (7967MB), run=120046-120046msec

Disk stats (read/write):
  nvme2n1: ios=120775/121406, merge=9/21, ticks=2697966/4378140, in_queue=3716808, util=99.99

GP2

throughput-test-job: (groupid=0, jobs=4): err= 0: pid=4933: Tue Dec  8 11:55:14 2020
  read: IOPS=1031, BW=64.4MiB/s (67.6MB/s)(7735MiB/120038msec)
    slat (nsec): min=1838, max=37381k, avg=1908799.09, stdev=4725123.56
    clat (msec): min=9, max=164, avg=115.24, stdev=11.85
     lat (msec): min=9, max=177, avg=117.15, stdev=12.75
    clat percentiles (msec):
     |  1.00th=[   49],  5.00th=[  109], 10.00th=[  109], 20.00th=[  110],
     | 30.00th=[  110], 40.00th=[  110], 50.00th=[  111], 60.00th=[  123],
     | 70.00th=[  124], 80.00th=[  124], 90.00th=[  124], 95.00th=[  124],
     | 99.00th=[  138], 99.50th=[  138], 99.90th=[  150], 99.95th=[  150],
     | 99.99th=[  163]
   bw (  KiB/s): min=12416, max=46464, per=24.98%, avg=16482.66, stdev=2140.85, samples=960
   iops        : min=  194, max=  726, avg=257.52, stdev=33.45, samples=960
  write: IOPS=1036, BW=64.8MiB/s (67.9MB/s)(7776MiB/120038msec); 0 zone resets
    slat (usec): min=2, max=33424, avg=1957.05, stdev=4774.57
    clat (msec): min=12, max=232, avg=128.47, stdev=19.62
     lat (msec): min=12, max=232, avg=130.42, stdev=20.26
    clat percentiles (msec):
     |  1.00th=[   39],  5.00th=[  109], 10.00th=[  110], 20.00th=[  111],
     | 30.00th=[  123], 40.00th=[  124], 50.00th=[  124], 60.00th=[  125],
     | 70.00th=[  138], 80.00th=[  138], 90.00th=[  150], 95.00th=[  165],
     | 99.00th=[  178], 99.50th=[  180], 99.90th=[  192], 99.95th=[  205],
     | 99.99th=[  220]
   bw (  KiB/s): min=13568, max=49920, per=24.98%, avg=16571.29, stdev=2250.00, samples=960
   iops        : min=  212, max=  780, avg=258.91, stdev=35.16, samples=960
  lat (msec)   : 10=0.01%, 20=0.03%, 50=1.00%, 100=0.25%, 250=98.72%
  cpu          : usr=0.11%, sys=0.16%, ctx=36183, majf=0, minf=43
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=123765,124418,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=64.4MiB/s (67.6MB/s), 64.4MiB/s-64.4MiB/s (67.6MB/s-67.6MB/s), io=7735MiB (8111MB), run=120038-120038msec
  WRITE: bw=64.8MiB/s (67.9MB/s), 64.8MiB/s-64.8MiB/s (67.9MB/s-67.9MB/s), io=7776MiB (8154MB), run=120038-120038msec

Disk stats (read/write):
  nvme3n1: ios=123660/124258, merge=8/15, ticks=2661447/4411715, in_queue=3716812, util=99.98%

ST1

throughput-test-job: (groupid=0, jobs=4): err= 0: pid=4941: Tue Dec  8 11:57:54 2020
  read: IOPS=16, BW=1025KiB/s (1050kB/s)(122MiB/121995msec)
    slat (nsec): min=1862, max=1490.1M, avg=118153243.49, stdev=294777046.18
    clat (msec): min=1459, max=9424, avg=7031.26, stdev=896.27
     lat (msec): min=1970, max=9515, avg=7149.41, stdev=933.58
    clat percentiles (msec):
     |  1.00th=[ 2769],  5.00th=[ 5940], 10.00th=[ 6745], 20.00th=[ 6812],
     | 30.00th=[ 6812], 40.00th=[ 6812], 50.00th=[ 6812], 60.00th=[ 7617],
     | 70.00th=[ 7617], 80.00th=[ 7617], 90.00th=[ 7617], 95.00th=[ 7617],
     | 99.00th=[ 8490], 99.50th=[ 8557], 99.90th=[ 9329], 99.95th=[ 9463],
     | 99.99th=[ 9463]
   bw (  KiB/s): min=  127, max= 1024, per=43.93%, avg=450.27, stdev=199.23, samples=523
   iops        : min=    1, max=   16, avg= 6.99, stdev= 3.13, samples=523
  write: IOPS=16, BW=1086KiB/s (1112kB/s)(129MiB/121995msec); 0 zone resets
    slat (usec): min=3, max=2693.6k, avg=121362.88, stdev=301987.09
    clat (msec): min=1120, max=12701, avg=7861.34, stdev=1320.27
     lat (msec): min=1970, max=12701, avg=7982.70, stdev=1342.85
    clat percentiles (msec):
     |  1.00th=[ 2836],  5.00th=[ 6342], 10.00th=[ 6812], 20.00th=[ 6812],
     | 30.00th=[ 7617], 40.00th=[ 7617], 50.00th=[ 7617], 60.00th=[ 7617],
     | 70.00th=[ 8490], 80.00th=[ 8490], 90.00th=[ 9329], 95.00th=[10134],
     | 99.00th=[10939], 99.50th=[11073], 99.90th=[11879], 99.95th=[11879],
     | 99.99th=[12684]
   bw (  KiB/s): min=  127, max=  896, per=44.08%, avg=478.25, stdev=190.90, samples=517
   iops        : min=    1, max=   14, avg= 7.43, stdev= 2.98, samples=517
  cpu          : usr=0.01%, sys=0.00%, ctx=631, majf=0, minf=41
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=3.2%, >=64=93.7%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=1954,2070,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1025KiB/s (1050kB/s), 1025KiB/s-1025KiB/s (1050kB/s-1050kB/s), io=122MiB (128MB), run=121995-121995msec
  WRITE: bw=1086KiB/s (1112kB/s), 1086KiB/s-1086KiB/s (1112kB/s-1112kB/s), io=129MiB (136MB), run=121995-121995msec

Disk stats (read/write):
  nvme1n1: ios=2003/2067, merge=0/0, ticks=2621725/4511318, in_queue=3767852, util=100.00%

LATENCY RANDREAD

fio --filename=device --direct=1 --rw=randread --bs=4k --ioengine=libaio --iodepth=1 --numjobs=1 --time_based --group_reporting --name=readlatency-test-job --runtime=120 --eta-newline=1

GP3

readlatency-test-job: (groupid=0, jobs=1): err= 0: pid=5002: Tue Dec  8 12:14:58 2020
  read: IOPS=1511, BW=6046KiB/s (6191kB/s)(708MiB/120002msec)
    slat (nsec): min=2191, max=26814, avg=2417.91, stdev=412.79
    clat (usec): min=137, max=6898, avg=658.42, stdev=880.73
     lat (usec): min=139, max=6901, avg=660.94, stdev=880.71
    clat percentiles (usec):
     |  1.00th=[  161],  5.00th=[  167], 10.00th=[  172], 20.00th=[  182],
     | 30.00th=[  198], 40.00th=[  281], 50.00th=[  322], 60.00th=[  343],
     | 70.00th=[  392], 80.00th=[  898], 90.00th=[ 1827], 95.00th=[ 2769],
     | 99.00th=[ 4424], 99.50th=[ 4752], 99.90th=[ 4948], 99.95th=[ 4948],
     | 99.99th=[ 5014]
   bw (  KiB/s): min= 5184, max= 6968, per=100.00%, avg=6045.35, stdev=321.84, samples=240
   iops        : min= 1296, max= 1742, avg=1511.33, stdev=80.46, samples=240
  lat (usec)   : 250=37.55%, 500=36.35%, 750=3.72%, 1000=4.03%
  lat (msec)   : 2=9.48%, 4=7.21%, 10=1.66%
  cpu          : usr=0.33%, sys=0.45%, ctx=181453, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=181372,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=6046KiB/s (6191kB/s), 6046KiB/s-6046KiB/s (6191kB/s-6191kB/s), io=708MiB (743MB), run=120002-120002msec

Disk stats (read/write):
  nvme2n1: ios=181227/0, merge=0/0, ticks=119280/0, in_queue=119032, util=99.23%

GP2

readlatency-test-job: (groupid=0, jobs=1): err= 0: pid=5008: Tue Dec  8 12:21:18 2020
  read: IOPS=3024, BW=11.8MiB/s (12.4MB/s)(1418MiB/120001msec)
    slat (nsec): min=2215, max=156932, avg=2401.67, stdev=435.21
    clat (nsec): min=1584, max=19853k, avg=327461.06, stdev=105418.30
     lat (usec): min=132, max=19855, avg=329.95, stdev=105.41
    clat percentiles (usec):
     |  1.00th=[  157],  5.00th=[  176], 10.00th=[  194], 20.00th=[  231],
     | 30.00th=[  289], 40.00th=[  314], 50.00th=[  326], 60.00th=[  338],
     | 70.00th=[  359], 80.00th=[  420], 90.00th=[  461], 95.00th=[  482],
     | 99.00th=[  529], 99.50th=[  570], 99.90th=[  783], 99.95th=[  873],
     | 99.99th=[ 1221]
   bw (  KiB/s): min=11976, max=16144, per=100.00%, avg=12099.72, stdev=603.68, samples=239
   iops        : min= 2994, max= 4036, avg=3024.93, stdev=150.92, samples=239
  lat (usec)   : 2=0.01%, 250=23.44%, 500=74.16%, 750=2.27%, 1000=0.10%
  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=0.66%, sys=0.88%, ctx=363151, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=362996,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=11.8MiB/s (12.4MB/s), 11.8MiB/s-11.8MiB/s (12.4MB/s-12.4MB/s), io=1418MiB (1487MB), run=120001-120001msec

Disk stats (read/write):
  nvme3n1: ios=362696/0, merge=0/0, ticks=118713/0, in_queue=119476, util=99.60%

ST1

readlatency-test-job: (groupid=0, jobs=1): err= 0: pid=5099: Tue Dec  8 14:25:22 2020
  read: IOPS=33, BW=133KiB/s (136kB/s)(15.6MiB/120003msec)
    slat (nsec): min=2314, max=26839, avg=4111.66, stdev=1294.69
    clat (usec): min=97, max=70677, avg=30047.33, stdev=5787.25
     lat (usec): min=100, max=70682, avg=30051.64, stdev=5787.20
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[18744], 10.00th=[25560], 20.00th=[29754],
     | 30.00th=[30278], 40.00th=[30278], 50.00th=[30278], 60.00th=[30278],
     | 70.00th=[30278], 80.00th=[30802], 90.00th=[34866], 95.00th=[40109],
     | 99.00th=[46400], 99.50th=[49021], 99.90th=[54264], 99.95th=[63701],
     | 99.99th=[70779]
   bw (  KiB/s): min=  127, max=  384, per=100.00%, avg=133.03, stdev=16.77, samples=240
   iops        : min=   31, max=   96, avg=33.23, stdev= 4.20, samples=240
  lat (usec)   : 100=0.03%, 250=0.35%, 500=0.05%, 750=0.08%, 1000=0.03%
  lat (msec)   : 2=0.05%, 4=0.15%, 10=0.33%, 20=4.61%, 50=94.01%
  lat (msec)   : 100=0.33%
  cpu          : usr=0.01%, sys=0.01%, ctx=3995, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3993,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=133KiB/s (136kB/s), 133KiB/s-133KiB/s (136kB/s-136kB/s), io=15.6MiB (16.4MB), run=120003-120003msec

Disk stats (read/write):
  nvme1n1: ios=3989/0, merge=0/0, ticks=119854/0, in_queue=119852, util=99.91%

LATENCY RANDRW

fio --filename=device --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=1 --numjobs=1 --time_based --group_reporting --name=readlatency-test-job --runtime=120 --eta-newline=1

GP3

rwlatency-test-job: (groupid=0, jobs=1): err= 0: pid=5102: Tue Dec  8 14:28:21 2020
  read: IOPS=680, BW=2721KiB/s (2787kB/s)(319MiB/120001msec)
    slat (nsec): min=2190, max=27544, avg=2436.86, stdev=433.44
    clat (usec): min=145, max=5660, avg=652.73, stdev=878.30
     lat (usec): min=148, max=5663, avg=655.26, stdev=878.28
    clat percentiles (usec):
     |  1.00th=[  161],  5.00th=[  167], 10.00th=[  169], 20.00th=[  178],
     | 30.00th=[  188], 40.00th=[  251], 50.00th=[  322], 60.00th=[  338],
     | 70.00th=[  396], 80.00th=[  881], 90.00th=[ 1811], 95.00th=[ 2737],
     | 99.00th=[ 4424], 99.50th=[ 4752], 99.90th=[ 4948], 99.95th=[ 4948],
     | 99.99th=[ 5080]
   bw (  KiB/s): min= 2184, max= 3168, per=100.00%, avg=2721.06, stdev=154.96, samples=240
   iops        : min=  546, max=  792, avg=680.24, stdev=38.77, samples=240
  write: IOPS=677, BW=2712KiB/s (2777kB/s)(318MiB/120001msec); 0 zone resets
    slat (nsec): min=2265, max=169813, avg=2656.61, stdev=740.21
    clat (usec): min=298, max=31654, avg=813.37, stdev=850.47
     lat (usec): min=322, max=31657, avg=816.12, stdev=850.44
    clat percentiles (usec):
     |  1.00th=[  383],  5.00th=[  392], 10.00th=[  396], 20.00th=[  408],
     | 30.00th=[  424], 40.00th=[  453], 50.00th=[  474], 60.00th=[  506],
     | 70.00th=[  586], 80.00th=[  906], 90.00th=[ 1844], 95.00th=[ 2802],
     | 99.00th=[ 4490], 99.50th=[ 4817], 99.90th=[ 5014], 99.95th=[ 6980],
     | 99.99th=[11207]
   bw (  KiB/s): min= 2288, max= 3112, per=100.00%, avg=2711.29, stdev=146.42, samples=240
   iops        : min=  572, max=  778, avg=677.80, stdev=36.62, samples=240
  lat (usec)   : 250=20.02%, 500=46.40%, 750=10.97%, 1000=4.20%
  lat (msec)   : 2=9.53%, 4=7.19%, 10=1.69%, 20=0.01%, 50=0.01%
  cpu          : usr=0.35%, sys=0.38%, ctx=163062, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=81638,81346,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=2721KiB/s (2787kB/s), 2721KiB/s-2721KiB/s (2787kB/s-2787kB/s), io=319MiB (334MB), run=120001-120001msec
  WRITE: bw=2712KiB/s (2777kB/s), 2712KiB/s-2712KiB/s (2777kB/s-2777kB/s), io=318MiB (333MB), run=120001-120001msec

Disk stats (read/write):
  nvme2n1: ios=81628/81260, merge=0/0, ticks=53288/66077, in_queue=119200, util=99.36%

GP2

rwlatency-test-job: (groupid=0, jobs=1): err= 0: pid=5107: Tue Dec  8 14:31:30 2020
  read: IOPS=1400, BW=5600KiB/s (5735kB/s)(656MiB/120001msec)
    slat (nsec): min=2240, max=183344, avg=2429.48, stdev=570.94
    clat (nsec): min=1386, max=28928k, avg=231410.77, stdev=114596.33
     lat (usec): min=127, max=28930, avg=233.94, stdev=114.60
    clat percentiles (usec):
     |  1.00th=[  145],  5.00th=[  153], 10.00th=[  157], 20.00th=[  163],
     | 30.00th=[  169], 40.00th=[  178], 50.00th=[  192], 60.00th=[  265],
     | 70.00th=[  281], 80.00th=[  302], 90.00th=[  326], 95.00th=[  347],
     | 99.00th=[  441], 99.50th=[  502], 99.90th=[  742], 99.95th=[  865],
     | 99.99th=[ 1909]
   bw (  KiB/s): min= 4624, max= 6440, per=100.00%, avg=5600.59, stdev=310.01, samples=239
   iops        : min= 1156, max= 1610, avg=1400.15, stdev=77.50, samples=239
  write: IOPS=1399, BW=5597KiB/s (5731kB/s)(656MiB/120001msec); 0 zone resets
    slat (nsec): min=2289, max=35142, avg=2573.56, stdev=373.34
    clat (usec): min=339, max=25415, avg=476.60, stdev=116.37
     lat (usec): min=341, max=25418, avg=479.27, stdev=116.38
    clat percentiles (usec):
     |  1.00th=[  383],  5.00th=[  400], 10.00th=[  408], 20.00th=[  420],
     | 30.00th=[  433], 40.00th=[  445], 50.00th=[  457], 60.00th=[  474],
     | 70.00th=[  490], 80.00th=[  519], 90.00th=[  570], 95.00th=[  619],
     | 99.00th=[  725], 99.50th=[  783], 99.90th=[  963], 99.95th=[ 1172],
     | 99.99th=[ 3851]
   bw (  KiB/s): min= 4736, max= 6208, per=100.00%, avg=5596.25, stdev=241.27, samples=239
   iops        : min= 1184, max= 1552, avg=1399.06, stdev=60.32, samples=239
  lat (usec)   : 2=0.01%, 250=28.40%, 500=58.11%, 750=13.07%, 1000=0.36%
  lat (msec)   : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=0.57%, sys=0.89%, ctx=336069, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=168012,167907,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=5600KiB/s (5735kB/s), 5600KiB/s-5600KiB/s (5735kB/s-5735kB/s), io=656MiB (688MB), run=120001-120001msec
  WRITE: bw=5597KiB/s (5731kB/s), 5597KiB/s-5597KiB/s (5731kB/s-5731kB/s), io=656MiB (688MB), run=120001-120001msec

Disk stats (read/write):
  nvme3n1: ios=167916/167761, merge=0/0, ticks=38846/79935, in_queue=118460, util=98.75%

ST1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment