Skip to content

Instantly share code, notes, and snippets.

@lightrush
Last active March 5, 2022 22:34
Show Gist options
  • Save lightrush/1bf07ab00ff4d15a1be64e119d11e53c to your computer and use it in GitHub Desktop.
Save lightrush/1bf07ab00ff4d15a1be64e119d11e53c to your computer and use it in GitHub Desktop.
Framework Storage Expansion Card (SEC) 1TB vs WD SN850 1TB in a Pluggable NVMe-to-USB enclosure

Framework Storage Expansion Card (SEC) 1TB vs WD SN850 1TB in a Pluggable NVMe-to-USB enclosure

Tests

4k-random-write

fio --name=write_iops --size=10G \
--time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 \
--verify=0 --bs=4K --iodepth=64 --rw=randwrite --group_reporting=1

1M-sequential-write

fio --name=write_throughput --numjobs=8 \
--size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio \
--direct=1 --verify=0 --bs=1M --iodepth=64 --rw=write \
--group_reporting=1

4K-random-read

fio --name=read_iops --size=10G \
--time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 \
--verify=0 --bs=4K --iodepth=64 --rw=randread --group_reporting=1

1M-sequential-read

fio --name=read_throughput --numjobs=8 \
--size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio \
--direct=1 --verify=0 --bs=1M --iodepth=64 --rw=read \
--group_reporting=1

Results

Summary

SEC 1TB SN850 1TB
4k-random-write 249.5 MiB/s 262 MiB/s
1M-sequential-write 722.5 MiB/s 979 MiB/s
4K-random-read 218.5 MiB/s 235 MiB/s
1M-sequential-read 988 MiB/s 1023 MiB/s

Framework Storage Expansion Card (SEC) 1TB

4k-random-write

Run 1:

write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process
write_iops: Laying out IO file (1 file / 10240MiB)

write_iops: (groupid=0, jobs=1): err= 0: pid=555361: Sat Mar  5 16:18:26 2022
  write: IOPS=69.9k, BW=273MiB/s (286MB/s)(15.0GiB/60001msec); 0 zone resets
    slat (nsec): min=1093, max=52085k, avg=13671.65, stdev=71591.99
    clat (usec): min=254, max=56921, avg=901.96, stdev=645.34
     lat (usec): min=258, max=56927, avg=915.69, stdev=650.79
    clat percentiles (usec):
     |  1.00th=[  510],  5.00th=[  578], 10.00th=[  603], 20.00th=[  660],
     | 30.00th=[  725], 40.00th=[  799], 50.00th=[  848], 60.00th=[  930],
     | 70.00th=[  979], 80.00th=[ 1074], 90.00th=[ 1172], 95.00th=[ 1237],
     | 99.00th=[ 2008], 99.50th=[ 2343], 99.90th=[ 7111], 99.95th=[ 8979],
     | 99.99th=[36439]
   bw (  KiB/s): min=244440, max=295872, per=100.00%, avg=279493.56, stdev=11282.74, samples=119
   iops        : min=61110, max=73968, avg=69873.34, stdev=2820.69, samples=119
  lat (usec)   : 500=0.63%, 750=34.02%, 1000=37.65%
  lat (msec)   : 2=26.71%, 4=0.88%, 10=0.08%, 20=0.01%, 50=0.02%
  lat (msec)   : 100=0.01%
  cpu          : usr=4.98%, sys=38.49%, ctx=397079, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,4192225,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=273MiB/s (286MB/s), 273MiB/s-273MiB/s (286MB/s-286MB/s), io=15.0GiB (17.2GB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=0/4313120, merge=0/55330, ticks=0/3417481, in_queue=3417622, util=99.86%

Run 2:

write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process

write_iops: (groupid=0, jobs=1): err= 0: pid=555879: Sat Mar  5 16:19:40 2022
  write: IOPS=57.6k, BW=225MiB/s (236MB/s)(13.2GiB/60005msec); 0 zone resets
    slat (nsec): min=1025, max=31872k, avg=16601.21, stdev=121301.79
    clat (usec): min=239, max=34416, avg=1094.77, stdev=1003.48
     lat (usec): min=244, max=34417, avg=1111.45, stdev=1012.11
    clat percentiles (usec):
     |  1.00th=[  445],  5.00th=[  586], 10.00th=[  603], 20.00th=[  709],
     | 30.00th=[  734], 40.00th=[  840], 50.00th=[  889], 60.00th=[  971],
     | 70.00th=[ 1074], 80.00th=[ 1188], 90.00th=[ 1582], 95.00th=[ 2114],
     | 99.00th=[ 6325], 99.50th=[ 7963], 99.90th=[10683], 99.95th=[11469],
     | 99.99th=[31327]
   bw (  KiB/s): min=107640, max=288936, per=99.99%, avg=230229.71, stdev=57019.06, samples=120
   iops        : min=26910, max=72234, avg=57557.40, stdev=14254.76, samples=120
  lat (usec)   : 250=0.01%, 500=2.03%, 750=30.27%, 1000=31.57%
  lat (msec)   : 2=30.40%, 4=4.12%, 10=1.43%, 20=0.16%, 50=0.02%
  cpu          : usr=4.19%, sys=18.20%, ctx=402356, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,3454191,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=13.2GiB (14.1GB), run=60005-60005msec

Disk stats (read/write):
  sda: ios=18/3592756, merge=40/3991, ticks=143/3408371, in_queue=3408689, util=99.84%

1M-sequential-write

Run 1:

write_throughput: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.16
Starting 8 processes

write_throughput: (groupid=0, jobs=8): err= 0: pid=574536: Sat Mar  5 16:59:00 2022
  write: IOPS=702, BW=711MiB/s (745MB/s)(42.1GiB/60722msec); 0 zone resets
    slat (usec): min=34, max=5041.6k, avg=11287.14, stdev=68759.80
    clat (msec): min=67, max=8860, avg=712.02, stdev=916.11
     lat (msec): min=68, max=8860, avg=723.28, stdev=926.64
    clat percentiles (msec):
     |  1.00th=[   72],  5.00th=[   75], 10.00th=[  426], 20.00th=[  523],
     | 30.00th=[  550], 40.00th=[  575], 50.00th=[  592], 60.00th=[  617],
     | 70.00th=[  642], 80.00th=[  667], 90.00th=[  709], 95.00th=[  768],
     | 99.00th=[ 5805], 99.50th=[ 6879], 99.90th=[ 7953], 99.95th=[ 8221],
     | 99.99th=[ 8557]
   bw (  KiB/s): min=16334, max=1887969, per=100.00%, avg=768946.88, stdev=86020.37, samples=903
   iops        : min=   14, max= 1843, avg=750.45, stdev=84.01, samples=903
  lat (msec)   : 100=9.67%, 250=0.07%, 500=5.90%, 750=79.66%, 1000=2.22%
  lat (msec)   : 2000=0.32%, >=2000=3.34%
  cpu          : usr=0.67%, sys=0.62%, ctx=27604, majf=0, minf=120
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,42637,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=711MiB/s (745MB/s), 711MiB/s-711MiB/s (745MB/s-745MB/s), io=42.1GiB (45.2GB), run=60722-60722msec

Disk stats (read/write):
  sda: ios=8/88672, merge=0/74, ticks=518/3528404, in_queue=3529979, util=99.87%

Run 2:

write_throughput: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.16
Starting 8 processes

write_throughput: (groupid=0, jobs=8): err= 0: pid=578655: Sat Mar  5 17:07:11 2022
  write: IOPS=730, BW=734MiB/s (769MB/s)(43.5GiB/60742msec); 0 zone resets
    slat (usec): min=33, max=707125, avg=10879.04, stdev=25589.64
    clat (msec): min=230, max=7716, avg=671.17, stdev=705.23
     lat (msec): min=331, max=7716, avg=682.34, stdev=715.70
    clat percentiles (msec):
     |  1.00th=[  384],  5.00th=[  430], 10.00th=[  460], 20.00th=[  493],
     | 30.00th=[  518], 40.00th=[  542], 50.00th=[  558], 60.00th=[  584],
     | 70.00th=[  609], 80.00th=[  634], 90.00th=[  676], 95.00th=[  735],
     | 99.00th=[ 5201], 99.50th=[ 6141], 99.90th=[ 7215], 99.95th=[ 7416],
     | 99.99th=[ 7684]
   bw (  KiB/s): min=22528, max=1310720, per=100.00%, avg=814583.35, stdev=38409.34, samples=886
   iops        : min=   22, max= 1280, avg=795.19, stdev=37.50, samples=886
  lat (msec)   : 250=0.01%, 500=22.60%, 750=73.44%, 1000=1.20%, 2000=0.52%
  lat (msec)   : >=2000=2.65%
  cpu          : usr=0.71%, sys=0.59%, ctx=29640, majf=0, minf=120
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=0.6%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,44377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=734MiB/s (769MB/s), 734MiB/s-734MiB/s (769MB/s-769MB/s), io=43.5GiB (46.7GB), run=60742-60742msec

Disk stats (read/write):
  sda: ios=3/89127, merge=0/47, ticks=1438/3549874, in_queue=3552534, util=99.86%

4K-random-read

Run 1:

read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process

read_iops: (groupid=0, jobs=1): err= 0: pid=572212: Sat Mar  5 16:52:56 2022
  read: IOPS=55.8k, BW=218MiB/s (229MB/s)(12.8GiB/60001msec)
    slat (nsec): min=850, max=5127.7k, avg=17307.11, stdev=57411.63
    clat (usec): min=322, max=29725, avg=1129.04, stdev=537.24
     lat (usec): min=354, max=29727, avg=1146.41, stdev=543.64
    clat percentiles (usec):
     |  1.00th=[  570],  5.00th=[  668], 10.00th=[  725], 20.00th=[  824],
     | 30.00th=[  914], 40.00th=[  996], 50.00th=[ 1074], 60.00th=[ 1156],
     | 70.00th=[ 1237], 80.00th=[ 1336], 90.00th=[ 1483], 95.00th=[ 1647],
     | 99.00th=[ 2606], 99.50th=[ 3228], 99.90th=[ 9372], 99.95th=[ 9765],
     | 99.99th=[13829]
   bw (  KiB/s): min=201832, max=247880, per=99.99%, avg=223240.36, stdev=9235.04, samples=120
   iops        : min=50458, max=61970, avg=55810.10, stdev=2308.72, samples=120
  lat (usec)   : 500=0.13%, 750=11.88%, 1000=29.03%
  lat (msec)   : 2=56.68%, 4=1.95%, 10=0.29%, 20=0.04%, 50=0.01%
  cpu          : usr=3.21%, sys=16.85%, ctx=391933, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=3348827,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=218MiB/s (229MB/s), 218MiB/s-218MiB/s (229MB/s-229MB/s), io=12.8GiB (13.7GB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=3458076/9, merge=3927/7, ticks=3506538/29, in_queue=3506586, util=99.88%

Run 2:

read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process

read_iops: (groupid=0, jobs=1): err= 0: pid=570106: Sat Mar  5 16:47:56 2022
  read: IOPS=56.1k, BW=219MiB/s (230MB/s)(12.8GiB/60001msec)
    slat (nsec): min=889, max=5151.8k, avg=17155.15, stdev=57727.15
    clat (usec): min=248, max=33164, avg=1124.16, stdev=544.54
     lat (usec): min=257, max=33165, avg=1141.39, stdev=550.88
    clat percentiles (usec):
     |  1.00th=[  545],  5.00th=[  652], 10.00th=[  717], 20.00th=[  816],
     | 30.00th=[  898], 40.00th=[  988], 50.00th=[ 1074], 60.00th=[ 1156],
     | 70.00th=[ 1237], 80.00th=[ 1336], 90.00th=[ 1500], 95.00th=[ 1663],
     | 99.00th=[ 2606], 99.50th=[ 3195], 99.90th=[ 9372], 99.95th=[ 9896],
     | 99.99th=[13960]
   bw (  KiB/s): min=202376, max=236920, per=100.00%, avg=224222.93, stdev=9162.66, samples=120
   iops        : min=50594, max=59230, avg=56055.75, stdev=2290.67, samples=120
  lat (usec)   : 250=0.01%, 500=0.37%, 750=12.79%, 1000=28.84%
  lat (msec)   : 2=55.60%, 4=2.08%, 10=0.27%, 20=0.05%, 50=0.01%
  cpu          : usr=3.58%, sys=18.52%, ctx=389783, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=3363453,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=219MiB/s (230MB/s), 219MiB/s-219MiB/s (230MB/s-230MB/s), io=12.8GiB (13.8GB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=3473045/10, merge=3948/7, ticks=3475528/27, in_queue=3475573, util=99.88%

1M-sequential-read

Run 1:

read_throughput: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.16
Starting 8 processes

read_throughput: (groupid=0, jobs=8): err= 0: pid=566707: Sat Mar  5 16:41:42 2022
  read: IOPS=979, BW=988MiB/s (1036MB/s)(57.0GiB/60041msec)
    slat (usec): min=17, max=103694, avg=8158.16, stdev=11001.20
    clat (msec): min=18, max=903, avg=512.16, stdev=94.17
     lat (msec): min=30, max=914, avg=520.32, stdev=95.04
    clat percentiles (msec):
     |  1.00th=[  292],  5.00th=[  363], 10.00th=[  401], 20.00th=[  435],
     | 30.00th=[  464], 40.00th=[  489], 50.00th=[  510], 60.00th=[  535],
     | 70.00th=[  558], 80.00th=[  592], 90.00th=[  634], 95.00th=[  667],
     | 99.00th=[  743], 99.50th=[  768], 99.90th=[  818], 99.95th=[  844],
     | 99.99th=[  869]
   bw (  KiB/s): min=579584, max=1550336, per=99.20%, avg=1003994.18, stdev=22820.20, samples=960
   iops        : min=  566, max= 1514, avg=980.07, stdev=22.30, samples=960
  lat (msec)   : 20=0.01%, 50=0.05%, 100=0.08%, 250=0.32%, 500=45.11%
  lat (msec)   : 750=54.51%, 1000=0.78%
  cpu          : usr=0.05%, sys=0.63%, ctx=33930, majf=0, minf=123
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=58832,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=988MiB/s (1036MB/s), 988MiB/s-988MiB/s (1036MB/s-1036MB/s), io=57.0GiB (62.2GB), run=60041-60041msec

Disk stats (read/write):
  sda: ios=121574/12, merge=2/7, ticks=3519202/6394, in_queue=3525674, util=99.92%

Run 2:

read_throughput: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.16
Starting 8 processes

read_throughput: (groupid=0, jobs=8): err= 0: pid=568603: Sat Mar  5 16:45:56 2022
  read: IOPS=979, BW=988MiB/s (1036MB/s)(57.9GiB/60049msec)
    slat (usec): min=17, max=107366, avg=8164.86, stdev=11020.93
    clat (msec): min=18, max=887, avg=512.59, stdev=91.69
     lat (msec): min=29, max=898, avg=520.76, stdev=92.58
    clat percentiles (msec):
     |  1.00th=[  309],  5.00th=[  368], 10.00th=[  401], 20.00th=[  439],
     | 30.00th=[  468], 40.00th=[  489], 50.00th=[  510], 60.00th=[  535],
     | 70.00th=[  558], 80.00th=[  592], 90.00th=[  634], 95.00th=[  667],
     | 99.00th=[  743], 99.50th=[  760], 99.90th=[  802], 99.95th=[  818],
     | 99.99th=[  860]
   bw (  KiB/s): min=608112, max=1505280, per=99.21%, avg=1003290.27, stdev=21985.68, samples=960
   iops        : min=  593, max= 1470, avg=979.44, stdev=21.47, samples=960
  lat (msec)   : 20=0.01%, 50=0.04%, 100=0.09%, 250=0.25%, 500=45.46%
  lat (msec)   : 750=54.30%, 1000=0.71%
  cpu          : usr=0.05%, sys=0.63%, ctx=33994, majf=0, minf=121
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=58793,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=988MiB/s (1036MB/s), 988MiB/s-988MiB/s (1036MB/s-1036MB/s), io=57.9GiB (62.2GB), run=60049-60049msec

Disk stats (read/write):
  sda: ios=121463/14, merge=3/11, ticks=3517759/7351, in_queue=3525187, util=99.93%

WD SN850 1TB in a Pluggable NVMe-to-USB enclosure

4k-random-write

Run 1:

write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process

write_iops: (groupid=0, jobs=1): err= 0: pid=546999: Sat Mar  5 15:58:29 2022
  write: IOPS=67.2k, BW=263MiB/s (275MB/s)(15.4GiB/60001msec); 0 zone resets
    slat (nsec): min=1081, max=16950k, avg=14244.89, stdev=36184.47
    clat (usec): min=254, max=27114, avg=937.14, stdev=366.22
     lat (usec): min=259, max=27120, avg=951.45, stdev=369.09
    clat percentiles (usec):
     |  1.00th=[  553],  5.00th=[  594], 10.00th=[  619], 20.00th=[  709],
     | 30.00th=[  758], 40.00th=[  840], 50.00th=[  906], 60.00th=[  963],
     | 70.00th=[ 1057], 80.00th=[ 1106], 90.00th=[ 1205], 95.00th=[ 1319],
     | 99.00th=[ 2040], 99.50th=[ 2409], 99.90th=[ 3097], 99.95th=[ 3458],
     | 99.99th=[14484]
   bw (  KiB/s): min=250736, max=284808, per=99.97%, avg=268896.10, stdev=6256.64, samples=120
   iops        : min=62684, max=71202, avg=67223.94, stdev=1564.20, samples=120
  lat (usec)   : 500=0.07%, 750=28.63%, 1000=36.14%
  lat (msec)   : 2=34.05%, 4=1.08%, 10=0.02%, 20=0.01%, 50=0.01%
  cpu          : usr=4.86%, sys=33.63%, ctx=420347, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,4034790,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=263MiB/s (275MB/s), 263MiB/s-263MiB/s (275MB/s-275MB/s), io=15.4GiB (16.5GB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=0/4144510, merge=0/52322, ticks=0/3457248, in_queue=3457258, util=99.88%

Run 2:

write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process

write_iops: (groupid=0, jobs=1): err= 0: pid=547527: Sat Mar  5 16:00:08 2022
  write: IOPS=66.8k, BW=261MiB/s (274MB/s)(15.3GiB/60001msec); 0 zone resets
    slat (nsec): min=966, max=9306.2k, avg=14266.75, stdev=35294.48
    clat (usec): min=240, max=11958, avg=943.05, stdev=300.77
     lat (usec): min=245, max=11965, avg=957.38, stdev=303.12
    clat percentiles (usec):
     |  1.00th=[  519],  5.00th=[  594], 10.00th=[  611], 20.00th=[  717],
     | 30.00th=[  742], 40.00th=[  840], 50.00th=[  938], 60.00th=[  963],
     | 70.00th=[ 1074], 80.00th=[ 1123], 90.00th=[ 1221], 95.00th=[ 1336],
     | 99.00th=[ 2057], 99.50th=[ 2409], 99.90th=[ 3032], 99.95th=[ 3326],
     | 99.99th=[ 3949]
   bw (  KiB/s): min=259776, max=282992, per=99.98%, avg=267248.71, stdev=4133.75, samples=120
   iops        : min=64944, max=70748, avg=66812.14, stdev=1033.45, samples=120
  lat (usec)   : 250=0.01%, 500=0.83%, 750=29.84%, 1000=34.28%
  lat (msec)   : 2=33.86%, 4=1.18%, 10=0.01%, 20=0.01%
  cpu          : usr=4.69%, sys=17.43%, ctx=496395, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,4009730,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=261MiB/s (274MB/s), 261MiB/s-261MiB/s (274MB/s-274MB/s), io=15.3GiB (16.4GB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=2/4116663, merge=0/4038, ticks=1/3430834, in_queue=3430846, util=99.85%

1M-sequential-write

Run 1:

write_throughput: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.16
Starting 8 processes
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)

write_throughput: (groupid=0, jobs=8): err= 0: pid=548819: Sat Mar  5 16:02:27 2022
  write: IOPS=984, BW=993MiB/s (1042MB/s)(58.2GiB/60044msec); 0 zone resets
    slat (usec): min=32, max=78417, avg=8116.27, stdev=9462.65
    clat (msec): min=16, max=778, avg=509.64, stdev=71.24
     lat (msec): min=29, max=785, avg=517.75, stdev=71.80
    clat percentiles (msec):
     |  1.00th=[  351],  5.00th=[  401], 10.00th=[  426], 20.00th=[  456],
     | 30.00th=[  477], 40.00th=[  493], 50.00th=[  510], 60.00th=[  527],
     | 70.00th=[  542], 80.00th=[  567], 90.00th=[  600], 95.00th=[  625],
     | 99.00th=[  684], 99.50th=[  701], 99.90th=[  743], 99.95th=[  751],
     | 99.99th=[  760]
   bw (  KiB/s): min=673792, max=1343488, per=99.20%, avg=1008934.51, stdev=16791.88, samples=960
   iops        : min=  658, max= 1312, avg=984.78, stdev=16.41, samples=960
  lat (msec)   : 20=0.01%, 50=0.06%, 100=0.07%, 250=0.26%, 500=44.81%
  lat (msec)   : 750=55.61%, 1000=0.05%
  cpu          : usr=0.80%, sys=0.65%, ctx=38332, majf=0, minf=130
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,59127,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=993MiB/s (1042MB/s), 993MiB/s-993MiB/s (1042MB/s-1042MB/s), io=58.2GiB (62.5GB), run=60044-60044msec

Disk stats (read/write):
  sda: ios=0/122169, merge=0/1383, ticks=0/3514331, in_queue=3514351, util=99.88%

Run 2:

write_throughput: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.16
Starting 8 processes

write_throughput: (groupid=0, jobs=8): err= 0: pid=550269: Sat Mar  5 16:06:24 2022
  write: IOPS=956, BW=965MiB/s (1012MB/s)(56.6GiB/60043msec); 0 zone resets
    slat (usec): min=33, max=104355, avg=8354.34, stdev=10726.77
    clat (msec): min=18, max=770, avg=524.32, stdev=73.06
     lat (msec): min=28, max=776, avg=532.67, stdev=73.50
    clat percentiles (msec):
     |  1.00th=[  355],  5.00th=[  414], 10.00th=[  435], 20.00th=[  464],
     | 30.00th=[  489], 40.00th=[  510], 50.00th=[  527], 60.00th=[  542],
     | 70.00th=[  567], 80.00th=[  584], 90.00th=[  617], 95.00th=[  642],
     | 99.00th=[  684], 99.50th=[  701], 99.90th=[  726], 99.95th=[  735],
     | 99.99th=[  760]
   bw (  KiB/s): min=692040, max=1351680, per=99.18%, avg=980403.84, stdev=16475.72, samples=960
   iops        : min=  675, max= 1320, avg=957.14, stdev=16.09, samples=960
  lat (msec)   : 20=0.01%, 50=0.04%, 100=0.09%, 250=0.26%, 500=35.74%
  lat (msec)   : 750=64.72%, 1000=0.02%
  cpu          : usr=0.68%, sys=0.49%, ctx=36887, majf=0, minf=120
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,57451,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=965MiB/s (1012MB/s), 965MiB/s-965MiB/s (1012MB/s-1012MB/s), io=56.6GiB (60.8GB), run=60043-60043msec

Disk stats (read/write):
  sda: ios=0/118751, merge=0/30, ticks=0/3519176, in_queue=3519199, util=99.89%

4K-random-read

Run 1:

read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process
read_iops: Laying out IO file (1 file / 10240MiB)

read_iops: (groupid=0, jobs=1): err= 0: pid=551355: Sat Mar  5 16:09:36 2022
  read: IOPS=59.0k, BW=234MiB/s (246MB/s)(13.7GiB/60001msec)
    slat (nsec): min=878, max=873052, avg=16072.77, stdev=36953.75
    clat (usec): min=301, max=6283, avg=1050.84, stdev=318.96
     lat (usec): min=334, max=6389, avg=1066.97, stdev=321.31
    clat percentiles (usec):
     |  1.00th=[  603],  5.00th=[  701], 10.00th=[  717], 20.00th=[  783],
     | 30.00th=[  848], 40.00th=[  955], 50.00th=[ 1004], 60.00th=[ 1090],
     | 70.00th=[ 1188], 80.00th=[ 1254], 90.00th=[ 1352], 95.00th=[ 1450],
     | 99.00th=[ 2278], 99.50th=[ 2638], 99.90th=[ 3359], 99.95th=[ 3621],
     | 99.99th=[ 4146]
   bw (  KiB/s): min=238352, max=251280, per=100.00%, avg=239868.24, stdev=1361.29, samples=120
   iops        : min=59588, max=62820, avg=59967.07, stdev=340.34, samples=120
  lat (usec)   : 500=0.01%, 750=17.95%, 1000=31.58%
  lat (msec)   : 2=48.81%, 4=1.64%, 10=0.02%
  cpu          : usr=3.60%, sys=11.69%, ctx=502117, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=3598143,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=234MiB/s (246MB/s), 234MiB/s-234MiB/s (246MB/s-246MB/s), io=13.7GiB (14.7GB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=3712850/4, merge=4058/1, ticks=3528865/3, in_queue=3528869, util=99.88%

Run 2:

read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process

read_iops: (groupid=0, jobs=1): err= 0: pid=552005: Sat Mar  5 16:10:58 2022
  read: IOPS=60.2k, BW=235MiB/s (247MB/s)(13.8GiB/60001msec)
    slat (nsec): min=871, max=6631.0k, avg=15953.01, stdev=37878.40
    clat (usec): min=139, max=8239, avg=1046.22, stdev=323.29
     lat (usec): min=212, max=8241, avg=1062.24, stdev=325.56
    clat percentiles (usec):
     |  1.00th=[  603],  5.00th=[  652], 10.00th=[  717], 20.00th=[  766],
     | 30.00th=[  848], 40.00th=[  955], 50.00th=[  996], 60.00th=[ 1090],
     | 70.00th=[ 1188], 80.00th=[ 1237], 90.00th=[ 1352], 95.00th=[ 1467],
     | 99.00th=[ 2278], 99.50th=[ 2638], 99.90th=[ 3392], 99.95th=[ 3687],
     | 99.99th=[ 4621]
   bw (  KiB/s): min=236720, max=255072, per=100.00%, avg=240928.98, stdev=2412.99, samples=120
   iops        : min=59180, max=63768, avg=60232.26, stdev=603.23, samples=120
  lat (usec)   : 250=0.01%, 500=0.13%, 750=18.47%, 1000=31.56%
  lat (msec)   : 2=48.20%, 4=1.62%, 10=0.03%
  cpu          : usr=3.86%, sys=12.64%, ctx=498236, majf=0, minf=15
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=3614057,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=235MiB/s (247MB/s), 235MiB/s-235MiB/s (247MB/s-247MB/s), io=13.8GiB (14.8GB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=3730935/14, merge=3985/9, ticks=3512795/13, in_queue=3512810, util=99.84%

1M-sequential-read

Run 1:

read_throughput: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.16
Starting 8 processes
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)

read_throughput: (groupid=0, jobs=8): err= 0: pid=586930: Sat Mar  5 17:21:33 2022
  read: IOPS=1014, BW=1023MiB/s (1073MB/s)(59.0GiB/60038msec)
    slat (usec): min=14, max=106977, avg=7879.32, stdev=10063.60
    clat (msec): min=15, max=824, avg=494.71, stdev=74.71
     lat (msec): min=30, max=824, avg=502.59, stdev=75.30
    clat percentiles (msec):
     |  1.00th=[  317],  5.00th=[  380], 10.00th=[  409], 20.00th=[  439],
     | 30.00th=[  460], 40.00th=[  477], 50.00th=[  493], 60.00th=[  510],
     | 70.00th=[  527], 80.00th=[  550], 90.00th=[  584], 95.00th=[  617],
     | 99.00th=[  676], 99.50th=[  701], 99.90th=[  743], 99.95th=[  768],
     | 99.99th=[  785]
   bw (  KiB/s): min=663400, max=1433600, per=99.22%, avg=1039631.31, stdev=17952.69, samples=960
   iops        : min=  647, max= 1400, avg=1014.97, stdev=17.53, samples=960
  lat (msec)   : 20=0.01%, 50=0.04%, 100=0.08%, 250=0.27%, 500=54.58%
  lat (msec)   : 750=45.77%, 1000=0.08%
  cpu          : usr=0.06%, sys=0.61%, ctx=35514, majf=0, minf=124
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=60922,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1023MiB/s (1073MB/s), 1023MiB/s-1023MiB/s (1073MB/s-1073MB/s), io=59.0GiB (64.4GB), run=60038-60038msec

Disk stats (read/write):
  sda: ios=125871/4, merge=0/3, ticks=3525405/80, in_queue=3525513, util=99.93%

Run 2:

read_throughput: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.16
Starting 8 processes

read_throughput: (groupid=0, jobs=8): err= 0: pid=588421: Sat Mar  5 17:23:40 2022
  read: IOPS=1013, BW=1022MiB/s (1072MB/s)(59.9GiB/60049msec)
    slat (usec): min=16, max=103922, avg=7885.94, stdev=9751.39
    clat (msec): min=16, max=800, avg=495.07, stdev=70.68
     lat (msec): min=30, max=800, avg=502.95, stdev=71.22
    clat percentiles (msec):
     |  1.00th=[  342],  5.00th=[  388], 10.00th=[  414], 20.00th=[  439],
     | 30.00th=[  460], 40.00th=[  477], 50.00th=[  493], 60.00th=[  510],
     | 70.00th=[  531], 80.00th=[  550], 90.00th=[  584], 95.00th=[  609],
     | 99.00th=[  667], 99.50th=[  693], 99.90th=[  726], 99.95th=[  743],
     | 99.99th=[  776]
   bw (  KiB/s): min=698368, max=1388544, per=99.24%, avg=1038770.43, stdev=16903.94, samples=960
   iops        : min=  682, max= 1356, avg=1014.11, stdev=16.51, samples=960
  lat (msec)   : 20=0.01%, 50=0.05%, 100=0.09%, 250=0.24%, 500=54.19%
  lat (msec)   : 750=46.22%, 1000=0.04%
  cpu          : usr=0.06%, sys=0.58%, ctx=36212, majf=0, minf=120
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=60872,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1022MiB/s (1072MB/s), 1022MiB/s-1022MiB/s (1072MB/s-1072MB/s), io=59.9GiB (64.4GB), run=60049-60049msec

Disk stats (read/write):
  sda: ios=125755/9, merge=0/7, ticks=3526714/199, in_queue=3526971, util=99.93%
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment