fio test on optane 905p

gistfile1.txt

> fio --name=benchmark --ioengine=libaio --rw=randrw --rwmixread=75 --bs=4k --numjobs=24 --size=1G --runtime=60 --time_based --direct=1 --group_reporting --filename=/src/test
benchmark: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.36
Starting 24 processes
Jobs: 24 (f=24): [m(24)][100.0%][r=2791MiB/s,w=930MiB/s][r=715k,w=238k IOPS][eta 00m:00s]
benchmark: (groupid=0, jobs=24): err= 0: pid=1660063: Sat Apr 27 10:46:10 2024
  read: IOPS=715k, BW=2793MiB/s (2929MB/s)(164GiB/60002msec)
    slat (nsec): min=1080, max=67616k, avg=22251.47, stdev=83419.17
    clat (nsec): min=160, max=28041k, avg=352.78, stdev=7823.12
     lat (nsec): min=1410, max=67628k, avg=22604.25, stdev=83846.01
    clat percentiles (nsec):
     |  1.00th=[  191],  5.00th=[  251], 10.00th=[  262], 20.00th=[  262],
     | 30.00th=[  270], 40.00th=[  282], 50.00th=[  282], 60.00th=[  290],
     | 70.00th=[  302], 80.00th=[  310], 90.00th=[  330], 95.00th=[  362],
     | 99.00th=[ 1096], 99.50th=[ 3728], 99.90th=[ 6752], 99.95th=[ 7840],
     | 99.99th=[20608]
   bw (  MiB/s): min= 2259, max= 3344, per=100.00%, avg=2796.22, stdev= 8.51, samples=2856
   iops        : min=578334, max=856255, avg=715829.56, stdev=2178.94, samples=2856
  write: IOPS=238k, BW=931MiB/s (976MB/s)(54.5GiB/60002msec); 0 zone resets
    slat (usec): min=2, max=62067, avg=30.39, stdev=121.31
    clat (nsec): min=180, max=22796k, avg=376.20, stdev=9258.15
     lat (usec): min=2, max=62070, avg=30.77, stdev=121.78
    clat percentiles (nsec):
     |  1.00th=[  221],  5.00th=[  262], 10.00th=[  262], 20.00th=[  270],
     | 30.00th=[  270], 40.00th=[  282], 50.00th=[  290], 60.00th=[  302],
     | 70.00th=[  302], 80.00th=[  322], 90.00th=[  342], 95.00th=[  390],
     | 99.00th=[ 1624], 99.50th=[ 4704], 99.90th=[ 7968], 99.95th=[ 9280],
     | 99.99th=[21376]
   bw (  KiB/s): min=771268, max=1144578, per=100.00%, avg=954093.58, stdev=2930.06, samples=2856
   iops        : min=192816, max=286143, avg=238521.28, stdev=732.52, samples=2856
  lat (nsec)   : 250=2.77%, 500=95.41%, 750=0.58%, 1000=0.16%
  lat (usec)   : 2=0.25%, 4=0.34%, 10=0.47%, 20=0.02%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=3.33%, sys=80.03%, ctx=1286746, majf=0, minf=300
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=42905472,14296885,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=2793MiB/s (2929MB/s), 2793MiB/s-2793MiB/s (2929MB/s-2929MB/s), io=164GiB (176GB), run=60002-60002msec
  WRITE: bw=931MiB/s (976MB/s), 931MiB/s-931MiB/s (976MB/s-976MB/s), io=54.5GiB (58.6GB), run=60002-60002msec

SN640
❯ fio --name=benchmark --ioengine=libaio --rw=randrw --rwmixread=75 --bs=4k --numjobs=24 --size=1G --runtime=60 --time_based --direct=1 --group_reporting --filename=./test
benchmark: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.36
Starting 24 processes
benchmark: Laying out IO file (1 file / 1024MiB)
Jobs: 24 (f=0): [f(24)][100.0%][r=648MiB/s,w=216MiB/s][r=166k,w=55.3k IOPS][eta 00m:00s]
benchmark: (groupid=0, jobs=24): err= 0: pid=1660406: Sat Apr 27 10:48:50 2024
read: IOPS=165k, BW=646MiB/s (678MB/s)(37.9GiB/60001msec)
slat (nsec): min=880, max=49270, avg=1351.74, stdev=309.28
clat (usec): min=14, max=4602, avg=135.31, stdev=74.99
lat (usec): min=15, max=4603, avg=136.66, stdev=74.99
clat percentiles (usec):
| 1.00th=[ 93], 5.00th=[ 97], 10.00th=[ 100], 20.00th=[ 104],
| 30.00th=[ 109], 40.00th=[ 112], 50.00th=[ 116], 60.00th=[ 121],
| 70.00th=[ 129], 80.00th=[ 149], 90.00th=[ 188], 95.00th=[ 235],
| 99.00th=[ 424], 99.50th=[ 515], 99.90th=[ 783], 99.95th=[ 1057],
| 99.99th=[ 2638]
bw ( KiB/s): min=525096, max=692896, per=100.00%, avg=662046.59, stdev=1351.09, samples=2856
iops : min=131274, max=173224, avg=165511.65, stdev=337.77, samples=2856
write: IOPS=55.2k, BW=216MiB/s (226MB/s)(12.6GiB/60001msec); 0 zone resets
slat (nsec): min=970, max=78130, avg=1514.81, stdev=353.76
clat (usec): min=12, max=1379, avg=22.14, stdev= 3.89
lat (usec): min=17, max=1383, avg=23.66, stdev= 3.90
clat percentiles (nsec):
| 1.00th=[17536], 5.00th=[18048], 10.00th=[19584], 20.00th=[20096],
| 30.00th=[20352], 40.00th=[20608], 50.00th=[21120], 60.00th=[21888],
| 70.00th=[22912], 80.00th=[24192], 90.00th=[26240], 95.00th=[28032],
| 99.00th=[31872], 99.50th=[33024], 99.90th=[36096], 99.95th=[37632],
| 99.99th=[41216]
bw ( KiB/s): min=170904, max=242296, per=100.00%, avg=220726.52, stdev=537.47, samples=2856
iops : min=42726, max=60574, avg=55181.63, stdev=134.37, samples=2856
lat (usec) : 20=4.46%, 50=20.62%, 100=7.87%, 250=63.98%, 500=2.64%
lat (usec) : 750=0.34%, 1000=0.04%
lat (msec) : 2=0.03%, 4=0.02%, 10=0.01%
cpu : usr=0.86%, sys=2.27%, ctx=13247137, majf=0, minf=346
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=9929712,3310511,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=646MiB/s (678MB/s), 646MiB/s-646MiB/s (678MB/s-678MB/s), io=37.9GiB (40.7GB), run=60001-60001msec
WRITE: bw=216MiB/s (226MB/s), 216MiB/s-216MiB/s (226MB/s-226MB/s), io=12.6GiB (13.6GB), run=60001-60001msec

Disk stats (read/write):
nvme3n1: ios=9898613/3300455, sectors=79188904/26403816, merge=0/4, ticks=1323169/66235, in_queue=1389404, util=99.85%