Skip to content

Instantly share code, notes, and snippets.

@petrilli

petrilli/benchmark-result-hostpath.txt

Last active August 6, 2022 22:49
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save petrilli/238e55dc2c4c53a46d51b570cf42db3f to your computer and use it in GitHub Desktop.
Save petrilli/238e55dc2c4c53a46d51b570cf42db3f to your computer and use it in GitHub Desktop.
Download ZIP
Flexible IO Benchmark Test (cluster with Mayastor)
Raw
benchmark-result-hostpath.txt
$ kubectl exec -it fio -- fio --name=benchtest --size=800m --filename=/volume/test --direct=1 --rw=randrw --ioengine=libaio --bs=4k --iodepth=16 --numjobs=8 --time_based --runtime=60
benchtest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.30
Starting 8 processes
benchtest: Laying out IO file (1 file / 800MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=181MiB/s,w=180MiB/s][r=46.4k,w=46.1k IOPS][eta 00m:00s]
benchtest: (groupid=0, jobs=1): err= 0: pid=21: Sat Aug 6 04:17:40 2022
read: IOPS=5780, BW=22.6MiB/s (23.7MB/s)(1355MiB/60001msec)
slat (usec): min=2, max=12815, avg=77.89, stdev=203.24
clat (usec): min=101, max=47149, avg=1244.34, stdev=774.39
lat (usec): min=108, max=47158, avg=1322.45, stdev=812.86
clat percentiles (usec):
| 1.00th=[ 355], 5.00th=[ 453], 10.00th=[ 529], 20.00th=[ 644],
| 30.00th=[ 775], 40.00th=[ 930], 50.00th=[ 1090], 60.00th=[ 1254],
| 70.00th=[ 1467], 80.00th=[ 1729], 90.00th=[ 2147], 95.00th=[ 2540],
| 99.00th=[ 3621], 99.50th=[ 4178], 99.90th=[ 7373], 99.95th=[10028],
| 99.99th=[14091]
bw ( KiB/s): min=20136, max=25440, per=12.53%, avg=23130.48, stdev=792.27, samples=119
iops : min= 5034, max= 6360, avg=5782.61, stdev=198.05, samples=119
write: IOPS=5770, BW=22.5MiB/s (23.6MB/s)(1353MiB/60001msec); 0 zone resets
slat (usec): min=2, max=24661, avg=73.24, stdev=197.84
clat (usec): min=24, max=47465, avg=1372.13, stdev=841.34
lat (usec): min=32, max=47792, avg=1445.59, stdev=875.53
clat percentiles (usec):
| 1.00th=[ 227], 5.00th=[ 383], 10.00th=[ 510], 20.00th=[ 709],
| 30.00th=[ 898], 40.00th=[ 1057], 50.00th=[ 1221], 60.00th=[ 1434],
| 70.00th=[ 1647], 80.00th=[ 1926], 90.00th=[ 2376], 95.00th=[ 2769],
| 99.00th=[ 3818], 99.50th=[ 4359], 99.90th=[ 7373], 99.95th=[ 9241],
| 99.99th=[13566]
bw ( KiB/s): min=20352, max=25464, per=12.51%, avg=23097.60, stdev=870.91, samples=119
iops : min= 5088, max= 6366, avg=5774.39, stdev=217.72, samples=119
lat (usec) : 50=0.01%, 100=0.04%, 250=0.69%, 500=8.02%, 750=16.25%
lat (usec) : 1000=15.34%
lat (msec) : 2=44.38%, 4=14.57%, 10=0.65%, 20=0.04%, 50=0.01%
cpu : usr=2.44%, sys=10.65%, ctx=253686, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=346812,346255,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=22: Sat Aug 6 04:17:40 2022
read: IOPS=5728, BW=22.4MiB/s (23.5MB/s)(1343MiB/60001msec)
slat (usec): min=2, max=32552, avg=78.36, stdev=211.42
clat (usec): min=104, max=135792, avg=1254.56, stdev=973.89
lat (usec): min=125, max=136049, avg=1333.14, stdev=1015.77
clat percentiles (usec):
| 1.00th=[ 359], 5.00th=[ 457], 10.00th=[ 529], 20.00th=[ 652],
| 30.00th=[ 783], 40.00th=[ 938], 50.00th=[ 1106], 60.00th=[ 1270],
| 70.00th=[ 1467], 80.00th=[ 1729], 90.00th=[ 2147], 95.00th=[ 2540],
| 99.00th=[ 3589], 99.50th=[ 4080], 99.90th=[ 7373], 99.95th=[10421],
| 99.99th=[17171]
bw ( KiB/s): min=17920, max=24984, per=12.42%, avg=22920.72, stdev=948.43, samples=119
iops : min= 4480, max= 6246, avg=5730.16, stdev=237.11, samples=119
write: IOPS=5745, BW=22.4MiB/s (23.5MB/s)(1347MiB/60001msec); 0 zone resets
slat (usec): min=2, max=26105, avg=73.60, stdev=203.24
clat (usec): min=29, max=136211, avg=1379.52, stdev=1046.53
lat (usec): min=48, max=136219, avg=1453.37, stdev=1078.57
clat percentiles (usec):
| 1.00th=[ 235], 5.00th=[ 388], 10.00th=[ 515], 20.00th=[ 717],
| 30.00th=[ 898], 40.00th=[ 1057], 50.00th=[ 1237], 60.00th=[ 1434],
| 70.00th=[ 1647], 80.00th=[ 1942], 90.00th=[ 2376], 95.00th=[ 2769],
| 99.00th=[ 3818], 99.50th=[ 4359], 99.90th=[ 7308], 99.95th=[10159],
| 99.99th=[15401]
bw ( KiB/s): min=16840, max=25424, per=12.45%, avg=22992.89, stdev=996.13, samples=119
iops : min= 4210, max= 6356, avg=5748.21, stdev=249.01, samples=119
lat (usec) : 50=0.01%, 100=0.03%, 250=0.66%, 500=7.90%, 750=16.05%
lat (usec) : 1000=15.27%
lat (msec) : 2=44.63%, 4=14.80%, 10=0.61%, 20=0.05%, 50=0.01%
lat (msec) : 250=0.01%
cpu : usr=2.46%, sys=10.52%, ctx=251232, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=343691,344716,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=23: Sat Aug 6 04:17:40 2022
read: IOPS=5787, BW=22.6MiB/s (23.7MB/s)(1357MiB/60001msec)
slat (usec): min=2, max=24869, avg=78.17, stdev=208.58
clat (usec): min=67, max=25886, avg=1243.41, stdev=754.18
lat (usec): min=130, max=26120, avg=1321.81, stdev=794.29
clat percentiles (usec):
| 1.00th=[ 359], 5.00th=[ 457], 10.00th=[ 529], 20.00th=[ 644],
| 30.00th=[ 775], 40.00th=[ 930], 50.00th=[ 1090], 60.00th=[ 1270],
| 70.00th=[ 1467], 80.00th=[ 1729], 90.00th=[ 2147], 95.00th=[ 2540],
| 99.00th=[ 3556], 99.50th=[ 4015], 99.90th=[ 7242], 99.95th=[ 9372],
| 99.99th=[15008]
bw ( KiB/s): min=21192, max=25464, per=12.55%, avg=23168.27, stdev=811.11, samples=119
iops : min= 5298, max= 6366, avg=5792.05, stdev=202.79, samples=119
write: IOPS=5776, BW=22.6MiB/s (23.7MB/s)(1354MiB/60001msec); 0 zone resets
slat (usec): min=2, max=13759, avg=72.72, stdev=191.51
clat (usec): min=18, max=25963, avg=1370.18, stdev=824.99
lat (usec): min=29, max=25972, avg=1443.12, stdev=856.69
clat percentiles (usec):
| 1.00th=[ 241], 5.00th=[ 388], 10.00th=[ 510], 20.00th=[ 717],
| 30.00th=[ 889], 40.00th=[ 1057], 50.00th=[ 1237], 60.00th=[ 1434],
| 70.00th=[ 1647], 80.00th=[ 1926], 90.00th=[ 2376], 95.00th=[ 2769],
| 99.00th=[ 3785], 99.50th=[ 4293], 99.90th=[ 7701], 99.95th=[ 9634],
| 99.99th=[15401]
bw ( KiB/s): min=21184, max=24848, per=12.52%, avg=23121.33, stdev=757.19, samples=119
iops : min= 5296, max= 6212, avg=5780.31, stdev=189.30, samples=119
lat (usec) : 20=0.01%, 50=0.01%, 100=0.02%, 250=0.60%, 500=8.10%
lat (usec) : 750=16.23%, 1000=15.47%
lat (msec) : 2=44.30%, 4=14.66%, 10=0.58%, 20=0.04%, 50=0.01%
cpu : usr=2.50%, sys=10.60%, ctx=253284, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=347273,346600,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=24: Sat Aug 6 04:17:40 2022
read: IOPS=5762, BW=22.5MiB/s (23.6MB/s)(1351MiB/60001msec)
slat (usec): min=2, max=14915, avg=78.09, stdev=206.74
clat (usec): min=91, max=25490, avg=1248.30, stdev=769.16
lat (usec): min=99, max=25497, avg=1326.61, stdev=808.28
clat percentiles (usec):
| 1.00th=[ 359], 5.00th=[ 457], 10.00th=[ 529], 20.00th=[ 644],
| 30.00th=[ 783], 40.00th=[ 930], 50.00th=[ 1090], 60.00th=[ 1270],
| 70.00th=[ 1467], 80.00th=[ 1729], 90.00th=[ 2147], 95.00th=[ 2540],
| 99.00th=[ 3589], 99.50th=[ 4113], 99.90th=[ 7504], 99.95th=[10552],
| 99.99th=[15008]
bw ( KiB/s): min=21336, max=24864, per=12.50%, avg=23074.92, stdev=768.03, samples=119
iops : min= 5334, max= 6216, avg=5768.71, stdev=191.99, samples=119
write: IOPS=5771, BW=22.5MiB/s (23.6MB/s)(1353MiB/60001msec); 0 zone resets
slat (usec): min=2, max=24173, avg=73.04, stdev=196.12
clat (usec): min=25, max=25916, avg=1372.03, stdev=841.90
lat (usec): min=33, max=25922, avg=1445.30, stdev=873.26
clat percentiles (usec):
| 1.00th=[ 231], 5.00th=[ 388], 10.00th=[ 510], 20.00th=[ 709],
| 30.00th=[ 889], 40.00th=[ 1057], 50.00th=[ 1237], 60.00th=[ 1434],
| 70.00th=[ 1647], 80.00th=[ 1926], 90.00th=[ 2376], 95.00th=[ 2769],
| 99.00th=[ 3785], 99.50th=[ 4293], 99.90th=[ 8094], 99.95th=[10945],
| 99.99th=[15795]
bw ( KiB/s): min=21264, max=25024, per=12.52%, avg=23111.38, stdev=749.41, samples=119
iops : min= 5316, max= 6256, avg=5777.82, stdev=187.35, samples=119
lat (usec) : 50=0.01%, 100=0.04%, 250=0.67%, 500=8.00%, 750=16.22%
lat (usec) : 1000=15.29%
lat (msec) : 2=44.53%, 4=14.59%, 10=0.60%, 20=0.06%, 50=0.01%
cpu : usr=2.62%, sys=10.44%, ctx=252283, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=345761,346309,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=25: Sat Aug 6 04:17:40 2022
read: IOPS=5767, BW=22.5MiB/s (23.6MB/s)(1352MiB/60001msec)
slat (usec): min=2, max=12043, avg=78.07, stdev=201.49
clat (usec): min=51, max=49366, avg=1247.54, stdev=793.44
lat (usec): min=114, max=49376, avg=1325.83, stdev=829.87
clat percentiles (usec):
| 1.00th=[ 359], 5.00th=[ 453], 10.00th=[ 529], 20.00th=[ 644],
| 30.00th=[ 783], 40.00th=[ 938], 50.00th=[ 1090], 60.00th=[ 1270],
| 70.00th=[ 1467], 80.00th=[ 1729], 90.00th=[ 2147], 95.00th=[ 2540],
| 99.00th=[ 3556], 99.50th=[ 4080], 99.90th=[ 7308], 99.95th=[ 9372],
| 99.99th=[15270]
bw ( KiB/s): min=21192, max=25064, per=12.50%, avg=23077.87, stdev=768.68, samples=119
iops : min= 5298, max= 6266, avg=5769.45, stdev=192.17, samples=119
write: IOPS=5776, BW=22.6MiB/s (23.7MB/s)(1354MiB/60001msec); 0 zone resets
slat (usec): min=2, max=46420, avg=72.77, stdev=205.48
clat (usec): min=29, max=48258, avg=1370.86, stdev=837.15
lat (usec): min=38, max=49191, avg=1443.88, stdev=871.73
clat percentiles (usec):
| 1.00th=[ 235], 5.00th=[ 383], 10.00th=[ 510], 20.00th=[ 709],
| 30.00th=[ 889], 40.00th=[ 1057], 50.00th=[ 1237], 60.00th=[ 1434],
| 70.00th=[ 1647], 80.00th=[ 1926], 90.00th=[ 2376], 95.00th=[ 2769],
| 99.00th=[ 3785], 99.50th=[ 4293], 99.90th=[ 7373], 99.95th=[ 9110],
| 99.99th=[13829]
bw ( KiB/s): min=21336, max=25224, per=12.52%, avg=23119.55, stdev=798.32, samples=119
iops : min= 5334, max= 6306, avg=5779.87, stdev=199.58, samples=119
lat (usec) : 50=0.01%, 100=0.03%, 250=0.63%, 500=8.16%, 750=16.08%
lat (usec) : 1000=15.32%
lat (msec) : 2=44.43%, 4=14.70%, 10=0.60%, 20=0.04%, 50=0.01%
cpu : usr=2.67%, sys=10.44%, ctx=253132, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=346028,346595,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=26: Sat Aug 6 04:17:40 2022
read: IOPS=5788, BW=22.6MiB/s (23.7MB/s)(1357MiB/60001msec)
slat (usec): min=2, max=13572, avg=77.31, stdev=200.86
clat (usec): min=57, max=21730, avg=1243.59, stdev=751.95
lat (usec): min=112, max=21739, avg=1321.12, stdev=790.32
clat percentiles (usec):
| 1.00th=[ 359], 5.00th=[ 457], 10.00th=[ 529], 20.00th=[ 644],
| 30.00th=[ 775], 40.00th=[ 930], 50.00th=[ 1090], 60.00th=[ 1270],
| 70.00th=[ 1467], 80.00th=[ 1729], 90.00th=[ 2147], 95.00th=[ 2540],
| 99.00th=[ 3556], 99.50th=[ 4080], 99.90th=[ 6915], 99.95th=[ 8848],
| 99.99th=[14222]
bw ( KiB/s): min=21502, max=24944, per=12.55%, avg=23167.70, stdev=647.51, samples=119
iops : min= 5375, max= 6236, avg=5791.90, stdev=161.91, samples=119
write: IOPS=5782, BW=22.6MiB/s (23.7MB/s)(1355MiB/60001msec); 0 zone resets
slat (usec): min=2, max=19344, avg=73.30, stdev=196.16
clat (usec): min=26, max=22437, avg=1368.55, stdev=826.12
lat (usec): min=33, max=22450, avg=1442.07, stdev=858.72
clat percentiles (usec):
| 1.00th=[ 235], 5.00th=[ 388], 10.00th=[ 510], 20.00th=[ 709],
| 30.00th=[ 889], 40.00th=[ 1057], 50.00th=[ 1221], 60.00th=[ 1418],
| 70.00th=[ 1647], 80.00th=[ 1926], 90.00th=[ 2376], 95.00th=[ 2769],
| 99.00th=[ 3818], 99.50th=[ 4359], 99.90th=[ 7570], 99.95th=[ 9503],
| 99.99th=[15270]
bw ( KiB/s): min=21021, max=25384, per=12.54%, avg=23146.14, stdev=718.22, samples=119
iops : min= 5255, max= 6346, avg=5786.52, stdev=179.56, samples=119
lat (usec) : 50=0.01%, 100=0.04%, 250=0.64%, 500=8.07%, 750=16.24%
lat (usec) : 1000=15.41%
lat (msec) : 2=44.41%, 4=14.51%, 10=0.63%, 20=0.03%, 50=0.01%
cpu : usr=2.81%, sys=10.33%, ctx=253180, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=347324,346957,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=27: Sat Aug 6 04:17:40 2022
read: IOPS=5745, BW=22.4MiB/s (23.5MB/s)(1347MiB/60001msec)
slat (usec): min=2, max=13611, avg=78.37, stdev=199.48
clat (usec): min=78, max=22265, avg=1251.95, stdev=785.95
lat (usec): min=128, max=22273, avg=1330.54, stdev=822.32
clat percentiles (usec):
| 1.00th=[ 359], 5.00th=[ 453], 10.00th=[ 529], 20.00th=[ 644],
| 30.00th=[ 775], 40.00th=[ 930], 50.00th=[ 1090], 60.00th=[ 1270],
| 70.00th=[ 1467], 80.00th=[ 1729], 90.00th=[ 2147], 95.00th=[ 2573],
| 99.00th=[ 3621], 99.50th=[ 4178], 99.90th=[ 7046], 99.95th=[ 9896],
| 99.99th=[18744]
bw ( KiB/s): min=20816, max=24447, per=12.46%, avg=22998.94, stdev=764.92, samples=119
iops : min= 5204, max= 6111, avg=5749.71, stdev=191.23, samples=119
write: IOPS=5746, BW=22.4MiB/s (23.5MB/s)(1347MiB/60001msec); 0 zone resets
slat (usec): min=2, max=18330, avg=73.44, stdev=204.56
clat (usec): min=29, max=22367, avg=1377.94, stdev=836.00
lat (usec): min=37, max=22730, avg=1451.60, stdev=870.65
clat percentiles (usec):
| 1.00th=[ 235], 5.00th=[ 388], 10.00th=[ 515], 20.00th=[ 717],
| 30.00th=[ 898], 40.00th=[ 1057], 50.00th=[ 1237], 60.00th=[ 1434],
| 70.00th=[ 1647], 80.00th=[ 1942], 90.00th=[ 2376], 95.00th=[ 2802],
| 99.00th=[ 3818], 99.50th=[ 4359], 99.90th=[ 7111], 99.95th=[ 9241],
| 99.99th=[18220]
bw ( KiB/s): min=20584, max=24432, per=12.46%, avg=22999.61, stdev=797.37, samples=119
iops : min= 5146, max= 6108, avg=5749.88, stdev=199.34, samples=119
lat (usec) : 50=0.01%, 100=0.03%, 250=0.64%, 500=8.04%, 750=16.14%
lat (usec) : 1000=15.22%
lat (msec) : 2=44.45%, 4=14.79%, 10=0.65%, 20=0.04%, 50=0.01%
cpu : usr=2.72%, sys=10.32%, ctx=252416, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=344747,344792,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=28: Sat Aug 6 04:17:40 2022
read: IOPS=5780, BW=22.6MiB/s (23.7MB/s)(1355MiB/60001msec)
slat (usec): min=2, max=13328, avg=77.62, stdev=197.43
clat (usec): min=107, max=22189, avg=1243.09, stdev=756.51
lat (usec): min=122, max=22195, avg=1320.93, stdev=793.13
clat percentiles (usec):
| 1.00th=[ 359], 5.00th=[ 453], 10.00th=[ 529], 20.00th=[ 644],
| 30.00th=[ 775], 40.00th=[ 930], 50.00th=[ 1090], 60.00th=[ 1270],
| 70.00th=[ 1467], 80.00th=[ 1713], 90.00th=[ 2147], 95.00th=[ 2540],
| 99.00th=[ 3556], 99.50th=[ 4080], 99.90th=[ 6783], 99.95th=[ 9503],
| 99.99th=[15795]
bw ( KiB/s): min=20488, max=25456, per=12.54%, avg=23136.32, stdev=865.21, samples=119
iops : min= 5122, max= 6364, avg=5784.06, stdev=216.31, samples=119
write: IOPS=5791, BW=22.6MiB/s (23.7MB/s)(1357MiB/60001msec); 0 zone resets
slat (usec): min=2, max=19426, avg=72.92, stdev=196.50
clat (usec): min=26, max=20441, avg=1368.88, stdev=815.03
lat (usec): min=34, max=20482, avg=1442.02, stdev=848.63
clat percentiles (usec):
| 1.00th=[ 235], 5.00th=[ 383], 10.00th=[ 510], 20.00th=[ 709],
| 30.00th=[ 889], 40.00th=[ 1057], 50.00th=[ 1237], 60.00th=[ 1434],
| 70.00th=[ 1647], 80.00th=[ 1926], 90.00th=[ 2343], 95.00th=[ 2769],
| 99.00th=[ 3752], 99.50th=[ 4293], 99.90th=[ 7373], 99.95th=[ 9372],
| 99.99th=[15270]
bw ( KiB/s): min=21064, max=24904, per=12.55%, avg=23179.55, stdev=772.73, samples=119
iops : min= 5266, max= 6226, avg=5794.87, stdev=193.19, samples=119
lat (usec) : 50=0.01%, 100=0.03%, 250=0.64%, 500=8.12%, 750=16.21%
lat (usec) : 1000=15.34%
lat (msec) : 2=44.45%, 4=14.56%, 10=0.60%, 20=0.04%, 50=0.01%
cpu : usr=2.77%, sys=10.34%, ctx=252919, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=346814,347473,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=180MiB/s (189MB/s), 22.4MiB/s-22.6MiB/s (23.5MB/s-23.7MB/s), io=10.6GiB (11.3GB), run=60001-60001msec
WRITE: bw=180MiB/s (189MB/s), 22.4MiB/s-22.6MiB/s (23.5MB/s-23.7MB/s), io=10.6GiB (11.3GB), run=60001-60001msec
Disk stats (read/write):
dm-0: ios=2763687/2766343, merge=0/0, ticks=1901436/2171824, in_queue=4073260, util=99.89%, aggrios=2765885/2766023, aggrmerge=2569/5039, aggrticks=1702438/1988307, aggrin_queue=3690848, aggrutil=99.80%
sda: ios=2765885/2766023, merge=2569/5039, ticks=1702438/1988307, in_queue=3690848, util=99.80%
Raw
benchmark-results-duplex.txt
$ kubectl exec -it fio -- fio --name=benchtest --size=800m --filename=/volume/test --direct=1 --rw=randrw --ioengine=libaio --bs=4k --iodepth=16 --numjobs=8 --time_based --runtime=60
benchtest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.30
Starting 8 processes
benchtest: Laying out IO file (1 file / 800MiB)
Jobs: 5 (f=0): [f(2),_(1),f(3),_(2)][100.0%][r=66.4MiB/s,w=66.9MiB/s][r=17.0k,w=17.1k IOPS][eta 00m:00s]
benchtest: (groupid=0, jobs=1): err= 0: pid=18: Sat Aug 6 04:04:46 2022
read: IOPS=2199, BW=8796KiB/s (9008kB/s)(515MiB/60003msec)
slat (nsec): min=1780, max=37979k, avg=175894.11, stdev=526366.70
clat (usec): min=57, max=43530, avg=2763.23, stdev=1956.61
lat (usec): min=127, max=43563, avg=2939.63, stdev=2032.43
clat percentiles (usec):
| 1.00th=[ 334], 5.00th=[ 519], 10.00th=[ 652], 20.00th=[ 922],
| 30.00th=[ 1483], 40.00th=[ 2278], 50.00th=[ 2868], 60.00th=[ 3130],
| 70.00th=[ 3490], 80.00th=[ 4015], 90.00th=[ 4817], 95.00th=[ 5538],
| 99.00th=[ 7373], 99.50th=[ 8979], 99.90th=[22676], 99.95th=[28181],
| 99.99th=[39584]
bw ( KiB/s): min= 7480, max=10112, per=12.46%, avg=8804.42, stdev=472.74, samples=119
iops : min= 1870, max= 2528, avg=2201.10, stdev=118.19, samples=119
write: IOPS=2201, BW=8804KiB/s (9016kB/s)(516MiB/60003msec); 0 zone resets
slat (usec): min=2, max=33395, avg=176.40, stdev=488.52
clat (usec): min=273, max=44003, avg=4149.81, stdev=1685.06
lat (usec): min=412, max=44064, avg=4326.70, stdev=1733.76
clat percentiles (usec):
| 1.00th=[ 2057], 5.00th=[ 2671], 10.00th=[ 2835], 20.00th=[ 3064],
| 30.00th=[ 3294], 40.00th=[ 3556], 50.00th=[ 3818], 60.00th=[ 4146],
| 70.00th=[ 4555], 80.00th=[ 5014], 90.00th=[ 5735], 95.00th=[ 6390],
| 99.00th=[ 8586], 99.50th=[11469], 99.90th=[24249], 99.95th=[29230],
| 99.99th=[41157]
bw ( KiB/s): min= 6880, max= 9776, per=12.45%, avg=8806.89, stdev=446.69, samples=119
iops : min= 1720, max= 2444, avg=2201.71, stdev=111.68, samples=119
lat (usec) : 100=0.01%, 250=0.12%, 500=2.08%, 750=4.94%, 1000=3.79%
lat (msec) : 2=7.99%, 4=48.48%, 10=32.08%, 20=0.36%, 50=0.16%
cpu : usr=1.51%, sys=11.37%, ctx=120329, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=131953,132072,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=19: Sat Aug 6 04:04:46 2022
read: IOPS=2199, BW=8796KiB/s (9007kB/s)(515MiB/60002msec)
slat (nsec): min=1955, max=29168k, avg=170788.80, stdev=480511.08
clat (usec): min=66, max=49793, avg=2755.76, stdev=1943.65
lat (usec): min=126, max=49809, avg=2926.98, stdev=2009.22
clat percentiles (usec):
| 1.00th=[ 334], 5.00th=[ 519], 10.00th=[ 652], 20.00th=[ 914],
| 30.00th=[ 1483], 40.00th=[ 2278], 50.00th=[ 2868], 60.00th=[ 3130],
| 70.00th=[ 3490], 80.00th=[ 4015], 90.00th=[ 4817], 95.00th=[ 5538],
| 99.00th=[ 7308], 99.50th=[ 8979], 99.90th=[21627], 99.95th=[28181],
| 99.99th=[37487]
bw ( KiB/s): min= 7336, max=10184, per=12.45%, avg=8800.13, stdev=438.04, samples=119
iops : min= 1834, max= 2546, avg=2200.03, stdev=109.52, samples=119
write: IOPS=2210, BW=8841KiB/s (9053kB/s)(518MiB/60002msec); 0 zone resets
slat (usec): min=2, max=33470, avg=180.38, stdev=526.40
clat (usec): min=103, max=49948, avg=4140.86, stdev=1687.35
lat (usec): min=631, max=49981, avg=4321.68, stdev=1743.49
clat percentiles (usec):
| 1.00th=[ 1991], 5.00th=[ 2671], 10.00th=[ 2835], 20.00th=[ 3064],
| 30.00th=[ 3294], 40.00th=[ 3523], 50.00th=[ 3818], 60.00th=[ 4146],
| 70.00th=[ 4490], 80.00th=[ 5014], 90.00th=[ 5735], 95.00th=[ 6390],
| 99.00th=[ 8455], 99.50th=[11338], 99.90th=[23462], 99.95th=[30802],
| 99.99th=[38536]
bw ( KiB/s): min= 7664, max= 9888, per=12.51%, avg=8848.26, stdev=417.67, samples=119
iops : min= 1916, max= 2472, avg=2212.06, stdev=104.43, samples=119
lat (usec) : 100=0.01%, 250=0.13%, 500=2.08%, 750=4.96%, 1000=3.86%
lat (msec) : 2=7.88%, 4=48.54%, 10=32.03%, 20=0.37%, 50=0.13%
cpu : usr=1.65%, sys=11.17%, ctx=120917, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=131945,132619,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=20: Sat Aug 6 04:04:46 2022
read: IOPS=2220, BW=8883KiB/s (9097kB/s)(521MiB/60002msec)
slat (nsec): min=1860, max=39441k, avg=171853.23, stdev=516504.05
clat (usec): min=13, max=51380, avg=2739.69, stdev=1898.28
lat (usec): min=121, max=51400, avg=2911.98, stdev=1976.64
clat percentiles (usec):
| 1.00th=[ 330], 5.00th=[ 515], 10.00th=[ 644], 20.00th=[ 906],
| 30.00th=[ 1467], 40.00th=[ 2245], 50.00th=[ 2835], 60.00th=[ 3130],
| 70.00th=[ 3458], 80.00th=[ 4015], 90.00th=[ 4817], 95.00th=[ 5538],
| 99.00th=[ 7373], 99.50th=[ 8848], 99.90th=[18744], 99.95th=[25297],
| 99.99th=[39584]
bw ( KiB/s): min= 7240, max=10352, per=12.58%, avg=8889.18, stdev=470.04, samples=119
iops : min= 1810, max= 2588, avg=2222.29, stdev=117.51, samples=119
write: IOPS=2213, BW=8855KiB/s (9067kB/s)(519MiB/60002msec); 0 zone resets
slat (usec): min=2, max=24744, avg=176.29, stdev=490.55
clat (usec): min=260, max=52198, avg=4124.40, stdev=1703.63
lat (usec): min=473, max=52241, avg=4301.14, stdev=1751.90
clat percentiles (usec):
| 1.00th=[ 1926], 5.00th=[ 2638], 10.00th=[ 2835], 20.00th=[ 3064],
| 30.00th=[ 3261], 40.00th=[ 3523], 50.00th=[ 3818], 60.00th=[ 4113],
| 70.00th=[ 4490], 80.00th=[ 4948], 90.00th=[ 5735], 95.00th=[ 6390],
| 99.00th=[ 8586], 99.50th=[11469], 99.90th=[23725], 99.95th=[29754],
| 99.99th=[44303]
bw ( KiB/s): min= 7632, max= 9624, per=12.53%, avg=8863.44, stdev=401.88, samples=119
iops : min= 1908, max= 2406, avg=2215.85, stdev=100.49, samples=119
lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.13%, 500=2.14%
lat (usec) : 750=5.06%, 1000=3.82%
lat (msec) : 2=8.07%, 4=48.92%, 10=31.34%, 20=0.41%, 50=0.10%
lat (msec) : 100=0.01%
cpu : usr=1.72%, sys=11.19%, ctx=120865, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=133255,132823,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=21: Sat Aug 6 04:04:46 2022
read: IOPS=2184, BW=8740KiB/s (8950kB/s)(512MiB/60002msec)
slat (nsec): min=1942, max=51047k, avg=174620.50, stdev=531093.41
clat (usec): min=28, max=96898, avg=2776.72, stdev=2159.65
lat (usec): min=119, max=96920, avg=2951.80, stdev=2229.59
clat percentiles (usec):
| 1.00th=[ 334], 5.00th=[ 519], 10.00th=[ 652], 20.00th=[ 914],
| 30.00th=[ 1500], 40.00th=[ 2278], 50.00th=[ 2868], 60.00th=[ 3130],
| 70.00th=[ 3490], 80.00th=[ 4015], 90.00th=[ 4817], 95.00th=[ 5538],
| 99.00th=[ 7439], 99.50th=[ 9372], 99.90th=[22414], 99.95th=[34341],
| 99.99th=[52167]
bw ( KiB/s): min= 6736, max=10064, per=12.38%, avg=8746.55, stdev=468.70, samples=119
iops : min= 1684, max= 2516, avg=2186.64, stdev=117.17, samples=119
write: IOPS=2196, BW=8784KiB/s (8995kB/s)(515MiB/60002msec); 0 zone resets
slat (usec): min=2, max=92274, avg=179.32, stdev=569.38
clat (usec): min=107, max=96914, avg=4163.69, stdev=1928.80
lat (usec): min=579, max=96985, avg=4343.45, stdev=1995.74
clat percentiles (usec):
| 1.00th=[ 2114], 5.00th=[ 2671], 10.00th=[ 2868], 20.00th=[ 3064],
| 30.00th=[ 3294], 40.00th=[ 3523], 50.00th=[ 3818], 60.00th=[ 4146],
| 70.00th=[ 4555], 80.00th=[ 5014], 90.00th=[ 5735], 95.00th=[ 6390],
| 99.00th=[ 8586], 99.50th=[11994], 99.90th=[25297], 99.95th=[37487],
| 99.99th=[52691]
bw ( KiB/s): min= 7016, max= 9960, per=12.42%, avg=8787.74, stdev=491.42, samples=119
iops : min= 1754, max= 2490, avg=2196.93, stdev=122.85, samples=119
lat (usec) : 50=0.01%, 100=0.01%, 250=0.13%, 500=2.07%, 750=4.90%
lat (usec) : 1000=3.86%
lat (msec) : 2=7.75%, 4=48.85%, 10=31.89%, 20=0.38%, 50=0.17%
lat (msec) : 100=0.01%
cpu : usr=1.59%, sys=11.23%, ctx=119709, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=131102,131768,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=22: Sat Aug 6 04:04:46 2022
read: IOPS=2207, BW=8828KiB/s (9040kB/s)(517MiB/60002msec)
slat (nsec): min=1978, max=42347k, avg=172863.76, stdev=521285.44
clat (usec): min=16, max=50882, avg=2751.10, stdev=1952.29
lat (usec): min=136, max=50893, avg=2924.41, stdev=2025.86
clat percentiles (usec):
| 1.00th=[ 334], 5.00th=[ 515], 10.00th=[ 652], 20.00th=[ 922],
| 30.00th=[ 1483], 40.00th=[ 2278], 50.00th=[ 2835], 60.00th=[ 3130],
| 70.00th=[ 3458], 80.00th=[ 3982], 90.00th=[ 4817], 95.00th=[ 5538],
| 99.00th=[ 7373], 99.50th=[ 9372], 99.90th=[20055], 99.95th=[30016],
| 99.99th=[41157]
bw ( KiB/s): min= 7344, max= 9904, per=12.50%, avg=8835.96, stdev=437.83, samples=119
iops : min= 1836, max= 2476, avg=2208.98, stdev=109.47, samples=119
write: IOPS=2213, BW=8853KiB/s (9065kB/s)(519MiB/60002msec); 0 zone resets
slat (usec): min=2, max=28332, avg=176.43, stdev=489.08
clat (usec): min=236, max=51046, avg=4130.57, stdev=1706.39
lat (usec): min=487, max=51971, avg=4307.56, stdev=1757.34
clat percentiles (usec):
| 1.00th=[ 2057], 5.00th=[ 2671], 10.00th=[ 2835], 20.00th=[ 3064],
| 30.00th=[ 3261], 40.00th=[ 3523], 50.00th=[ 3818], 60.00th=[ 4146],
| 70.00th=[ 4490], 80.00th=[ 5014], 90.00th=[ 5735], 95.00th=[ 6390],
| 99.00th=[ 8586], 99.50th=[11338], 99.90th=[22414], 99.95th=[33424],
| 99.99th=[46400]
bw ( KiB/s): min= 7416, max= 9880, per=12.52%, avg=8853.79, stdev=452.76, samples=119
iops : min= 1854, max= 2470, avg=2213.46, stdev=113.20, samples=119
lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.11%, 500=2.15%
lat (usec) : 750=4.92%, 1000=3.77%
lat (msec) : 2=7.93%, 4=49.24%, 10=31.32%, 20=0.44%, 50=0.11%
lat (msec) : 100=0.01%
cpu : usr=1.54%, sys=11.34%, ctx=120687, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=132425,132794,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=23: Sat Aug 6 04:04:46 2022
read: IOPS=2219, BW=8877KiB/s (9090kB/s)(520MiB/60003msec)
slat (nsec): min=1951, max=40372k, avg=171467.34, stdev=489761.88
clat (nsec): min=1216, max=55805k, avg=2738662.67, stdev=1929227.81
lat (usec): min=127, max=55815, avg=2910.57, stdev=1994.35
clat percentiles (usec):
| 1.00th=[ 334], 5.00th=[ 515], 10.00th=[ 652], 20.00th=[ 906],
| 30.00th=[ 1467], 40.00th=[ 2245], 50.00th=[ 2835], 60.00th=[ 3130],
| 70.00th=[ 3458], 80.00th=[ 3982], 90.00th=[ 4752], 95.00th=[ 5473],
| 99.00th=[ 7242], 99.50th=[ 8586], 99.90th=[21365], 99.95th=[25560],
| 99.99th=[46924]
bw ( KiB/s): min= 7126, max=10048, per=12.57%, avg=8883.33, stdev=476.21, samples=119
iops : min= 1781, max= 2512, avg=2220.82, stdev=119.07, samples=119
write: IOPS=2212, BW=8850KiB/s (9062kB/s)(519MiB/60003msec); 0 zone resets
slat (usec): min=2, max=45790, avg=176.20, stdev=519.78
clat (usec): min=260, max=55879, avg=4129.66, stdev=1751.69
lat (usec): min=607, max=55918, avg=4306.32, stdev=1806.72
clat percentiles (usec):
| 1.00th=[ 1942], 5.00th=[ 2671], 10.00th=[ 2835], 20.00th=[ 3064],
| 30.00th=[ 3261], 40.00th=[ 3523], 50.00th=[ 3818], 60.00th=[ 4113],
| 70.00th=[ 4490], 80.00th=[ 4948], 90.00th=[ 5735], 95.00th=[ 6456],
| 99.00th=[ 8586], 99.50th=[11600], 99.90th=[23987], 99.95th=[29230],
| 99.99th=[48497]
bw ( KiB/s): min= 7174, max= 9704, per=12.53%, avg=8860.67, stdev=419.57, samples=119
iops : min= 1793, max= 2426, avg=2215.16, stdev=104.91, samples=119
lat (usec) : 2=0.01%, 50=0.01%, 100=0.01%, 250=0.12%, 500=2.17%
lat (usec) : 750=5.02%, 1000=3.87%
lat (msec) : 2=7.97%, 4=48.97%, 10=31.39%, 20=0.34%, 50=0.15%
lat (msec) : 100=0.01%
cpu : usr=1.49%, sys=11.37%, ctx=120734, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=133162,132756,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=24: Sat Aug 6 04:04:46 2022
read: IOPS=2214, BW=8857KiB/s (9069kB/s)(519MiB/60002msec)
slat (nsec): min=1929, max=60909k, avg=173337.62, stdev=556587.22
clat (usec): min=41, max=65588, avg=2746.47, stdev=1963.01
lat (usec): min=142, max=65613, avg=2920.25, stdev=2046.05
clat percentiles (usec):
| 1.00th=[ 330], 5.00th=[ 519], 10.00th=[ 652], 20.00th=[ 914],
| 30.00th=[ 1483], 40.00th=[ 2278], 50.00th=[ 2835], 60.00th=[ 3130],
| 70.00th=[ 3490], 80.00th=[ 4015], 90.00th=[ 4817], 95.00th=[ 5473],
| 99.00th=[ 7308], 99.50th=[ 8848], 99.90th=[21365], 99.95th=[27395],
| 99.99th=[53216]
bw ( KiB/s): min= 7464, max= 9712, per=12.53%, avg=8857.34, stdev=381.18, samples=119
iops : min= 1866, max= 2428, avg=2214.33, stdev=95.30, samples=119
write: IOPS=2211, BW=8848KiB/s (9060kB/s)(518MiB/60002msec); 0 zone resets
slat (usec): min=2, max=35896, avg=176.39, stdev=490.65
clat (usec): min=257, max=65606, avg=4127.12, stdev=1716.21
lat (usec): min=572, max=65638, avg=4303.93, stdev=1767.58
clat percentiles (usec):
| 1.00th=[ 2040], 5.00th=[ 2671], 10.00th=[ 2835], 20.00th=[ 3064],
| 30.00th=[ 3294], 40.00th=[ 3523], 50.00th=[ 3818], 60.00th=[ 4146],
| 70.00th=[ 4490], 80.00th=[ 5014], 90.00th=[ 5669], 95.00th=[ 6390],
| 99.00th=[ 8455], 99.50th=[11207], 99.90th=[23725], 99.95th=[28705],
| 99.99th=[55313]
bw ( KiB/s): min= 7336, max= 9728, per=12.51%, avg=8851.02, stdev=407.11, samples=119
iops : min= 1834, max= 2432, avg=2212.75, stdev=101.78, samples=119
lat (usec) : 50=0.01%, 100=0.01%, 250=0.12%, 500=2.12%, 750=4.95%
lat (usec) : 1000=3.79%
lat (msec) : 2=8.00%, 4=48.97%, 10=31.53%, 20=0.37%, 50=0.14%
lat (msec) : 100=0.01%
cpu : usr=1.60%, sys=11.30%, ctx=120845, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=132852,132721,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=25: Sat Aug 6 04:04:46 2022
read: IOPS=2223, BW=8893KiB/s (9107kB/s)(521MiB/60001msec)
slat (nsec): min=1903, max=35494k, avg=170435.56, stdev=502390.72
clat (usec): min=29, max=65089, avg=2729.10, stdev=1929.57
lat (usec): min=134, max=65101, avg=2899.98, stdev=2002.69
clat percentiles (usec):
| 1.00th=[ 330], 5.00th=[ 510], 10.00th=[ 644], 20.00th=[ 898],
| 30.00th=[ 1467], 40.00th=[ 2212], 50.00th=[ 2835], 60.00th=[ 3130],
| 70.00th=[ 3458], 80.00th=[ 3982], 90.00th=[ 4752], 95.00th=[ 5473],
| 99.00th=[ 7242], 99.50th=[ 8848], 99.90th=[20055], 99.95th=[25560],
| 99.99th=[40109]
bw ( KiB/s): min= 6685, max=10480, per=12.60%, avg=8902.40, stdev=486.58, samples=119
iops : min= 1671, max= 2620, avg=2225.60, stdev=121.65, samples=119
write: IOPS=2223, BW=8894KiB/s (9108kB/s)(521MiB/60001msec); 0 zone resets
slat (usec): min=2, max=40101, avg=177.08, stdev=512.88
clat (usec): min=103, max=67191, avg=4112.39, stdev=1692.37
lat (usec): min=492, max=67203, avg=4289.89, stdev=1748.67
clat percentiles (usec):
| 1.00th=[ 1827], 5.00th=[ 2638], 10.00th=[ 2835], 20.00th=[ 3064],
| 30.00th=[ 3261], 40.00th=[ 3523], 50.00th=[ 3818], 60.00th=[ 4146],
| 70.00th=[ 4490], 80.00th=[ 4948], 90.00th=[ 5735], 95.00th=[ 6390],
| 99.00th=[ 8356], 99.50th=[10814], 99.90th=[22938], 99.95th=[27657],
| 99.99th=[45876]
bw ( KiB/s): min= 7599, max=10760, per=12.58%, avg=8899.47, stdev=412.23, samples=119
iops : min= 1899, max= 2690, avg=2224.86, stdev=103.08, samples=119
lat (usec) : 50=0.01%, 100=0.01%, 250=0.14%, 500=2.18%, 750=5.03%
lat (usec) : 1000=3.89%
lat (msec) : 2=8.18%, 4=48.63%, 10=31.46%, 20=0.36%, 50=0.12%
lat (msec) : 100=0.01%
cpu : usr=1.57%, sys=11.35%, ctx=121076, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=133404,133415,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=69.0MiB/s (72.4MB/s), 8740KiB/s-8893KiB/s (8950kB/s-9107kB/s), io=4141MiB (4342MB), run=60001-60003msec
WRITE: bw=69.1MiB/s (72.4MB/s), 8784KiB/s-8894KiB/s (8995kB/s-9108kB/s), io=4144MiB (4346MB), run=60001-60003msec
Disk stats (read/write):
nvme0n1: ios=1057679/1058469, merge=0/18, ticks=1884787/3027440, in_queue=4912233, util=99.87%
Raw
benchmark-results-iscsi-synology.txt
$ kubectl exec -it fio-iscsi -- fio --name=benchtest --size=800m --filename=/volume/test --direct=1 --rw=randrw --ioengine=libaio --bs=4k --iodepth=16 --numjobs=8 --time_based --runtime=60
benchtest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.30
Starting 8 processes
benchtest: Laying out IO file (1 file / 800MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=1617KiB/s,w=1469KiB/s][r=404,w=367 IOPS][eta 00m:00s]
benchtest: (groupid=0, jobs=1): err= 0: pid=30: Sat Aug 6 22:48:16 2022
read: IOPS=54, BW=217KiB/s (222kB/s)(12.8MiB/60238msec)
slat (usec): min=3, max=82377, avg=121.66, stdev=2473.63
clat (usec): min=132, max=2179.7k, avg=58085.92, stdev=131276.03
lat (usec): min=694, max=2179.7k, avg=58208.89, stdev=131335.22
clat percentiles (usec):
| 1.00th=[ 1123], 5.00th=[ 1827], 10.00th=[ 2409],
| 20.00th=[ 3949], 30.00th=[ 6325], 40.00th=[ 10683],
| 50.00th=[ 16319], 60.00th=[ 24773], 70.00th=[ 39060],
| 80.00th=[ 69731], 90.00th=[ 143655], 95.00th=[ 246416],
| 99.00th=[ 641729], 99.50th=[ 809501], 99.90th=[1216349],
| 99.95th=[2164261], 99.99th=[2164261]
bw ( KiB/s): min= 8, max= 592, per=12.97%, avg=228.91, stdev=155.60, samples=114
iops : min= 2, max= 148, avg=57.23, stdev=38.90, samples=114
write: IOPS=56, BW=225KiB/s (230kB/s)(13.2MiB/60238msec); 0 zone resets
slat (usec): min=3, max=269057, avg=93.58, stdev=4627.71
clat (usec): min=677, max=4075.9k, avg=228383.02, stdev=323225.58
lat (usec): min=684, max=4075.9k, avg=228477.93, stdev=323275.96
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 10], 10.00th=[ 22], 20.00th=[ 47],
| 30.00th=[ 77], 40.00th=[ 106], 50.00th=[ 140], 60.00th=[ 176],
| 70.00th=[ 226], 80.00th=[ 309], 90.00th=[ 506], 95.00th=[ 718],
| 99.00th=[ 1737], 99.50th=[ 2366], 99.90th=[ 2836], 99.95th=[ 3440],
| 99.99th=[ 4077]
bw ( KiB/s): min= 16, max= 656, per=13.47%, avg=240.86, stdev=156.53, samples=112
iops : min= 4, max= 164, avg=60.21, stdev=39.13, samples=112
lat (usec) : 250=0.05%, 750=0.06%, 1000=0.27%
lat (msec) : 2=3.08%, 4=7.91%, 10=10.41%, 20=9.94%, 50=15.82%
lat (msec) : 100=14.09%, 250=22.44%, 500=10.03%, 750=3.20%, 1000=1.14%
lat (msec) : 2000=1.11%, >=2000=0.45%
cpu : usr=0.05%, sys=0.26%, ctx=6386, majf=0, minf=15
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3264,3386,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=31: Sat Aug 6 22:48:16 2022
read: IOPS=52, BW=210KiB/s (215kB/s)(12.4MiB/60517msec)
slat (usec): min=3, max=68984, avg=52.04, stdev=1553.53
clat (usec): min=149, max=2529.7k, avg=56011.02, stdev=127016.54
lat (usec): min=513, max=2529.7k, avg=56064.38, stdev=127007.88
clat percentiles (usec):
| 1.00th=[ 1254], 5.00th=[ 1942], 10.00th=[ 2737],
| 20.00th=[ 4424], 30.00th=[ 7373], 40.00th=[ 11863],
| 50.00th=[ 17957], 60.00th=[ 26870], 70.00th=[ 41157],
| 80.00th=[ 70779], 90.00th=[ 137364], 95.00th=[ 229639],
| 99.00th=[ 549454], 99.50th=[ 725615], 99.90th=[1484784],
| 99.95th=[2365588], 99.99th=[2533360]
bw ( KiB/s): min= 8, max= 680, per=12.69%, avg=223.16, stdev=154.71, samples=114
iops : min= 2, max= 170, avg=55.79, stdev=38.68, samples=114
write: IOPS=55, BW=221KiB/s (226kB/s)(13.1MiB/60517msec); 0 zone resets
slat (usec): min=3, max=292948, avg=153.07, stdev=5339.05
clat (usec): min=320, max=3518.9k, avg=235787.91, stdev=353691.52
lat (usec): min=330, max=3518.9k, avg=235942.28, stdev=353664.72
clat percentiles (usec):
| 1.00th=[ 1696], 5.00th=[ 9241], 10.00th=[ 21365],
| 20.00th=[ 46924], 30.00th=[ 76022], 40.00th=[ 105382],
| 50.00th=[ 139461], 60.00th=[ 175113], 70.00th=[ 227541],
| 80.00th=[ 312476], 90.00th=[ 505414], 95.00th=[ 759170],
| 99.00th=[2164261], 99.50th=[2701132], 99.90th=[3405775],
| 99.95th=[3506439], 99.99th=[3506439]
bw ( KiB/s): min= 8, max= 624, per=13.30%, avg=237.93, stdev=153.99, samples=112
iops : min= 2, max= 156, avg=59.48, stdev=38.50, samples=112
lat (usec) : 250=0.05%, 500=0.03%, 750=0.05%, 1000=0.25%
lat (msec) : 2=2.85%, 4=6.74%, 10=10.20%, 20=10.02%, 50=16.76%
lat (msec) : 100=14.37%, 250=22.41%, 500=10.40%, 750=3.05%, 1000=1.33%
lat (msec) : 2000=0.87%, >=2000=0.61%
cpu : usr=0.05%, sys=0.26%, ctx=6281, majf=0, minf=17
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3181,3346,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=32: Sat Aug 6 22:48:16 2022
read: IOPS=54, BW=218KiB/s (223kB/s)(12.9MiB/60704msec)
slat (usec): min=3, max=183372, avg=76.98, stdev=3226.67
clat (usec): min=114, max=2226.1k, avg=55942.89, stdev=121439.51
lat (usec): min=597, max=2226.1k, avg=56021.19, stdev=121470.03
clat percentiles (usec):
| 1.00th=[ 1106], 5.00th=[ 1696], 10.00th=[ 2409],
| 20.00th=[ 4080], 30.00th=[ 6783], 40.00th=[ 10945],
| 50.00th=[ 17171], 60.00th=[ 26084], 70.00th=[ 40109],
| 80.00th=[ 70779], 90.00th=[ 137364], 95.00th=[ 233833],
| 99.00th=[ 549454], 99.50th=[ 784335], 99.90th=[1182794],
| 99.95th=[1853883], 99.99th=[2231370]
bw ( KiB/s): min= 24, max= 712, per=13.31%, avg=234.12, stdev=161.54, samples=113
iops : min= 6, max= 178, avg=58.53, stdev=40.39, samples=113
write: IOPS=56, BW=225KiB/s (231kB/s)(13.3MiB/60704msec); 0 zone resets
slat (usec): min=3, max=79829, avg=101.83, stdev=2238.51
clat (usec): min=458, max=3735.6k, avg=229661.16, stdev=346877.19
lat (usec): min=673, max=3735.6k, avg=229764.31, stdev=346829.80
clat percentiles (usec):
| 1.00th=[ 1369], 5.00th=[ 6980], 10.00th=[ 17695],
| 20.00th=[ 43254], 30.00th=[ 70779], 40.00th=[ 101188],
| 50.00th=[ 133694], 60.00th=[ 173016], 70.00th=[ 221250],
| 80.00th=[ 308282], 90.00th=[ 492831], 95.00th=[ 759170],
| 99.00th=[2071987], 99.50th=[2634023], 99.90th=[3170894],
| 99.95th=[3741320], 99.99th=[3741320]
bw ( KiB/s): min= 16, max= 672, per=13.36%, avg=238.95, stdev=164.71, samples=114
iops : min= 4, max= 168, avg=59.74, stdev=41.18, samples=114
lat (usec) : 250=0.03%, 500=0.01%, 750=0.07%, 1000=0.25%
lat (msec) : 2=4.06%, 4=6.75%, 10=10.79%, 20=9.87%, 50=16.08%
lat (msec) : 100=14.17%, 250=22.28%, 500=10.03%, 750=2.71%, 1000=1.09%
lat (msec) : 2000=1.25%, >=2000=0.55%
cpu : usr=0.03%, sys=0.28%, ctx=6523, majf=0, minf=15
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3310,3417,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=33: Sat Aug 6 22:48:16 2022
read: IOPS=55, BW=221KiB/s (226kB/s)(13.0MiB/60166msec)
slat (usec): min=3, max=219922, avg=123.40, stdev=4131.40
clat (usec): min=150, max=2566.6k, avg=60385.85, stdev=151973.53
lat (usec): min=662, max=2566.6k, avg=60510.56, stdev=152105.56
clat percentiles (usec):
| 1.00th=[ 1188], 5.00th=[ 1876], 10.00th=[ 2638],
| 20.00th=[ 4293], 30.00th=[ 7046], 40.00th=[ 11469],
| 50.00th=[ 17433], 60.00th=[ 26608], 70.00th=[ 41681],
| 80.00th=[ 71828], 90.00th=[ 141558], 95.00th=[ 242222],
| 99.00th=[ 675283], 99.50th=[1044382], 99.90th=[2197816],
| 99.95th=[2533360], 99.99th=[2566915]
bw ( KiB/s): min= 8, max= 656, per=13.20%, avg=232.91, stdev=159.29, samples=114
iops : min= 2, max= 164, avg=58.23, stdev=39.82, samples=114
write: IOPS=55, BW=223KiB/s (228kB/s)(13.1MiB/60166msec); 0 zone resets
slat (usec): min=3, max=62522, avg=77.97, stdev=1775.83
clat (usec): min=527, max=4394.6k, avg=227443.17, stdev=329158.02
lat (usec): min=680, max=4394.7k, avg=227522.47, stdev=329131.07
clat percentiles (usec):
| 1.00th=[ 1516], 5.00th=[ 8094], 10.00th=[ 20317],
| 20.00th=[ 46400], 30.00th=[ 72877], 40.00th=[ 107480],
| 50.00th=[ 139461], 60.00th=[ 177210], 70.00th=[ 227541],
| 80.00th=[ 308282], 90.00th=[ 492831], 95.00th=[ 708838],
| 99.00th=[1820328], 99.50th=[2499806], 99.90th=[2969568],
| 99.95th=[3372221], 99.99th=[4395631]
bw ( KiB/s): min= 16, max= 648, per=13.25%, avg=236.04, stdev=154.71, samples=113
iops : min= 4, max= 162, avg=59.01, stdev=38.68, samples=113
lat (usec) : 250=0.04%, 750=0.07%, 1000=0.31%
lat (msec) : 2=3.37%, 4=6.90%, 10=10.66%, 20=9.81%, 50=16.31%
lat (msec) : 100=14.01%, 250=22.73%, 500=10.35%, 750=2.61%, 1000=1.32%
lat (msec) : 2000=0.94%, >=2000=0.55%
cpu : usr=0.05%, sys=0.27%, ctx=6408, majf=0, minf=17
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3322,3347,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=34: Sat Aug 6 22:48:16 2022
read: IOPS=56, BW=225KiB/s (230kB/s)(13.2MiB/60422msec)
slat (usec): min=3, max=58022, avg=37.16, stdev=1153.64
clat (usec): min=223, max=2293.1k, avg=61384.54, stdev=144999.76
lat (usec): min=685, max=2293.1k, avg=61423.03, stdev=144996.40
clat percentiles (usec):
| 1.00th=[ 1221], 5.00th=[ 1844], 10.00th=[ 2573],
| 20.00th=[ 4293], 30.00th=[ 6849], 40.00th=[ 11469],
| 50.00th=[ 17957], 60.00th=[ 26870], 70.00th=[ 41681],
| 80.00th=[ 74974], 90.00th=[ 145753], 95.00th=[ 250610],
| 99.00th=[ 658506], 99.50th=[ 977273], 99.90th=[2139096],
| 99.95th=[2231370], 99.99th=[2298479]
bw ( KiB/s): min= 8, max= 712, per=13.48%, avg=237.54, stdev=164.24, samples=114
iops : min= 2, max= 178, avg=59.39, stdev=41.06, samples=114
write: IOPS=55, BW=221KiB/s (226kB/s)(13.0MiB/60422msec); 0 zone resets
slat (usec): min=3, max=185785, avg=161.03, stdev=4073.50
clat (usec): min=474, max=3906.2k, avg=226861.27, stdev=325058.53
lat (usec): min=631, max=3906.2k, avg=227023.62, stdev=325034.80
clat percentiles (usec):
| 1.00th=[ 1696], 5.00th=[ 8848], 10.00th=[ 22152],
| 20.00th=[ 47449], 30.00th=[ 72877], 40.00th=[ 102237],
| 50.00th=[ 137364], 60.00th=[ 177210], 70.00th=[ 229639],
| 80.00th=[ 316670], 90.00th=[ 488637], 95.00th=[ 734004],
| 99.00th=[1820328], 99.50th=[2298479], 99.90th=[3271558],
| 99.95th=[3607102], 99.99th=[3909092]
bw ( KiB/s): min= 8, max= 664, per=13.08%, avg=233.68, stdev=155.07, samples=114
iops : min= 2, max= 166, avg=58.42, stdev=38.77, samples=114
lat (usec) : 250=0.01%, 500=0.01%, 750=0.04%, 1000=0.24%
lat (msec) : 2=3.43%, 4=6.94%, 10=10.83%, 20=9.54%, 50=16.37%
lat (msec) : 100=14.72%, 250=21.79%, 500=10.50%, 750=2.78%, 1000=1.34%
lat (msec) : 2000=0.94%, >=2000=0.51%
cpu : usr=0.08%, sys=0.23%, ctx=6508, majf=0, minf=15
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3392,3339,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=35: Sat Aug 6 22:48:16 2022
read: IOPS=57, BW=230KiB/s (235kB/s)(13.6MiB/60667msec)
slat (usec): min=3, max=43167, avg=25.24, stdev=765.36
clat (usec): min=136, max=2212.1k, avg=57582.13, stdev=130563.53
lat (usec): min=513, max=2212.1k, avg=57608.69, stdev=130558.83
clat percentiles (usec):
| 1.00th=[ 1205], 5.00th=[ 1876], 10.00th=[ 2540],
| 20.00th=[ 4228], 30.00th=[ 6783], 40.00th=[ 10945],
| 50.00th=[ 16581], 60.00th=[ 25297], 70.00th=[ 39060],
| 80.00th=[ 69731], 90.00th=[ 149947], 95.00th=[ 248513],
| 99.00th=[ 583009], 99.50th=[ 901776], 99.90th=[1736442],
| 99.95th=[2164261], 99.99th=[2197816]
bw ( KiB/s): min= 24, max= 736, per=13.88%, avg=244.49, stdev=170.63, samples=114
iops : min= 6, max= 184, avg=61.12, stdev=42.66, samples=114
write: IOPS=56, BW=224KiB/s (230kB/s)(13.3MiB/60667msec); 0 zone resets
slat (usec): min=3, max=73024, avg=118.74, stdev=2442.50
clat (usec): min=457, max=3270.0k, avg=226124.35, stdev=316182.64
lat (usec): min=779, max=3270.0k, avg=226244.43, stdev=316122.21
clat percentiles (usec):
| 1.00th=[ 1582], 5.00th=[ 8979], 10.00th=[ 20579],
| 20.00th=[ 45876], 30.00th=[ 71828], 40.00th=[ 103285],
| 50.00th=[ 135267], 60.00th=[ 173016], 70.00th=[ 227541],
| 80.00th=[ 308282], 90.00th=[ 505414], 95.00th=[ 742392],
| 99.00th=[1719665], 99.50th=[2298479], 99.90th=[2835350],
| 99.95th=[2902459], 99.99th=[3271558]
bw ( KiB/s): min= 6, max= 616, per=13.19%, avg=235.60, stdev=152.56, samples=115
iops : min= 1, max= 154, avg=58.90, stdev=38.15, samples=115
lat (usec) : 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.20%
lat (msec) : 2=3.37%, 4=7.23%, 10=10.89%, 20=10.57%, 50=16.13%
lat (msec) : 100=14.20%, 250=21.59%, 500=10.02%, 750=2.98%, 1000=1.35%
lat (msec) : 2000=1.02%, >=2000=0.38%
cpu : usr=0.05%, sys=0.27%, ctx=6647, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3486,3400,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=36: Sat Aug 6 22:48:16 2022
read: IOPS=57, BW=230KiB/s (235kB/s)(13.5MiB/60222msec)
slat (usec): min=3, max=75795, avg=82.61, stdev=1958.31
clat (usec): min=25, max=2367.2k, avg=55258.74, stdev=123050.59
lat (usec): min=561, max=2367.2k, avg=55342.66, stdev=123056.62
clat percentiles (usec):
| 1.00th=[ 1188], 5.00th=[ 1795], 10.00th=[ 2474],
| 20.00th=[ 3884], 30.00th=[ 6259], 40.00th=[ 10028],
| 50.00th=[ 16319], 60.00th=[ 25035], 70.00th=[ 39060],
| 80.00th=[ 72877], 90.00th=[ 141558], 95.00th=[ 229639],
| 99.00th=[ 557843], 99.50th=[ 851444], 99.90th=[1300235],
| 99.95th=[2231370], 99.99th=[2365588]
bw ( KiB/s): min= 8, max= 680, per=13.77%, avg=242.67, stdev=167.79, samples=114
iops : min= 2, max= 170, avg=60.67, stdev=41.95, samples=114
write: IOPS=57, BW=229KiB/s (235kB/s)(13.5MiB/60222msec); 0 zone resets
slat (usec): min=3, max=305538, avg=129.48, stdev=5292.14
clat (usec): min=685, max=4103.3k, avg=223379.80, stdev=324615.66
lat (usec): min=720, max=4103.3k, avg=223510.59, stdev=324628.92
clat percentiles (usec):
| 1.00th=[ 1565], 5.00th=[ 6325], 10.00th=[ 18220],
| 20.00th=[ 43254], 30.00th=[ 72877], 40.00th=[ 103285],
| 50.00th=[ 135267], 60.00th=[ 175113], 70.00th=[ 225444],
| 80.00th=[ 308282], 90.00th=[ 492831], 95.00th=[ 692061],
| 99.00th=[1736442], 99.50th=[2365588], 99.90th=[3439330],
| 99.95th=[3909092], 99.99th=[4110418]
bw ( KiB/s): min= 8, max= 680, per=13.53%, avg=241.47, stdev=161.30, samples=114
iops : min= 2, max= 170, avg=60.37, stdev=40.32, samples=114
lat (usec) : 50=0.01%, 250=0.06%, 750=0.03%, 1000=0.32%
lat (msec) : 2=3.66%, 4=7.87%, 10=11.28%, 20=9.69%, 50=15.66%
lat (msec) : 100=13.61%, 250=22.27%, 500=10.15%, 750=2.92%, 1000=1.23%
lat (msec) : 2000=0.82%, >=2000=0.42%
cpu : usr=0.03%, sys=0.30%, ctx=6670, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3462,3453,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=37: Sat Aug 6 22:48:16 2022
read: IOPS=54, BW=216KiB/s (222kB/s)(12.7MiB/60165msec)
slat (usec): min=3, max=286493, avg=158.62, stdev=5304.19
clat (usec): min=78, max=2529.0k, avg=59955.13, stdev=153345.02
lat (usec): min=565, max=2529.1k, avg=60115.09, stdev=153421.22
clat percentiles (usec):
| 1.00th=[ 1123], 5.00th=[ 1795], 10.00th=[ 2474],
| 20.00th=[ 4178], 30.00th=[ 6849], 40.00th=[ 11207],
| 50.00th=[ 17695], 60.00th=[ 26346], 70.00th=[ 40109],
| 80.00th=[ 69731], 90.00th=[ 143655], 95.00th=[ 244319],
| 99.00th=[ 583009], 99.50th=[ 960496], 99.90th=[2264925],
| 99.95th=[2332034], 99.99th=[2533360]
bw ( KiB/s): min= 8, max= 672, per=12.97%, avg=228.42, stdev=158.97, samples=114
iops : min= 2, max= 168, avg=57.11, stdev=39.74, samples=114
write: IOPS=55, BW=223KiB/s (228kB/s)(13.1MiB/60165msec); 0 zone resets
slat (usec): min=3, max=68694, avg=62.88, stdev=1590.43
clat (usec): min=728, max=5095.8k, avg=228688.18, stdev=327453.83
lat (usec): min=736, max=5095.8k, avg=228752.35, stdev=327426.81
clat percentiles (usec):
| 1.00th=[ 1680], 5.00th=[ 9765], 10.00th=[ 21365],
| 20.00th=[ 46400], 30.00th=[ 76022], 40.00th=[ 104334],
| 50.00th=[ 139461], 60.00th=[ 177210], 70.00th=[ 231736],
| 80.00th=[ 316670], 90.00th=[ 501220], 95.00th=[ 717226],
| 99.00th=[1736442], 99.50th=[2365588], 99.90th=[3070231],
| 99.95th=[3305112], 99.99th=[5066720]
bw ( KiB/s): min= 16, max= 664, per=13.13%, avg=234.18, stdev=156.38, samples=114
iops : min= 4, max= 166, avg=58.54, stdev=39.10, samples=114
lat (usec) : 100=0.02%, 250=0.05%, 750=0.05%, 1000=0.32%
lat (msec) : 2=3.40%, 4=6.78%, 10=10.29%, 20=9.88%, 50=16.40%
lat (msec) : 100=14.72%, 250=21.67%, 500=10.58%, 750=3.09%, 1000=1.33%
lat (msec) : 2000=0.92%, >=2000=0.50%
cpu : usr=0.07%, sys=0.23%, ctx=6380, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3256,3352,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=1758KiB/s (1800kB/s), 210KiB/s-230KiB/s (215kB/s-235kB/s), io=104MiB (109MB), run=60165-60704msec
WRITE: bw=1782KiB/s (1825kB/s), 221KiB/s-229KiB/s (226kB/s-235kB/s), io=106MiB (111MB), run=60165-60704msec
Disk stats (read/write):
sda: ios=26670/27195, merge=1/55, ticks=1541600/6192986, in_queue=7734586, util=99.09%
Raw
benchmark-results-simplex.txt
kubectl exec -it fio -- fio --name=benchtest --size=800m --filename=/volume/test --direct=1 --rw=randrw --ioengine=libaio --bs=4k --iodepth=16 --numjobs=8 --time_based --runtime=60
benchtest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.30
Starting 8 processes
benchtest: Laying out IO file (1 file / 800MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=73.1MiB/s,w=73.2MiB/s][r=18.7k,w=18.7k IOPS][eta 00m:00s]
benchtest: (groupid=0, jobs=1): err= 0: pid=18: Sat Aug 6 04:08:58 2022
read: IOPS=2282, BW=9131KiB/s (9350kB/s)(535MiB/60003msec)
slat (nsec): min=1868, max=19811k, avg=166560.40, stdev=402370.22
clat (usec): min=227, max=39169, avg=3297.30, stdev=1425.57
lat (usec): min=388, max=39189, avg=3464.29, stdev=1459.97
clat percentiles (usec):
| 1.00th=[ 1336], 5.00th=[ 2040], 10.00th=[ 2212], 20.00th=[ 2376],
| 30.00th=[ 2540], 40.00th=[ 2737], 50.00th=[ 2999], 60.00th=[ 3294],
| 70.00th=[ 3621], 80.00th=[ 4080], 90.00th=[ 4752], 95.00th=[ 5407],
| 99.00th=[ 7242], 99.50th=[ 9241], 99.90th=[19268], 99.95th=[23725],
| 99.99th=[29492]
bw ( KiB/s): min= 7704, max=10208, per=12.51%, avg=9135.41, stdev=506.42, samples=119
iops : min= 1926, max= 2552, avg=2283.84, stdev=126.61, samples=119
write: IOPS=2284, BW=9140KiB/s (9359kB/s)(536MiB/60003msec); 0 zone resets
slat (usec): min=2, max=33430, avg=170.78, stdev=419.72
clat (usec): min=189, max=38106, avg=3365.33, stdev=1451.04
lat (usec): min=516, max=39202, avg=3536.53, stdev=1488.57
clat percentiles (usec):
| 1.00th=[ 1369], 5.00th=[ 2040], 10.00th=[ 2212], 20.00th=[ 2376],
| 30.00th=[ 2573], 40.00th=[ 2835], 50.00th=[ 3097], 60.00th=[ 3359],
| 70.00th=[ 3720], 80.00th=[ 4146], 90.00th=[ 4817], 95.00th=[ 5473],
| 99.00th=[ 7308], 99.50th=[ 9372], 99.90th=[20579], 99.95th=[23987],
| 99.99th=[30278]
bw ( KiB/s): min= 7576, max=10266, per=12.51%, avg=9142.58, stdev=501.04, samples=119
iops : min= 1894, max= 2566, avg=2285.63, stdev=125.25, samples=119
lat (usec) : 250=0.01%, 500=0.01%, 750=0.07%, 1000=0.27%
lat (msec) : 2=3.86%, 4=73.47%, 10=21.90%, 20=0.32%, 50=0.10%
cpu : usr=1.68%, sys=11.40%, ctx=160142, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=136972,137102,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=19: Sat Aug 6 04:08:58 2022
read: IOPS=2276, BW=9106KiB/s (9325kB/s)(534MiB/60001msec)
slat (nsec): min=1944, max=19388k, avg=166724.75, stdev=407091.51
clat (usec): min=4, max=64528, avg=3298.77, stdev=1465.03
lat (usec): min=440, max=64548, avg=3465.92, stdev=1500.14
clat percentiles (usec):
| 1.00th=[ 1450], 5.00th=[ 2057], 10.00th=[ 2212], 20.00th=[ 2376],
| 30.00th=[ 2540], 40.00th=[ 2737], 50.00th=[ 2999], 60.00th=[ 3261],
| 70.00th=[ 3621], 80.00th=[ 4047], 90.00th=[ 4752], 95.00th=[ 5407],
| 99.00th=[ 7242], 99.50th=[ 8848], 99.90th=[18482], 99.95th=[21103],
| 99.99th=[52167]
bw ( KiB/s): min= 7720, max= 9992, per=12.47%, avg=9108.35, stdev=492.16, samples=119
iops : min= 1930, max= 2498, avg=2277.07, stdev=123.05, samples=119
write: IOPS=2285, BW=9143KiB/s (9363kB/s)(536MiB/60001msec); 0 zone resets
slat (usec): min=2, max=22168, avg=171.53, stdev=404.32
clat (usec): min=265, max=67564, avg=3370.78, stdev=1499.08
lat (usec): min=461, max=67584, avg=3542.74, stdev=1528.38
clat percentiles (usec):
| 1.00th=[ 1434], 5.00th=[ 2073], 10.00th=[ 2245], 20.00th=[ 2409],
| 30.00th=[ 2573], 40.00th=[ 2835], 50.00th=[ 3097], 60.00th=[ 3359],
| 70.00th=[ 3720], 80.00th=[ 4146], 90.00th=[ 4817], 95.00th=[ 5473],
| 99.00th=[ 7373], 99.50th=[ 8979], 99.90th=[18482], 99.95th=[21365],
| 99.99th=[53216]
bw ( KiB/s): min= 7672, max=10600, per=12.52%, avg=9151.23, stdev=491.75, samples=119
iops : min= 1918, max= 2650, avg=2287.78, stdev=122.94, samples=119
lat (usec) : 10=0.01%, 250=0.01%, 500=0.01%, 750=0.05%, 1000=0.23%
lat (msec) : 2=3.60%, 4=73.98%, 10=21.73%, 20=0.33%, 50=0.06%
lat (msec) : 100=0.01%
cpu : usr=1.58%, sys=11.52%, ctx=160075, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=136594,137150,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=20: Sat Aug 6 04:08:58 2022
read: IOPS=2282, BW=9130KiB/s (9349kB/s)(535MiB/60002msec)
slat (nsec): min=1857, max=45528k, avg=167508.82, stdev=427688.07
clat (usec): min=267, max=48094, avg=3299.74, stdev=1449.94
lat (usec): min=485, max=48113, avg=3467.89, stdev=1492.22
clat percentiles (usec):
| 1.00th=[ 1483], 5.00th=[ 2073], 10.00th=[ 2245], 20.00th=[ 2376],
| 30.00th=[ 2540], 40.00th=[ 2737], 50.00th=[ 2999], 60.00th=[ 3261],
| 70.00th=[ 3621], 80.00th=[ 4047], 90.00th=[ 4752], 95.00th=[ 5407],
| 99.00th=[ 7242], 99.50th=[ 8979], 99.90th=[18220], 99.95th=[24511],
| 99.99th=[39060]
bw ( KiB/s): min= 7616, max=10464, per=12.51%, avg=9137.07, stdev=494.14, samples=119
iops : min= 1904, max= 2616, avg=2284.24, stdev=123.54, samples=119
write: IOPS=2274, BW=9097KiB/s (9315kB/s)(533MiB/60002msec); 0 zone resets
slat (usec): min=2, max=37988, avg=170.16, stdev=409.82
clat (usec): min=97, max=49234, avg=3379.45, stdev=1512.38
lat (usec): min=445, max=49258, avg=3550.06, stdev=1542.93
clat percentiles (usec):
| 1.00th=[ 1467], 5.00th=[ 2073], 10.00th=[ 2245], 20.00th=[ 2409],
| 30.00th=[ 2573], 40.00th=[ 2835], 50.00th=[ 3097], 60.00th=[ 3359],
| 70.00th=[ 3720], 80.00th=[ 4178], 90.00th=[ 4883], 95.00th=[ 5538],
| 99.00th=[ 7373], 99.50th=[ 9241], 99.90th=[19006], 99.95th=[24773],
| 99.99th=[47973]
bw ( KiB/s): min= 7368, max= 9968, per=12.45%, avg=9103.46, stdev=499.85, samples=119
iops : min= 1842, max= 2492, avg=2275.85, stdev=124.96, samples=119
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.04%, 1000=0.17%
lat (msec) : 2=3.46%, 4=74.12%, 10=21.81%, 20=0.31%, 50=0.08%
cpu : usr=1.63%, sys=11.44%, ctx=160226, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=136957,136462,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=21: Sat Aug 6 04:08:58 2022
read: IOPS=2276, BW=9107KiB/s (9326kB/s)(534MiB/60003msec)
slat (nsec): min=1879, max=24205k, avg=167865.92, stdev=403069.88
clat (usec): min=221, max=50175, avg=3296.92, stdev=1460.02
lat (usec): min=561, max=50209, avg=3465.21, stdev=1493.34
clat percentiles (usec):
| 1.00th=[ 1434], 5.00th=[ 2073], 10.00th=[ 2212], 20.00th=[ 2376],
| 30.00th=[ 2540], 40.00th=[ 2737], 50.00th=[ 2999], 60.00th=[ 3294],
| 70.00th=[ 3621], 80.00th=[ 4047], 90.00th=[ 4752], 95.00th=[ 5407],
| 99.00th=[ 7177], 99.50th=[ 8586], 99.90th=[16450], 99.95th=[24249],
| 99.99th=[48497]
bw ( KiB/s): min= 7680, max=10328, per=12.47%, avg=9110.30, stdev=541.30, samples=119
iops : min= 1920, max= 2582, avg=2277.56, stdev=135.32, samples=119
write: IOPS=2288, BW=9156KiB/s (9375kB/s)(536MiB/60003msec); 0 zone resets
slat (nsec): min=1935, max=47400k, avg=171301.55, stdev=436114.61
clat (usec): min=282, max=50303, avg=3367.01, stdev=1488.00
lat (usec): min=539, max=50333, avg=3538.72, stdev=1530.30
clat percentiles (usec):
| 1.00th=[ 1434], 5.00th=[ 2057], 10.00th=[ 2212], 20.00th=[ 2409],
| 30.00th=[ 2606], 40.00th=[ 2835], 50.00th=[ 3097], 60.00th=[ 3359],
| 70.00th=[ 3720], 80.00th=[ 4146], 90.00th=[ 4817], 95.00th=[ 5473],
| 99.00th=[ 7242], 99.50th=[ 8717], 99.90th=[18744], 99.95th=[24773],
| 99.99th=[48497]
bw ( KiB/s): min= 7640, max=10088, per=12.53%, avg=9161.92, stdev=524.42, samples=119
iops : min= 1910, max= 2522, avg=2290.47, stdev=131.11, samples=119
lat (usec) : 250=0.01%, 500=0.01%, 750=0.05%, 1000=0.21%
lat (msec) : 2=3.68%, 4=73.93%, 10=21.74%, 20=0.30%, 50=0.07%
lat (msec) : 100=0.01%
cpu : usr=1.57%, sys=11.46%, ctx=159675, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=136617,137341,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=22: Sat Aug 6 04:08:58 2022
read: IOPS=2290, BW=9161KiB/s (9381kB/s)(537MiB/60003msec)
slat (nsec): min=1788, max=36776k, avg=167401.08, stdev=421910.62
clat (usec): min=210, max=41574, avg=3282.65, stdev=1370.68
lat (usec): min=544, max=41592, avg=3450.50, stdev=1413.16
clat percentiles (usec):
| 1.00th=[ 1336], 5.00th=[ 2057], 10.00th=[ 2212], 20.00th=[ 2376],
| 30.00th=[ 2540], 40.00th=[ 2737], 50.00th=[ 2999], 60.00th=[ 3261],
| 70.00th=[ 3621], 80.00th=[ 4047], 90.00th=[ 4752], 95.00th=[ 5407],
| 99.00th=[ 7046], 99.50th=[ 8586], 99.90th=[17171], 99.95th=[22676],
| 99.99th=[36963]
bw ( KiB/s): min= 7896, max=10760, per=12.55%, avg=9167.88, stdev=507.40, samples=119
iops : min= 1974, max= 2690, avg=2291.96, stdev=126.84, samples=119
write: IOPS=2297, BW=9190KiB/s (9410kB/s)(538MiB/60003msec); 0 zone resets
slat (usec): min=2, max=25181, avg=169.89, stdev=410.57
clat (usec): min=100, max=41544, avg=3349.60, stdev=1374.47
lat (usec): min=485, max=41564, avg=3519.94, stdev=1410.82
clat percentiles (usec):
| 1.00th=[ 1319], 5.00th=[ 2057], 10.00th=[ 2212], 20.00th=[ 2409],
| 30.00th=[ 2573], 40.00th=[ 2835], 50.00th=[ 3064], 60.00th=[ 3359],
| 70.00th=[ 3687], 80.00th=[ 4113], 90.00th=[ 4817], 95.00th=[ 5473],
| 99.00th=[ 7242], 99.50th=[ 8717], 99.90th=[17171], 99.95th=[21890],
| 99.99th=[29230]
bw ( KiB/s): min= 7920, max=10992, per=12.57%, avg=9192.82, stdev=525.13, samples=119
iops : min= 1980, max= 2748, avg=2298.18, stdev=131.29, samples=119
lat (usec) : 250=0.01%, 500=0.01%, 750=0.07%, 1000=0.30%
lat (msec) : 2=3.74%, 4=74.00%, 10=21.55%, 20=0.27%, 50=0.07%
cpu : usr=1.64%, sys=11.55%, ctx=161182, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=137417,137854,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=23: Sat Aug 6 04:08:58 2022
read: IOPS=2284, BW=9136KiB/s (9355kB/s)(535MiB/60002msec)
slat (nsec): min=1832, max=46816k, avg=168066.93, stdev=445100.57
clat (usec): min=284, max=50283, avg=3300.92, stdev=1460.93
lat (usec): min=587, max=50302, avg=3469.39, stdev=1506.18
clat percentiles (usec):
| 1.00th=[ 1516], 5.00th=[ 2073], 10.00th=[ 2245], 20.00th=[ 2376],
| 30.00th=[ 2540], 40.00th=[ 2737], 50.00th=[ 2999], 60.00th=[ 3261],
| 70.00th=[ 3621], 80.00th=[ 4047], 90.00th=[ 4752], 95.00th=[ 5407],
| 99.00th=[ 7177], 99.50th=[ 8848], 99.90th=[16712], 99.95th=[26084],
| 99.99th=[41681]
bw ( KiB/s): min= 7840, max=10120, per=12.51%, avg=9137.66, stdev=505.74, samples=119
iops : min= 1960, max= 2530, avg=2284.39, stdev=126.44, samples=119
write: IOPS=2277, BW=9110KiB/s (9329kB/s)(534MiB/60002msec); 0 zone resets
slat (nsec): min=1987, max=40527k, avg=170496.65, stdev=430814.23
clat (usec): min=274, max=50248, avg=3370.28, stdev=1488.87
lat (usec): min=523, max=50278, avg=3541.19, stdev=1525.74
clat percentiles (usec):
| 1.00th=[ 1500], 5.00th=[ 2073], 10.00th=[ 2245], 20.00th=[ 2409],
| 30.00th=[ 2573], 40.00th=[ 2835], 50.00th=[ 3064], 60.00th=[ 3359],
| 70.00th=[ 3720], 80.00th=[ 4146], 90.00th=[ 4817], 95.00th=[ 5473],
| 99.00th=[ 7242], 99.50th=[ 9110], 99.90th=[17957], 99.95th=[25560],
| 99.99th=[41681]
bw ( KiB/s): min= 7272, max=10168, per=12.46%, avg=9112.08, stdev=541.92, samples=119
iops : min= 1818, max= 2542, avg=2278.01, stdev=135.48, samples=119
lat (usec) : 500=0.01%, 750=0.03%, 1000=0.11%
lat (msec) : 2=3.55%, 4=74.06%, 10=21.86%, 20=0.32%, 50=0.07%
lat (msec) : 100=0.01%
cpu : usr=1.41%, sys=11.64%, ctx=160372, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=137048,136655,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=24: Sat Aug 6 04:08:58 2022
read: IOPS=2292, BW=9171KiB/s (9391kB/s)(537MiB/60003msec)
slat (nsec): min=1853, max=20714k, avg=168414.26, stdev=411707.51
clat (usec): min=206, max=114837, avg=3284.13, stdev=1542.79
lat (usec): min=482, max=115469, avg=3452.98, stdev=1580.45
clat percentiles (usec):
| 1.00th=[ 1303], 5.00th=[ 2040], 10.00th=[ 2212], 20.00th=[ 2376],
| 30.00th=[ 2540], 40.00th=[ 2737], 50.00th=[ 2999], 60.00th=[ 3261],
| 70.00th=[ 3589], 80.00th=[ 4047], 90.00th=[ 4752], 95.00th=[ 5407],
| 99.00th=[ 7111], 99.50th=[ 8848], 99.90th=[15926], 99.95th=[20055],
| 99.99th=[32113]
bw ( KiB/s): min= 6640, max=11032, per=12.56%, avg=9176.36, stdev=521.01, samples=119
iops : min= 1660, max= 2758, avg=2294.08, stdev=130.24, samples=119
write: IOPS=2291, BW=9168KiB/s (9388kB/s)(537MiB/60003msec); 0 zone resets
slat (nsec): min=1932, max=28883k, avg=168101.76, stdev=401110.32
clat (usec): min=354, max=114570, avg=3353.45, stdev=1604.67
lat (usec): min=427, max=114603, avg=3521.97, stdev=1634.92
clat percentiles (usec):
| 1.00th=[ 1287], 5.00th=[ 2040], 10.00th=[ 2212], 20.00th=[ 2376],
| 30.00th=[ 2573], 40.00th=[ 2835], 50.00th=[ 3064], 60.00th=[ 3359],
| 70.00th=[ 3720], 80.00th=[ 4146], 90.00th=[ 4817], 95.00th=[ 5473],
| 99.00th=[ 7242], 99.50th=[ 8717], 99.90th=[15401], 99.95th=[20579],
| 99.99th=[30278]
bw ( KiB/s): min= 6520, max=10984, per=12.55%, avg=9172.50, stdev=566.08, samples=119
iops : min= 1630, max= 2746, avg=2293.12, stdev=141.52, samples=119
lat (usec) : 250=0.01%, 500=0.01%, 750=0.11%, 1000=0.35%
lat (msec) : 2=3.87%, 4=73.54%, 10=21.77%, 20=0.29%, 50=0.05%
lat (msec) : 250=0.01%
cpu : usr=1.51%, sys=11.70%, ctx=160990, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=137569,137524,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
benchtest: (groupid=0, jobs=1): err= 0: pid=25: Sat Aug 6 04:08:58 2022
read: IOPS=2273, BW=9096KiB/s (9314kB/s)(533MiB/60002msec)
slat (nsec): min=1828, max=22156k, avg=169789.45, stdev=420296.64
clat (usec): min=227, max=47918, avg=3310.95, stdev=1447.85
lat (usec): min=528, max=55906, avg=3481.17, stdev=1487.55
clat percentiles (usec):
| 1.00th=[ 1500], 5.00th=[ 2073], 10.00th=[ 2245], 20.00th=[ 2376],
| 30.00th=[ 2540], 40.00th=[ 2737], 50.00th=[ 2999], 60.00th=[ 3294],
| 70.00th=[ 3621], 80.00th=[ 4047], 90.00th=[ 4752], 95.00th=[ 5473],
| 99.00th=[ 7242], 99.50th=[ 8979], 99.90th=[17433], 99.95th=[21103],
| 99.99th=[46400]
bw ( KiB/s): min= 7462, max=10264, per=12.46%, avg=9101.20, stdev=529.04, samples=119
iops : min= 1865, max= 2566, avg=2275.28, stdev=132.26, samples=119
write: IOPS=2275, BW=9101KiB/s (9320kB/s)(533MiB/60002msec); 0 zone resets
slat (usec): min=2, max=37555, avg=170.29, stdev=419.29
clat (usec): min=219, max=49275, avg=3376.88, stdev=1464.08
lat (usec): min=619, max=49296, avg=3547.61, stdev=1499.80
clat percentiles (usec):
| 1.00th=[ 1483], 5.00th=[ 2073], 10.00th=[ 2245], 20.00th=[ 2409],
| 30.00th=[ 2573], 40.00th=[ 2835], 50.00th=[ 3097], 60.00th=[ 3359],
| 70.00th=[ 3720], 80.00th=[ 4146], 90.00th=[ 4883], 95.00th=[ 5538],
| 99.00th=[ 7439], 99.50th=[ 9110], 99.90th=[17957], 99.95th=[21627],
| 99.99th=[45876]
bw ( KiB/s): min= 7632, max=10096, per=12.46%, avg=9108.00, stdev=502.18, samples=119
iops : min= 1908, max= 2524, avg=2276.98, stdev=125.55, samples=119
lat (usec) : 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.14%
lat (msec) : 2=3.40%, 4=74.22%, 10=21.82%, 20=0.32%, 50=0.07%
cpu : usr=1.51%, sys=11.58%, ctx=160605, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=136441,136523,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=71.3MiB/s (74.8MB/s), 9096KiB/s-9171KiB/s (9314kB/s-9391kB/s), io=4280MiB (4488MB), run=60001-60003msec
WRITE: bw=71.4MiB/s (74.9MB/s), 9097KiB/s-9190KiB/s (9315kB/s-9410kB/s), io=4284MiB (4492MB), run=60001-60003msec
Disk stats (read/write):
nvme0n1: ios=1093961/1095028, merge=0/18, ticks=2536186/2466225, in_queue=5002411, util=99.89%
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment