Skip to content

Instantly share code, notes, and snippets.

@JustHumanz

JustHumanz/gist:5e2243257e725faa5a17f58c5b5fd0a8 Secret

Created July 25, 2024 11:47
Show Gist options
  • Save JustHumanz/5e2243257e725faa5a17f58c5b5fd0a8 to your computer and use it in GitHub Desktop.
Save JustHumanz/5e2243257e725faa5a17f58c5b5fd0a8 to your computer and use it in GitHub Desktop.
Download ZIP
Fio test
Raw
gistfile1.txt
### SSD(cache)
sudo fio --name=write_iops --filename=/dev/sdq --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=256 --rw=randwrite --group_reporting=1 --iodepth_batch_submit=256 --iodepth_batch_complete_max=256 --numjob=16
```bash
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.28
Starting 16 processes
Jobs: 16 (f=16): [w(16)][3.5%][w=91.3MiB/s][w=23.4k IOPS][eta 28m:48s]
write_iops: (groupid=0, jobs=16): err= 0: pid=102122: Sat Jul 20 19:10:16 2024
write: IOPS=24.0k, BW=94.0MiB/s (98.6MB/s)(5650MiB/60116msec); 0 zone resets
slat (usec): min=2, max=266084, avg=110037.91, stdev=42542.18
clat (nsec): min=1200, max=325062k, avg=35869515.08, stdev=50321983.46
lat (usec): min=1584, max=503913, avg=145893.20, stdev=49271.77
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6],
| 30.00th=[ 7], 40.00th=[ 12], 50.00th=[ 5800], 60.00th=[ 25035],
| 70.00th=[ 45876], 80.00th=[ 74974], 90.00th=[120062], 95.00th=[152044],
| 99.00th=[181404], 99.50th=[189793], 99.90th=[214959], 99.95th=[227541],
| 99.99th=[254804]
bw ( KiB/s): min=60479, max=135014, per=100.00%, avg=96279.34, stdev=950.12, samples=1912
iops : min=15119, max=33752, avg=24069.57, stdev=237.53, samples=1912
lat (usec) : 2=0.13%, 4=8.80%, 10=29.98%, 20=5.53%, 50=2.61%
lat (usec) : 100=0.02%, 250=0.07%, 500=0.15%, 750=0.15%, 1000=0.18%
lat (msec) : 2=0.49%, 4=1.06%, 10=3.46%, 20=5.20%, 50=13.89%
lat (msec) : 100=14.97%, 250=13.49%, 500=0.01%
cpu : usr=0.09%, sys=1.17%, ctx=675977, majf=0, minf=932
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.5%, 16=5.6%, 32=13.6%, >=64=80.2%
submit : 0=0.0%, 4=1.4%, 8=1.6%, 16=3.5%, 32=6.5%, 64=13.0%, >=64=73.9%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=0.1%, 64=0.1%, >=64=99.9%
issued rwts: total=0,1442818,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=94.0MiB/s (98.6MB/s), 94.0MiB/s-94.0MiB/s (98.6MB/s-98.6MB/s), io=5650MiB (5925MB), run=60116-60116msec
Disk stats (read/write):
sdq: ios=83/1495049, merge=0/263, ticks=15/15681573, in_queue=15681588, util=99.96%
```
sudo fio --name=read_iops --filename=/dev/sdq --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=256 --rw=randread --group_reporting=1 --iodepth_batch_submit=256 --iodepth_batch_complete_max=256
```bash
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=274MiB/s][r=70.2k IOPS][eta 00m:00s]
read_iops: (groupid=0, jobs=1): err= 0: pid=10988: Mon Jul 15 11:24:01 2024
read: IOPS=63.3k, BW=247MiB/s (259MB/s)(14.5GiB/60004msec)
slat (nsec): min=970, max=337554, avg=7550.64, stdev=11885.59
clat (usec): min=108, max=65947, avg=4033.51, stdev=2526.30
lat (usec): min=111, max=65950, avg=4041.06, stdev=2527.17
clat percentiles (usec):
| 1.00th=[ 502], 5.00th=[ 824], 10.00th=[ 1172], 20.00th=[ 1844],
| 30.00th=[ 2507], 40.00th=[ 3195], 50.00th=[ 3851], 60.00th=[ 4555],
| 70.00th=[ 5211], 80.00th=[ 5866], 90.00th=[ 6718], 95.00th=[ 7701],
| 99.00th=[10290], 99.50th=[13829], 99.90th=[26346], 99.95th=[31327],
| 99.99th=[41681]
bw ( KiB/s): min=170557, max=298688, per=100.00%, avg=253410.87, stdev=35848.99, samples=120
iops : min=42639, max=74672, avg=63352.63, stdev=8962.25, samples=120
lat (usec) : 250=0.01%, 500=0.96%, 750=3.01%, 1000=3.55%
lat (msec) : 2=14.79%, 4=29.88%, 10=46.67%, 20=0.90%, 50=0.23%
lat (msec) : 100=0.01%
cpu : usr=6.36%, sys=27.20%, ctx=3656009, majf=0, minf=58
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=99.0%, 8=0.9%, 16=0.1%, 32=0.1%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.0%, 8=0.9%, 16=0.1%, 32=0.1%, 64=0.0%, >=64=0.1%
issued rwts: total=3799544,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=247MiB/s (259MB/s), 247MiB/s-247MiB/s (259MB/s-259MB/s), io=14.5GiB (15.6GB), run=60004-60004msec
Disk stats (read/write):
sdq: ios=3916219/0, merge=12975/0, ticks=15778872/0, in_queue=15778872, util=99.89%
```
## HDD(back)
sudo fio --name=write_iops --filename=/dev/sda --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=256 --rw=randwrite --group_reporting=1 --iodepth_batch_submit=256 --iodepth_batch_complete_max=256 --numjob=16
```bash
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.28
Starting 16 processes
Jobs: 11 (f=11): [f(1),_(1),w(2),_(2),w(6),_(1),w(2),_(1)][17.5%][w=5761KiB/s][w=1440 IOPS][eta 05m:06s]
write_iops: (groupid=0, jobs=16): err= 0: pid=47660: Sat Jul 20 18:31:03 2024
write: IOPS=1095, BW=4642KiB/s (4753kB/s)(284MiB/62592msec); 0 zone resets
slat (msec): min=11, max=5378, avg=2349.92, stdev=1198.37
clat (usec): min=4, max=5937.2k, avg=779968.89, stdev=1139843.13
lat (msec): min=88, max=10454, avg=3134.42, stdev=1438.62
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 15], 10.00th=[ 18],
| 20.00th=[ 23], 30.00th=[ 27], 40.00th=[ 31],
| 50.00th=[ 88605], 60.00th=[ 541066], 70.00th=[ 943719],
| 80.00th=[1518339], 90.00th=[2466251], 95.00th=[3472884],
| 99.00th=[4529849], 99.50th=[4664067], 99.90th=[5402264],
| 99.95th=[5469373], 99.99th=[5536482]
bw ( KiB/s): min=25715, max=31720, per=100.00%, avg=30037.55, stdev=94.96, samples=302
iops : min= 6424, max= 7930, avg=7508.24, stdev=23.81, samples=302
lat (usec) : 10=0.81%, 20=14.21%, 50=33.06%
lat (msec) : 20=0.62%, 50=0.97%, 100=0.63%, 250=2.88%, 500=4.80%
lat (msec) : 750=8.06%, 1000=5.10%, 2000=15.47%, >=2000=13.82%
cpu : usr=0.00%, sys=0.14%, ctx=31672, majf=0, minf=932
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.7%, 16=5.6%, 32=14.9%, >=64=81.0%
submit : 0=0.0%, 4=1.6%, 8=1.4%, 16=3.3%, 32=6.9%, 64=13.0%, >=64=73.8%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.6%, 32=1.3%, 64=1.3%, >=64=96.9%
issued rwts: total=0,68541,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=4642KiB/s (4753kB/s), 4642KiB/s-4642KiB/s (4753kB/s-4753kB/s), io=284MiB (298MB), run=62592-62592msec
Disk stats (read/write):
sda: ios=65/76539, merge=0/11, ticks=1021/16592010, in_queue=16593031, util=99.98%
```
sudo fio --name=read_iops --filename=/dev/sda --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=256 --rw=randread --group_reporting=1 --iodepth_batch_submit=256 --iodepth_batch_complete_max
=256 --numjob=16
```bash
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.28
Starting 16 processes
Jobs: 8 (f=8): [r(2),_(1),r(1),_(1),r(1),_(2),r(2),_(2),r(1),_(2),r(1)][12.4%][r=3277KiB/s][r=819 IOPS][eta 08m:08s]
read_iops: (groupid=0, jobs=16): err= 0: pid=47553: Sat Jul 20 18:29:04 2024
read: IOPS=708, BW=3094KiB/s (3168kB/s)(192MiB/63481msec)
slat (msec): min=173, max=6936, avg=3604.09, stdev=1359.02
clat (usec): min=5, max=6930.4k, avg=1280467.20, stdev=1734760.68
lat (msec): min=646, max=13746, avg=4717.02, stdev=1666.62
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 14], 10.00th=[ 17],
| 20.00th=[ 21], 30.00th=[ 23], 40.00th=[ 25],
| 50.00th=[ 32], 60.00th=[ 809501], 70.00th=[1988101],
| 80.00th=[2801796], 90.00th=[4143973], 95.00th=[4932502],
| 99.00th=[5872026], 99.50th=[6140462], 99.90th=[6543115],
| 99.95th=[6677332], 99.99th=[6945768]
bw ( KiB/s): min=25905, max=31549, per=100.00%, avg=30179.90, stdev=103.91, samples=194
iops : min= 6471, max= 7882, avg=7541.37, stdev=25.99, samples=194
lat (usec) : 10=0.94%, 20=19.01%, 50=33.45%
lat (msec) : 500=2.14%, 750=4.31%, 1000=1.80%, 2000=9.00%, >=2000=29.90%
cpu : usr=0.01%, sys=0.10%, ctx=20912, majf=0, minf=934
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=4.6%, 32=14.8%, >=64=80.2%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=3.0%, 32=4.1%, 64=10.6%, >=64=82.4%
complete : 0=0.0%, 4=0.0%, 8=0.5%, 16=0.5%, 32=0.5%, 64=0.0%, >=64=98.6%
issued rwts: total=45004,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=3094KiB/s (3168kB/s), 3094KiB/s-3094KiB/s (3168kB/s-3168kB/s), io=192MiB (201MB), run=63481-63481msec
Disk stats (read/write):
sda: ios=48930/0, merge=14/0, ticks=17091509/0, in_queue=17091510, util=99.92%
```
## SSD(raid1)
sudo fio --name=write_iops --filename=/dev/md1 --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=256 --rw=randwrite --group_reporting=1 --iodepth_batch_submit=256 --iodepth_batch_complete_max=256 --numjob=16
```bash
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.28
Starting 16 processes
Jobs: 16 (f=13): [w(3),f(1),w(1),f(1),w(3),f(1),w(6)][3.2%][w=82.9MiB/s][w=21.2k IOPS][eta 31m:26s]
write_iops: (groupid=0, jobs=16): err= 0: pid=47991: Sat Jul 20 18:37:16 2024
write: IOPS=22.3k, BW=87.1MiB/s (91.4MB/s)(5237MiB/60097msec); 0 zone resets
slat (usec): min=331, max=249329, avg=122621.09, stdev=44581.30
clat (nsec): min=1360, max=298047k, avg=30978496.79, stdev=48300462.17
lat (usec): min=689, max=442882, avg=153500.60, stdev=42028.59
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5],
| 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 6849],
| 70.00th=[ 35390], 80.00th=[ 72877], 90.00th=[115868], 95.00th=[141558],
| 99.00th=[166724], 99.50th=[177210], 99.90th=[198181], 99.95th=[206570],
| 99.99th=[221250]
bw ( KiB/s): min=64812, max=118452, per=99.92%, avg=89168.44, stdev=935.03, samples=1909
iops : min=16202, max=29613, avg=22292.10, stdev=233.76, samples=1909
lat (usec) : 2=0.25%, 4=9.65%, 10=42.96%, 20=3.67%, 50=0.58%
lat (usec) : 100=0.01%, 250=0.03%, 500=0.19%, 750=0.10%, 1000=0.13%
lat (msec) : 2=0.27%, 4=0.87%, 10=2.69%, 20=3.53%, 50=10.02%
lat (msec) : 100=11.92%, 250=13.15%, 500=0.01%
cpu : usr=0.06%, sys=2.73%, ctx=1174033, majf=0, minf=932
IO depths : 1=0.1%, 2=0.6%, 4=1.2%, 8=3.1%, 16=9.3%, 32=18.9%, >=64=67.1%
submit : 0=0.0%, 4=1.3%, 8=1.1%, 16=3.1%, 32=7.0%, 64=13.8%, >=64=73.7%
complete : 0=0.0%, 4=0.2%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=99.8%
issued rwts: total=0,1337366,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=87.1MiB/s (91.4MB/s), 87.1MiB/s-87.1MiB/s (91.4MB/s-91.4MB/s), io=5237MiB (5492MB), run=60097-60097msec
Disk stats (read/write):
md1: ios=92/1386618, merge=0/0, ticks=100/87216448, in_queue=87216548, util=99.91%, aggrios=76/1386615, aggrmerge=0/6, aggrticks=50/494586, aggrin_queue=494637, aggrutil=99.75%
sdq: ios=134/1386614, merge=0/7, ticks=67/420967, in_queue=421035, util=99.75%
sdr: ios=19/1386616, merge=0/5, ticks=33/568206, in_queue=568239, util=99.74%
```
sudo fio --name=read_iops --filename=/dev/md1 --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=256 --rw=randread --group_reporting=1 --iodepth_batch_submit=256 --iodepth_batch_complete_max=256 --numjob=16
```bash
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.28
Starting 16 processes
Jobs: 16 (f=7): [f(1),r(1),f(3),r(1),f(4),r(3),f(1),r(2)][20.1%][r=542MiB/s][r=139k IOPS][eta 04m:11s]
read_iops: (groupid=0, jobs=16): err= 0: pid=47867: Sat Jul 20 18:35:55 2024
read: IOPS=137k, BW=535MiB/s (561MB/s)(31.4GiB/60027msec)
slat (usec): min=2, max=40043, avg=18430.98, stdev=7054.63
clat (nsec): min=1110, max=71294k, avg=7663779.02, stdev=9770280.29
lat (usec): min=423, max=96157, avg=26094.13, stdev=9315.47
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5],
| 30.00th=[ 6], 40.00th=[ 18], 50.00th=[ 2343], 60.00th=[ 6194],
| 70.00th=[10683], 80.00th=[17433], 90.00th=[24511], 95.00th=[27395],
| 99.00th=[31327], 99.50th=[32900], 99.90th=[35914], 99.95th=[38011],
| 99.99th=[48497]
bw ( KiB/s): min=505086, max=595570, per=100.00%, avg=548668.75, stdev=1218.76, samples=1906
iops : min=126270, max=148892, avg=137166.59, stdev=304.68, samples=1906
lat (usec) : 2=0.27%, 4=14.05%, 10=22.39%, 20=5.13%, 50=2.16%
lat (usec) : 100=0.14%, 250=0.22%, 500=0.44%, 750=0.68%, 1000=0.72%
lat (msec) : 2=2.84%, 4=5.48%, 10=14.04%, 20=14.17%, 50=17.28%
lat (msec) : 100=0.01%
cpu : usr=0.35%, sys=3.32%, ctx=2261260, majf=0, minf=933
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.1%, 16=1.1%, 32=12.0%, >=64=86.8%
submit : 0=0.0%, 4=1.7%, 8=1.7%, 16=3.4%, 32=6.8%, 64=13.4%, >=64=73.2%
complete : 0=0.0%, 4=0.2%, 8=0.1%, 16=0.1%, 32=0.1%, 64=0.1%, >=64=99.8%
issued rwts: total=8221968,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=535MiB/s (561MB/s), 535MiB/s-535MiB/s (561MB/s-561MB/s), io=31.4GiB (33.7GB), run=60027-60027msec
Disk stats (read/write):
md1: ios=8497779/0, merge=0/0, ticks=32445924/0, in_queue=32445924, util=99.94%, aggrios=4248123/0, aggrmerge=781/0, aggrticks=15738431/0, aggrin_queue=15738431, aggrutil=99.78%
sdq: ios=4358569/0, merge=822/0, ticks=15738282/0, in_queue=15738282, util=99.78%
sdr: ios=4137677/0, merge=741/0, ticks=15738580/0, in_queue=15738580, util=99.78%
```
## Bcahce
sudo fio --name=write_iops --filename=/dev/bcache0 --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=256 --rw=randwrite --group_reporting=1 --iodepth_batch_submit=256 --iodepth_batch_complete_max=256 --numjob=16
```bash
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.28
Starting 16 processes
Jobs: 16 (f=16): [w(16)][100.0%][w=30.3MiB/s][w=7745 IOPS][eta 00m:00s]
write_iops: (groupid=0, jobs=16): err= 0: pid=48965: Sat Jul 20 18:46:37 2024
write: IOPS=9235, BW=36.3MiB/s (38.1MB/s)(2205MiB/60689msec); 0 zone resets
slat (usec): min=6, max=1945.9k, avg=217362.90, stdev=239995.87
clat (usec): min=2, max=4905.6k, avg=187463.26, stdev=239474.35
lat (usec): min=280, max=4953.6k, avg=404755.70, stdev=349197.53
clat percentiles (usec):
| 1.00th=[ 9], 5.00th=[ 13], 10.00th=[ 18],
| 20.00th=[ 2573], 30.00th=[ 12649], 40.00th=[ 24249],
| 50.00th=[ 54789], 60.00th=[ 183501], 70.00th=[ 320865],
| 80.00th=[ 358613], 90.00th=[ 404751], 95.00th=[ 708838],
| 99.00th=[1082131], 99.50th=[1149240], 99.90th=[1551893],
| 99.95th=[1853883], 99.99th=[2298479]
bw ( KiB/s): min= 4881, max=137426, per=100.00%, avg=44309.56, stdev=1582.07, samples=1624
iops : min= 1220, max=34354, avg=11075.87, stdev=395.52, samples=1624
lat (usec) : 4=0.01%, 10=1.94%, 20=8.78%, 50=4.34%, 100=0.34%
lat (usec) : 250=0.43%, 500=0.21%, 750=0.13%, 1000=0.12%
lat (msec) : 2=2.37%, 4=2.82%, 10=5.78%, 20=10.03%, 50=11.89%
lat (msec) : 100=5.42%, 250=9.21%, 500=28.95%, 750=4.76%, 1000=1.57%
lat (msec) : 2000=1.28%, >=2000=0.03%
cpu : usr=0.06%, sys=0.71%, ctx=80112, majf=0, minf=932
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.3%, 16=2.1%, 32=4.1%, >=64=93.5%
submit : 0=0.0%, 4=9.9%, 8=7.4%, 16=11.5%, 32=10.5%, 64=10.0%, >=64=50.6%
complete : 0=0.0%, 4=5.3%, 8=1.9%, 16=3.5%, 32=5.6%, 64=8.7%, >=64=75.0%
issued rwts: total=0,560497,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=36.3MiB/s (38.1MB/s), 36.3MiB/s-36.3MiB/s (38.1MB/s-38.1MB/s), io=2205MiB (2312MB), run=60689-60689msec
Disk stats (read/write):
bcache0: ios=265/583991, merge=0/0, ticks=416/32214740, in_queue=32215156, util=100.00%, aggrios=337/294117, aggrmerge=0/10, aggrticks=244/13467433, aggrin_queue=13467677, aggrutil=99.77%
sda: ios=45/73549, merge=0/20, ticks=208/15770591, in_queue=15770798, util=99.77%
md1: ios=629/514686, merge=0/0, ticks=280/11164276, in_queue=11164556, util=29.26%, aggrios=314/17482, aggrmerge=0/498124, aggrticks=191/62472, aggrin_queue=63992, aggrutil=28.71%
sdq: ios=382/17870, merge=0/497737, ticks=200/62323, in_queue=63655, util=28.71%
sdr: ios=247/17095, merge=0/498512, ticks=182/62622, in_queue=64329, util=28.45%
```
sudo fio --name=read_iops --filename=/dev/bcache0 --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=256 --rw=randread --group_reporting=1 --iodepth_batch_submit=256 --iodepth_batch_complete_max=256 --numjob=16
```bash
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.28
Starting 16 processes
Jobs: 13 (f=13): [r(7),_(1),r(2),_(1),r(4),_(1)][26.3%][r=7229KiB/s][r=1807 IOPS][eta 02m:59s]
read_iops: (groupid=0, jobs=16): err= 0: pid=62377: Sat Jul 20 18:59:17 2024
read: IOPS=7403, BW=29.2MiB/s (30.6MB/s)(1810MiB/62046msec)
slat (usec): min=2, max=4783.3k, avg=337063.59, stdev=928591.30
clat (nsec): min=1830, max=5577.9M, avg=137445900.20, stdev=515959136.23
lat (usec): min=198, max=9533.4k, avg=474435.83, stdev=1215385.23
clat percentiles (usec):
| 1.00th=[ 7], 5.00th=[ 12], 10.00th=[ 18],
| 20.00th=[ 635], 30.00th=[ 1778], 40.00th=[ 3359],
| 50.00th=[ 8717], 60.00th=[ 19792], 70.00th=[ 38011],
| 80.00th=[ 78119], 90.00th=[ 183501], 95.00th=[ 501220],
| 99.00th=[3204449], 99.50th=[4110418], 99.90th=[4664067],
| 99.95th=[4999611], 99.99th=[5335155]
bw ( KiB/s): min=23017, max=460776, per=100.00%, avg=119281.92, stdev=7359.99, samples=496
iops : min= 5753, max=115194, avg=29819.85, stdev=1840.00, samples=496
lat (usec) : 2=0.01%, 4=0.16%, 10=3.39%, 20=7.58%, 50=5.63%
lat (usec) : 100=0.73%, 250=1.16%, 500=0.94%, 750=1.08%, 1000=1.64%
lat (msec) : 2=9.90%, 4=9.94%, 10=9.84%, 20=8.53%, 50=13.42%
lat (msec) : 100=9.54%, 250=9.20%, 500=2.71%, 750=0.73%, 1000=0.63%
lat (msec) : 2000=1.80%, >=2000=1.90%
cpu : usr=0.04%, sys=0.48%, ctx=36411, majf=0, minf=932
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.3%, 16=3.3%, 32=4.7%, >=64=91.6%
submit : 0=0.0%, 4=42.9%, 8=10.9%, 16=11.4%, 32=10.8%, 64=8.2%, >=64=15.9%
complete : 0=0.0%, 4=45.9%, 8=10.2%, 16=10.2%, 32=9.5%, 64=7.1%, >=64=17.2%
issued rwts: total=459346,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=29.2MiB/s (30.6MB/s), 29.2MiB/s-29.2MiB/s (30.6MB/s-30.6MB/s), io=1810MiB (1898MB), run=62046-62046msec
Disk stats (read/write):
bcache0: ios=651942/0, merge=0/0, ticks=20095924/0, in_queue=20095924, util=99.91%, aggrios=326826/21743, aggrmerge=6/7, aggrticks=9582825/10053, aggrin_queue=9592878, aggrutil=99.44%
sda: ios=58415/51, merge=12/15, ticks=15843718/2490, in_queue=15846208, util=99.44%
md1: ios=595238/43435, merge=0/0, ticks=3321932/17616, in_queue=3339548, util=94.59%, aggrios=94524/43233, aggrmerge=203109/310, aggrticks=534013/6026, aggrin_queue=540108, aggrutil=94.44%
sdq: ios=94889/43237, merge=199541/307, ticks=540076/6416, in_queue=546576, util=94.44%
sdr: ios=94159/43230, merge=206677/314, ticks=527951/5636, in_queue=533641, util=94.35%
```
## rbd(replication)
fio --name=write_iops --filename=/dev/rbd0 --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=256 --rw=randwrite --group_reporting=1 --iodepth_batch_submit=256 --iodepth_batch_complete_max=256 --numjob=16
```bash
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.28
Starting 16 processes
Jobs: 16 (f=16): [w(16)][2.9%][w=76.3MiB/s][w=19.5k IOPS][eta 35m:30s]
write_iops: (groupid=0, jobs=16): err= 0: pid=1226654: Wed Jul 24 01:01:45 2024
write: IOPS=19.9k, BW=78.1MiB/s (81.9MB/s)(4705MiB/60257msec); 0 zone resets
slat (nsec): min=1220, max=584477k, avg=56525310.64, stdev=65262603.38
clat (usec): min=2, max=1084.6k, avg=142559.78, stdev=118428.47
lat (msec): min=7, max=1183, avg=199.08, stdev=126.17
clat percentiles (usec):
| 1.00th=[ 10], 5.00th=[ 18], 10.00th=[ 14746], 20.00th=[ 49021],
| 30.00th=[ 68682], 40.00th=[ 89654], 50.00th=[113771], 60.00th=[143655],
| 70.00th=[179307], 80.00th=[223347], 90.00th=[299893], 95.00th=[375391],
| 99.00th=[549454], 99.50th=[599786], 99.90th=[708838], 99.95th=[750781],
| 99.99th=[859833]
bw ( KiB/s): min=27753, max=138712, per=100.00%, avg=80094.16, stdev=1286.69, samples=1920
iops : min= 6938, max=34678, avg=20023.11, stdev=321.67, samples=1920
lat (usec) : 4=0.01%, 10=1.03%, 20=5.05%, 50=2.20%, 100=0.27%
lat (usec) : 250=0.30%, 500=0.09%, 750=0.02%, 1000=0.01%
lat (msec) : 2=0.07%, 4=0.12%, 10=0.42%, 20=1.05%, 50=9.96%
lat (msec) : 100=24.02%, 250=40.02%, 500=13.97%, 750=1.55%, 1000=0.05%
lat (msec) : 2000=0.01%
cpu : usr=0.24%, sys=0.66%, ctx=164839, majf=0, minf=932
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.1%, >=64=99.8%
submit : 0=0.0%, 4=88.8%, 8=0.7%, 16=1.0%, 32=2.1%, 64=3.3%, >=64=4.1%
complete : 0=0.0%, 4=90.8%, 8=0.4%, 16=0.5%, 32=1.3%, 64=2.5%, >=64=4.5%
issued rwts: total=0,1200660,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=78.1MiB/s (81.9MB/s), 78.1MiB/s-78.1MiB/s (81.9MB/s-81.9MB/s), io=4705MiB (4934MB), run=60257-60257msec
Disk stats (read/write):
rbd0: ios=0/1242023, merge=0/1, ticks=0/160155144, in_queue=160155143, util=100.00%
```
sudo fio --name=read_iops --filename=/dev/rbd0 --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=256 --rw=randread --group_reporting=1 --iodepth_batch_submit=256 --iodepth_batch_complete_max=256 --numjob=16
```bash
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.28
Starting 16 processes
Jobs: 16 (f=16): [r(16)][3.4%][r=6468KiB/s][r=1617 IOPS][eta 30m:41s]
read_iops: (groupid=0, jobs=16): err= 0: pid=1225553: Wed Jul 24 00:58:53 2024
read: IOPS=25.9k, BW=101MiB/s (106MB/s)(6270MiB/61872msec)
slat (nsec): min=1120, max=1166.8M, avg=66799936.21, stdev=134171369.08
clat (nsec): min=1320, max=4214.8M, avg=78897569.89, stdev=235794414.30
lat (usec): min=80, max=4612.6k, avg=145637.99, stdev=287144.26
clat percentiles (usec):
| 1.00th=[ 4], 5.00th=[ 8], 10.00th=[ 11],
| 20.00th=[ 18], 30.00th=[ 273], 40.00th=[ 2147],
| 50.00th=[ 4817], 60.00th=[ 10683], 70.00th=[ 24249],
| 80.00th=[ 60556], 90.00th=[ 196084], 95.00th=[ 446694],
| 99.00th=[1266680], 99.50th=[1686111], 99.90th=[2365588],
| 99.95th=[2600469], 99.99th=[3103785]
bw ( KiB/s): min=14513, max=569739, per=100.00%, avg=116431.86, stdev=6965.30, samples=1762
iops : min= 3628, max=142434, avg=29107.58, stdev=1741.33, samples=1762
lat (usec) : 2=0.02%, 4=1.07%, 10=8.74%, 20=13.01%, 50=5.75%
lat (usec) : 100=0.55%, 250=0.80%, 500=2.55%, 750=2.24%, 1000=1.15%
lat (msec) : 2=3.56%, 4=8.17%, 10=11.54%, 20=8.79%, 50=10.21%
lat (msec) : 100=6.91%, 250=6.55%, 500=4.22%, 750=1.92%, 1000=0.80%
lat (msec) : 2000=1.30%, >=2000=0.26%
cpu : usr=0.28%, sys=0.84%, ctx=358384, majf=0, minf=932
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.8%, >=64=99.1%
submit : 0=0.0%, 4=94.5%, 8=0.5%, 16=0.5%, 32=0.8%, 64=1.2%, >=64=2.6%
complete : 0=0.0%, 4=95.8%, 8=0.4%, 16=0.3%, 32=0.4%, 64=0.6%, >=64=2.5%
issued rwts: total=1601417,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=101MiB/s (106MB/s), 101MiB/s-101MiB/s (106MB/s-106MB/s), io=6270MiB (6575MB), run=61872-61872msec
Disk stats (read/write):
rbd0: ios=1710210/0, merge=1/0, ticks=97661553/0, in_queue=97661553, util=100.00%
```
## rbd(ec)
fio --name=write_iops --filename=/dev/rbd0 --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=256 --rw=randwrite --group_reporting=1 --iodepth_batch_submit=256 --iodepth_batch_complete_max=256 --numjob=16ck-adm# fio --name=write_iops --filename=/dev/rbd0 --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=256 --rw=randwrite --group_reporting=1 --iodepth_batch_submit=256 --iodepth_bat
```bash
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.28
Starting 16 processes
Jobs: 10 (f=6): [f(2),w(1),_(1),w(1),_(1),w(1),_(3),w(1),f(1),w(1),f(1),_(1),w(1)][15.4%][w=7704KiB/s][w=1926 IOPS][eta 07m:14s]
write_iops: (groupid=0, jobs=16): err= 0: pid=1229490: Wed Jul 24 01:10:00 2024
write: IOPS=4294, BW=17.0MiB/s (17.8MB/s)(1294MiB/76216msec); 0 zone resets
slat (nsec): min=1220, max=6128.2M, avg=208546710.01, stdev=353474672.59
clat (usec): min=2, max=40137k, avg=671433.37, stdev=2897026.96
lat (msec): min=12, max=40136, avg=879.57, stdev=2936.93
clat percentiles (usec):
| 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 12],
| 20.00th=[ 26], 30.00th=[ 60031], 40.00th=[ 93848],
| 50.00th=[ 126354], 60.00th=[ 166724], 70.00th=[ 214959],
| 80.00th=[ 304088], 90.00th=[ 566232], 95.00th=[ 1551893],
| 99.00th=[16441672], 99.50th=[17112761], 99.90th=[17112761],
| 99.95th=[17112761], 99.99th=[17112761]
bw ( KiB/s): min= 6358, max=68995, per=100.00%, avg=27372.40, stdev=877.05, samples=1537
iops : min= 1588, max=17248, avg=6842.38, stdev=219.26, samples=1537
lat (usec) : 4=0.01%, 10=7.19%, 20=12.36%, 50=1.32%, 100=0.36%
lat (usec) : 250=0.26%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.03%, 4=0.10%, 10=0.33%, 20=0.88%, 50=4.86%
lat (msec) : 100=14.51%, 250=33.19%, 500=14.02%, 750=3.64%, 1000=1.43%
lat (msec) : 2000=1.86%, >=2000=4.56%
cpu : usr=0.04%, sys=0.12%, ctx=54121, majf=0, minf=932
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=99.9%
submit : 0=0.0%, 4=85.9%, 8=0.5%, 16=1.4%, 32=2.9%, 64=4.8%, >=64=4.4%
complete : 0=0.0%, 4=88.0%, 8=0.3%, 16=0.8%, 32=1.8%, 64=3.8%, >=64=5.3%
issued rwts: total=0,327297,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=17.0MiB/s (17.8MB/s), 17.0MiB/s-17.0MiB/s (17.8MB/s-17.8MB/s), io=1294MiB (1357MB), run=76216-76216msec
Disk stats (read/write):
rbd0: ios=0/343622, merge=0/1, ticks=0/179487876, in_queue=179487876, util=100.00%
```
sudo fio --name=read_iops --filename=/dev/rbd0 --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=256 --rw=randread --group_reporting=1 --iodepth_batch_submit=256 --iodepth_batch_complete_max=256 --numjob=16
```bash
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.28
Starting 16 processes
Jobs: 15 (f=15): [r(5),_(1),r(10)][52.4%][r=763KiB/s][r=190 IOPS][eta 01m:00s]
read_iops: (groupid=0, jobs=16): err= 0: pid=1231810: Wed Jul 24 01:13:14 2024
read: IOPS=17.8k, BW=69.6MiB/s (73.0MB/s)(4401MiB/63246msec)
slat (nsec): min=1060, max=1172.3M, avg=58483976.58, stdev=119565188.26
clat (nsec): min=1340, max=12502M, avg=162955573.13, stdev=918696367.15
lat (usec): min=99, max=12505k, avg=221382.07, stdev=930433.93
clat percentiles (usec):
| 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 11],
| 20.00th=[ 19], 30.00th=[ 2540], 40.00th=[ 6849],
| 50.00th=[ 13698], 60.00th=[ 26084], 70.00th=[ 47973],
| 80.00th=[ 91751], 90.00th=[ 202376], 95.00th=[ 375391],
| 99.00th=[ 3271558], 99.50th=[ 9462350], 99.90th=[11072963],
| 99.95th=[11341399], 99.99th=[11744052]
bw ( KiB/s): min= 8511, max=604752, per=100.00%, avg=77897.27, stdev=5732.32, samples=1847
iops : min= 2127, max=151188, avg=19473.86, stdev=1433.08, samples=1847
lat (usec) : 2=0.02%, 4=0.61%, 10=7.86%, 20=12.58%, 50=2.49%
lat (usec) : 100=0.37%, 250=0.33%, 500=0.39%, 750=0.97%, 1000=1.33%
lat (msec) : 2=2.05%, 4=4.57%, 10=11.67%, 20=10.78%, 50=14.85%
lat (msec) : 100=10.73%, 250=10.55%, 500=4.53%, 750=1.17%, 1000=0.42%
lat (msec) : 2000=0.56%, >=2000=1.40%
cpu : usr=0.22%, sys=0.59%, ctx=283112, majf=0, minf=933
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=91.9%, 8=0.7%, 16=1.1%, 32=1.8%, 64=2.4%, >=64=2.1%
complete : 0=0.0%, 4=93.0%, 8=0.5%, 16=0.8%, 32=1.4%, 64=2.0%, >=64=2.4%
issued rwts: total=1122637,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=69.6MiB/s (73.0MB/s), 69.6MiB/s-69.6MiB/s (73.0MB/s-73.0MB/s), io=4401MiB (4615MB), run=63246-63246msec
Disk stats (read/write):
rbd0: ios=1181242/0, merge=1/0, ticks=163443379/0, in_queue=163443379, util=100.00%
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment