Benchmark fio 4k random write+fsync load in LXD VM (qemu) with different storage backends.

readme.md

What

Benchmark fio 4k random write+fsync load in LXD VM (qemu) with different storage backends.

Results

test	write IOPS	write speed
lvm	114	26.8
zfs-zstd	97	22.8
zfs-nc	93	22.0
dir-lvm	109	25.7
dir-md	58	13.7

Test labels

• lvm - type=lvm on LVM raid1 converted to thinpool
• zfs-zstd - type=zfs with compression=zstd
• zfs-nc - type=zfs with compression=off
• dir-lvm - type=dir on LVM full disk on md mirror
• dir-md - type=dir on md mirror

Methodology

• prepare free space on two SSD drives
• for X in labels:
• (LVM)

vgcreate --alloc normal vg0 /dev/sd[ab]4
lvcreate --type raid1 -m1 -n pool vg0
lvcreate --type raid1 -m1 -n meta vg0
lvconvert --type thin-pool --poolmetadata vg0/meta vg0/pool
lxc storage create X lvm source=vg0 lvm.thinpool_name=pool

https://unix.stackexchange.com/a/623348/12139

• (ZFS)

zpool create tank mirror /dev/sd[ab]4
zfs set atime=off xattr=sa tank
zfs create -o compression=VAR tank/lxc
lxc storage create X zfs source=tank/lxc

• lxc launch --vm --storage X images:debian/11 vm1
• lxc shell vm1
• apt install fio
• fio --name X --eta-newline=5s --filename=/iobench --rw=randrw --size=2g --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 --iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting

output.dir-lvm.fio.txt

dir-lvm: (groupid=0, jobs=1): err= 0: pid=776: Sat Aug 13 18:55:00 2022
  read: IOPS=108, BW=436KiB/s (446kB/s)(25.5MiB/60001msec)
    slat (usec): min=36, max=547, avg=80.78, stdev=18.61
    clat (usec): min=66, max=1767, avg=548.89, stdev=58.97
     lat (usec): min=361, max=1835, avg=630.83, stdev=59.84
    clat percentiles (usec):
     |  1.00th=[  404],  5.00th=[  441], 10.00th=[  469], 20.00th=[  506],
     | 30.00th=[  523], 40.00th=[  553], 50.00th=[  570], 60.00th=[  578],
     | 70.00th=[  586], 80.00th=[  594], 90.00th=[  603], 95.00th=[  611],
     | 99.00th=[  635], 99.50th=[  668], 99.90th=[  791], 99.95th=[  930],
     | 99.99th=[ 1762]
   bw (  KiB/s): min=  152, max=  681, per=99.86%, avg=435.50, stdev=139.31, samples=119
   iops        : min=   38, max=  170, avg=108.82, stdev=34.78, samples=119
  write: IOPS=109, BW=438KiB/s (448kB/s)(25.7MiB/60001msec); 0 zone resets
    slat (usec): min=58, max=3628, avg=88.94, stdev=45.45
    clat (usec): min=277, max=2602, avg=539.62, stdev=93.40
     lat (usec): min=386, max=4119, avg=629.80, stdev=103.38
    clat percentiles (usec):
     |  1.00th=[  404],  5.00th=[  437], 10.00th=[  457], 20.00th=[  494],
     | 30.00th=[  510], 40.00th=[  529], 50.00th=[  553], 60.00th=[  562],
     | 70.00th=[  570], 80.00th=[  578], 90.00th=[  586], 95.00th=[  594],
     | 99.00th=[  652], 99.50th=[  701], 99.90th=[ 2311], 99.95th=[ 2343],
     | 99.99th=[ 2606]
   bw (  KiB/s): min=  240, max=  577, per=99.82%, avg=437.24, stdev=119.37, samples=119
   iops        : min=   60, max=  144, avg=109.24, stdev=29.79, samples=119
  lat (usec)   : 100=0.02%, 250=0.02%, 500=21.08%, 750=78.60%, 1000=0.10%
  lat (msec)   : 2=0.10%, 4=0.08%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=495, max=76570, avg=1122.27, stdev=916.08
    sync percentiles (nsec):
     |  1.00th=[  868],  5.00th=[ 1048], 10.00th=[ 1064], 20.00th=[ 1064],
     | 30.00th=[ 1080], 40.00th=[ 1096], 50.00th=[ 1096], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1144], 90.00th=[ 1160], 95.00th=[ 1176],
     | 99.00th=[ 1208], 99.50th=[ 1208], 99.90th=[ 1928], 99.95th=[29312],
     | 99.99th=[32128]
  cpu          : usr=1.11%, sys=2.68%, ctx=39588, majf=0, minf=15
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=6534,6567,0,13098 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=436KiB/s (446kB/s), 436KiB/s-436KiB/s (446kB/s-446kB/s), io=25.5MiB (26.8MB), run=60001-60001msec
  WRITE: bw=438KiB/s (448kB/s), 438KiB/s-438KiB/s (448kB/s-448kB/s), io=25.7MiB (26.9MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=6515/29450, merge=0/9823, ticks=3688/53418, in_queue=95666, util=99.92%

output.dir-md.fio.txt

dir-md: (groupid=0, jobs=1): err= 0: pid=3060: Sat Aug 13 23:28:19 2022
  read: IOPS=56, BW=225KiB/s (231kB/s)(13.2MiB/60003msec)
    slat (usec): min=10, max=763, avg=82.26, stdev=20.71
    clat (usec): min=8, max=2432, avg=546.16, stdev=67.69
     lat (usec): min=193, max=2537, avg=629.69, stdev=70.46
    clat percentiles (usec):
     |  1.00th=[  375],  5.00th=[  429], 10.00th=[  465], 20.00th=[  506],
     | 30.00th=[  537], 40.00th=[  553], 50.00th=[  562], 60.00th=[  570],
     | 70.00th=[  578], 80.00th=[  586], 90.00th=[  594], 95.00th=[  603],
     | 99.00th=[  644], 99.50th=[  660], 99.90th=[  906], 99.95th=[ 1467],
     | 99.99th=[ 2442]
   bw (  KiB/s): min=   16, max=  632, per=100.00%, avg=226.17, stdev=107.52, samples=118
   iops        : min=    4, max=  158, avg=56.54, stdev=26.88, samples=118
  write: IOPS=58, BW=233KiB/s (239kB/s)(13.7MiB/60003msec); 0 zone resets
    slat (usec): min=14, max=562, avg=90.35, stdev=18.90
    clat (usec): min=9, max=14131, avg=562.93, stdev=439.53
     lat (usec): min=166, max=14221, avg=654.66, stdev=439.63
    clat percentiles (usec):
     |  1.00th=[  375],  5.00th=[  424], 10.00th=[  453], 20.00th=[  478],
     | 30.00th=[  498], 40.00th=[  529], 50.00th=[  553], 60.00th=[  562],
     | 70.00th=[  562], 80.00th=[  570], 90.00th=[  586], 95.00th=[  742],
     | 99.00th=[ 1090], 99.50th=[ 1221], 99.90th=[10159], 99.95th=[13829],
     | 99.99th=[14091]
   bw (  KiB/s): min=   48, max=  544, per=100.00%, avg=235.12, stdev=104.70, samples=118
   iops        : min=   12, max=  136, avg=58.78, stdev=26.17, samples=118
  lat (usec)   : 10=0.06%, 250=0.13%, 500=23.64%, 750=73.66%, 1000=1.67%
  lat (msec)   : 2=0.71%, 4=0.06%, 10=0.01%, 20=0.06%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=141, max=106598, avg=1251.96, stdev=1446.44
    sync percentiles (nsec):
     |  1.00th=[   908],  5.00th=[  1144], 10.00th=[  1192], 20.00th=[  1208],
     | 30.00th=[  1208], 40.00th=[  1224], 50.00th=[  1224], 60.00th=[  1240],
     | 70.00th=[  1240], 80.00th=[  1256], 90.00th=[  1256], 95.00th=[  1272],
     | 99.00th=[  1288], 99.50th=[  1368], 99.90th=[  3920], 99.95th=[ 26752],
     | 99.99th=[107008]
  cpu          : usr=0.51%, sys=1.50%, ctx=20628, majf=0, minf=17
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3378,3500,0,6875 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=225KiB/s (231kB/s), 225KiB/s-225KiB/s (231kB/s-231kB/s), io=13.2MiB (13.8MB), run=60003-60003msec
  WRITE: bw=233KiB/s (239kB/s), 233KiB/s-233KiB/s (239kB/s-239kB/s), io=13.7MiB (14.3MB), run=60003-60003msec

Disk stats (read/write):
  sda: ios=3375/15600, merge=0/5238, ticks=1904/56468, in_queue=106551, util=99.93%

output.lvm.fio.txt

lvm: (groupid=0, jobs=1): err= 0: pid=3061: Sat Aug 13 22:53:25 2022
  read: IOPS=113, BW=455KiB/s (465kB/s)(26.6MiB/60003msec)
    slat (usec): min=38, max=1965, avg=79.90, stdev=27.38
    clat (usec): min=9, max=36350, avg=561.66, stdev=437.74
     lat (usec): min=347, max=36437, avg=642.77, stdev=438.52
    clat percentiles (usec):
     |  1.00th=[  383],  5.00th=[  437], 10.00th=[  465], 20.00th=[  510],
     | 30.00th=[  537], 40.00th=[  553], 50.00th=[  578], 60.00th=[  586],
     | 70.00th=[  594], 80.00th=[  603], 90.00th=[  611], 95.00th=[  619],
     | 99.00th=[  644], 99.50th=[  668], 99.90th=[  791], 99.95th=[  914],
     | 99.99th=[36439]
   bw (  KiB/s): min=  256, max=  648, per=99.89%, avg=454.72, stdev=83.67, samples=119
   iops        : min=   64, max=  162, avg=113.68, stdev=20.92, samples=119
  write: IOPS=114, BW=457KiB/s (468kB/s)(26.8MiB/60003msec); 0 zone resets
    slat (usec): min=52, max=5983, avg=88.09, stdev=72.49
    clat (usec): min=256, max=4344, avg=985.05, stdev=708.95
     lat (usec): min=345, max=6483, avg=1074.39, stdev=712.04
    clat percentiles (usec):
     |  1.00th=[  355],  5.00th=[  412], 10.00th=[  445], 20.00th=[  478],
     | 30.00th=[  515], 40.00th=[  553], 50.00th=[  562], 60.00th=[  594],
     | 70.00th=[ 1123], 80.00th=[ 2089], 90.00th=[ 2278], 95.00th=[ 2343],
     | 99.00th=[ 2606], 99.50th=[ 2638], 99.90th=[ 2704], 99.95th=[ 2769],
     | 99.99th=[ 4359]
   bw (  KiB/s): min=  328, max=  560, per=99.87%, avg=456.67, stdev=46.63, samples=119
   iops        : min=   82, max=  140, avg=114.17, stdev=11.66, samples=119
  lat (usec)   : 10=0.01%, 100=0.01%, 500=22.35%, 750=58.29%, 1000=1.10%
  lat (msec)   : 2=8.06%, 4=10.18%, 10=0.01%, 50=0.01%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=441, max=41319, avg=1142.80, stdev=891.06
    sync percentiles (nsec):
     |  1.00th=[  748],  5.00th=[  924], 10.00th=[ 1032], 20.00th=[ 1112],
     | 30.00th=[ 1128], 40.00th=[ 1128], 50.00th=[ 1128], 60.00th=[ 1144],
     | 70.00th=[ 1144], 80.00th=[ 1160], 90.00th=[ 1176], 95.00th=[ 1192],
     | 99.00th=[ 1208], 99.50th=[ 1224], 99.90th=[ 2928], 99.95th=[29824],
     | 99.99th=[34560]
  cpu          : usr=0.89%, sys=3.05%, ctx=41315, majf=0, minf=17
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=6818,6849,0,13664 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=455KiB/s (465kB/s), 455KiB/s-455KiB/s (465kB/s-465kB/s), io=26.6MiB (27.9MB), run=60003-60003msec
  WRITE: bw=457KiB/s (468kB/s), 457KiB/s-457KiB/s (468kB/s-468kB/s), io=26.8MiB (28.1MB), run=60003-60003msec

Disk stats (read/write):
  sda: ios=6821/30616, merge=0/10132, ticks=3963/53058, in_queue=91913, util=99.97%

output.zfs-nc.fio.txt

zfs-nc: (groupid=0, jobs=1): err= 0: pid=3062: Sat Aug 13 23:17:56 2022
  read: IOPS=93, BW=373KiB/s (382kB/s)(21.9MiB/60001msec)
    slat (usec): min=13, max=1249, avg=82.58, stdev=26.76
    clat (usec): min=10, max=3579, avg=454.98, stdev=83.56
     lat (usec): min=58, max=3711, avg=538.98, stdev=88.13
    clat percentiles (usec):
     |  1.00th=[  200],  5.00th=[  343], 10.00th=[  379], 20.00th=[  412],
     | 30.00th=[  433], 40.00th=[  461], 50.00th=[  474], 60.00th=[  482],
     | 70.00th=[  486], 80.00th=[  494], 90.00th=[  502], 95.00th=[  515],
     | 99.00th=[  619], 99.50th=[  725], 99.90th=[ 1123], 99.95th=[ 1156],
     | 99.99th=[ 3589]
   bw (  KiB/s): min=  152, max=  616, per=99.88%, avg=373.97, stdev=83.84, samples=119
   iops        : min=   38, max=  154, avg=93.48, stdev=20.98, samples=119
  write: IOPS=93, BW=376KiB/s (385kB/s)(22.0MiB/60001msec); 0 zone resets
    slat (usec): min=12, max=8976, avg=93.58, stdev=130.58
    clat (usec): min=11, max=1314, avg=502.34, stdev=72.83
     lat (usec): min=59, max=9506, avg=597.42, stdev=151.76
    clat percentiles (usec):
     |  1.00th=[  210],  5.00th=[  396], 10.00th=[  437], 20.00th=[  465],
     | 30.00th=[  494], 40.00th=[  515], 50.00th=[  519], 60.00th=[  523],
     | 70.00th=[  529], 80.00th=[  529], 90.00th=[  545], 95.00th=[  553],
     | 99.00th=[  668], 99.50th=[  824], 99.90th=[ 1123], 99.95th=[ 1221],
     | 99.99th=[ 1319]
   bw (  KiB/s): min=  152, max=  472, per=99.81%, avg=375.85, stdev=57.44, samples=119
   iops        : min=   38, max=  118, avg=93.95, stdev=14.39, samples=119
  lat (usec)   : 20=0.03%, 50=0.02%, 100=0.22%, 250=1.00%, 500=59.83%
  lat (usec)   : 750=38.36%, 1000=0.39%
  lat (msec)   : 2=0.14%, 4=0.01%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=122, max=80385, avg=1256.04, stdev=1432.32
    sync percentiles (nsec):
     |  1.00th=[  636],  5.00th=[ 1112], 10.00th=[ 1176], 20.00th=[ 1192],
     | 30.00th=[ 1208], 40.00th=[ 1224], 50.00th=[ 1224], 60.00th=[ 1224],
     | 70.00th=[ 1240], 80.00th=[ 1240], 90.00th=[ 1256], 95.00th=[ 1272],
     | 99.00th=[ 1656], 99.50th=[ 1832], 99.90th=[27008], 99.95th=[30592],
     | 99.99th=[78336]
  cpu          : usr=0.80%, sys=2.56%, ctx=33961, majf=0, minf=18
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5602,5636,0,11235 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=373KiB/s (382kB/s), 373KiB/s-373KiB/s (382kB/s-382kB/s), io=21.9MiB (22.9MB), run=60001-60001msec
  WRITE: bw=376KiB/s (385kB/s), 376KiB/s-376KiB/s (385kB/s-385kB/s), io=22.0MiB (23.1MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=5595/25345, merge=0/8492, ticks=2633/54751, in_queue=100895, util=99.96%

output.zfs-zstd.fio.txt

zfs-zstd: (groupid=0, jobs=1): err= 0: pid=3063: Sat Aug 13 23:10:27 2022
  read: IOPS=96, BW=385KiB/s (394kB/s)(22.6MiB/60004msec)
    slat (usec): min=6, max=754, avg=81.18, stdev=21.47
    clat (usec): min=9, max=6341, avg=451.51, stdev=107.81
     lat (usec): min=38, max=6457, avg=533.98, stdev=111.46
    clat percentiles (usec):
     |  1.00th=[  161],  5.00th=[  338], 10.00th=[  379], 20.00th=[  404],
     | 30.00th=[  433], 40.00th=[  461], 50.00th=[  469], 60.00th=[  478],
     | 70.00th=[  486], 80.00th=[  490], 90.00th=[  502], 95.00th=[  515],
     | 99.00th=[  619], 99.50th=[  660], 99.90th=[  979], 99.95th=[ 1680],
     | 99.99th=[ 6325]
   bw (  KiB/s): min=  128, max=  568, per=100.00%, avg=385.92, stdev=78.07, samples=119
   iops        : min=   32, max=  142, avg=96.46, stdev=19.54, samples=119
  write: IOPS=97, BW=388KiB/s (398kB/s)(22.8MiB/60004msec); 0 zone resets
    slat (usec): min=11, max=8840, avg=90.04, stdev=124.80
    clat (usec): min=37, max=1758, avg=499.95, stdev=79.58
     lat (usec): min=48, max=9320, avg=591.32, stdev=150.03
    clat percentiles (usec):
     |  1.00th=[  167],  5.00th=[  392], 10.00th=[  437], 20.00th=[  465],
     | 30.00th=[  490], 40.00th=[  510], 50.00th=[  519], 60.00th=[  523],
     | 70.00th=[  529], 80.00th=[  529], 90.00th=[  545], 95.00th=[  562],
     | 99.00th=[  668], 99.50th=[  709], 99.90th=[ 1254], 99.95th=[ 1598],
     | 99.99th=[ 1762]
   bw (  KiB/s): min=  160, max=  504, per=99.92%, avg=388.48, stdev=54.43, samples=119
   iops        : min=   40, max=  126, avg=97.10, stdev=13.64, samples=119
  lat (usec)   : 10=0.02%, 20=0.01%, 50=0.07%, 100=0.39%, 250=1.28%
  lat (usec)   : 500=59.09%, 750=38.81%, 1000=0.21%
  lat (msec)   : 2=0.12%, 10=0.01%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=65, max=79621, avg=1231.96, stdev=1218.04
    sync percentiles (nsec):
     |  1.00th=[  426],  5.00th=[ 1012], 10.00th=[ 1176], 20.00th=[ 1192],
     | 30.00th=[ 1208], 40.00th=[ 1224], 50.00th=[ 1224], 60.00th=[ 1240],
     | 70.00th=[ 1240], 80.00th=[ 1256], 90.00th=[ 1256], 95.00th=[ 1272],
     | 99.00th=[ 1608], 99.50th=[ 1784], 99.90th=[ 2256], 99.95th=[28288],
     | 99.99th=[79360]
  cpu          : usr=0.90%, sys=2.48%, ctx=35008, majf=0, minf=19
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5773,5825,0,11595 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=385KiB/s (394kB/s), 385KiB/s-385KiB/s (394kB/s-394kB/s), io=22.6MiB (23.6MB), run=60004-60004msec
  WRITE: bw=388KiB/s (398kB/s), 388KiB/s-388KiB/s (398kB/s-398kB/s), io=22.8MiB (23.9MB), run=60004-60004msec

Disk stats (read/write):
  sda: ios=5757/26058, merge=0/8707, ticks=2697/54574, in_queue=100063, util=99.97%