https://www.reddit.com/r/Proxmox/comments/y5fltr/

1D0S-read

1D0S-read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.30
Starting 1 process
1D0S-read: Laying out IO file (1 file / 5120MiB)

1D0S-read: (groupid=0, jobs=1): err= 0: pid=2160: Sun Oct 16 19:38:10 2022
  read: IOPS=2240, BW=8963KiB/s (9178kB/s)(263MiB/30001msec)
    clat (usec): min=216, max=89999, avg=444.53, stdev=925.64
     lat (usec): min=216, max=89999, avg=444.71, stdev=925.65
    clat percentiles (usec):
     |  1.00th=[  233],  5.00th=[  245], 10.00th=[  251], 20.00th=[  265],
     | 30.00th=[  277], 40.00th=[  322], 50.00th=[  424], 60.00th=[  449],
     | 70.00th=[  465], 80.00th=[  494], 90.00th=[  553], 95.00th=[  750],
     | 99.00th=[ 1106], 99.50th=[ 1237], 99.90th=[12256], 99.95th=[14353],
     | 99.99th=[49021]
   bw (  KiB/s): min= 5904, max=14208, per=99.43%, avg=8912.95, stdev=2061.26, samples=59
   iops        : min= 1476, max= 3552, avg=2228.24, stdev=515.32, samples=59
  lat (usec)   : 250=8.94%, 500=73.29%, 750=12.82%, 1000=3.51%
  lat (msec)   : 2=1.18%, 4=0.01%, 10=0.10%, 20=0.13%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.98%, sys=2.67%, ctx=67225, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=67225,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=8963KiB/s (9178kB/s), 8963KiB/s-8963KiB/s (9178kB/s-9178kB/s), io=263MiB (275MB), run=30001-30001msec

Disk stats (read/write):
  sdb: ios=66653/7, merge=0/3, ticks=28905/269, in_queue=29197, util=99.74%

1D0S-readwrite

1D0S-readwrite: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.30
Starting 1 process

1D0S-readwrite: (groupid=0, jobs=1): err= 0: pid=2166: Sun Oct 16 19:39:10 2022
  read: IOPS=283, BW=1132KiB/s (1159kB/s)(33.2MiB/30002msec)
    clat (usec): min=149, max=16232, avg=416.37, stdev=281.39
     lat (usec): min=149, max=16232, avg=416.60, stdev=281.42
    clat percentiles (usec):
     |  1.00th=[  184],  5.00th=[  208], 10.00th=[  237], 20.00th=[  281],
     | 30.00th=[  306], 40.00th=[  326], 50.00th=[  351], 60.00th=[  383],
     | 70.00th=[  482], 80.00th=[  562], 90.00th=[  668], 95.00th=[  750],
     | 99.00th=[ 1045], 99.50th=[ 1156], 99.90th=[ 1303], 99.95th=[ 1385],
     | 99.99th=[16188]
   bw (  KiB/s): min=  584, max= 1760, per=99.73%, avg=1129.63, stdev=248.83, samples=59
   iops        : min=  146, max=  440, avg=282.41, stdev=62.21, samples=59
  write: IOPS=283, BW=1135KiB/s (1162kB/s)(33.3MiB/30002msec); 0 zone resets
    clat (usec): min=1885, max=198527, avg=3104.73, stdev=4165.34
     lat (usec): min=1886, max=198527, avg=3105.00, stdev=4165.35
    clat percentiles (usec):
     |  1.00th=[  1991],  5.00th=[  2057], 10.00th=[  2114], 20.00th=[  2180],
     | 30.00th=[  2245], 40.00th=[  2311], 50.00th=[  2376], 60.00th=[  2442],
     | 70.00th=[  2507], 80.00th=[  2704], 90.00th=[  4555], 95.00th=[  5800],
     | 99.00th=[ 13566], 99.50th=[ 24773], 99.90th=[ 51119], 99.95th=[ 66323],
     | 99.99th=[198181]
   bw (  KiB/s): min=  600, max= 1496, per=99.82%, avg=1133.42, stdev=230.89, samples=59
   iops        : min=  150, max=  374, avg=283.36, stdev=57.72, samples=59
  lat (usec)   : 250=6.01%, 500=29.88%, 750=11.62%, 1000=1.79%
  lat (msec)   : 2=1.25%, 4=42.48%, 10=5.95%, 20=0.74%, 50=0.24%
  lat (msec)   : 100=0.04%, 250=0.01%
  cpu          : usr=0.12%, sys=0.97%, ctx=17006, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=8491,8513,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1132KiB/s (1159kB/s), 1132KiB/s-1132KiB/s (1159kB/s-1159kB/s), io=33.2MiB (34.8MB), run=30002-30002msec
  WRITE: bw=1135KiB/s (1162kB/s), 1135KiB/s-1135KiB/s (1162kB/s-1162kB/s), io=33.3MiB (34.9MB), run=30002-30002msec

Disk stats (read/write):
  sdb: ios=8457/8477, merge=0/0, ticks=3404/26230, in_queue=29640, util=99.77%

1D0S-write

1D0S-write: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.30
Starting 1 process

1D0S-write: (groupid=0, jobs=1): err= 0: pid=2163: Sun Oct 16 19:38:40 2022
  write: IOPS=321, BW=1284KiB/s (1315kB/s)(37.6MiB/30003msec); 0 zone resets
    clat (usec): min=1896, max=110550, avg=3112.28, stdev=2979.46
     lat (usec): min=1896, max=110551, avg=3112.53, stdev=2979.46
    clat percentiles (usec):
     |  1.00th=[  1991],  5.00th=[  2057], 10.00th=[  2114], 20.00th=[  2212],
     | 30.00th=[  2278], 40.00th=[  2343], 50.00th=[  2409], 60.00th=[  2474],
     | 70.00th=[  2606], 80.00th=[  2835], 90.00th=[  4817], 95.00th=[  5735],
     | 99.00th=[ 13173], 99.50th=[ 17695], 99.90th=[ 43779], 99.95th=[ 47973],
     | 99.99th=[110625]
   bw (  KiB/s): min=  800, max= 1688, per=100.00%, avg=1287.73, stdev=253.41, samples=59
   iops        : min=  200, max=  422, avg=321.93, stdev=63.35, samples=59
  lat (msec)   : 2=1.18%, 4=83.32%, 10=13.65%, 20=1.39%, 50=0.43%
  lat (msec)   : 100=0.02%, 250=0.01%
  cpu          : usr=0.19%, sys=0.49%, ctx=9633, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,9633,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1284KiB/s (1315kB/s), 1284KiB/s-1284KiB/s (1315kB/s-1315kB/s), io=37.6MiB (39.5MB), run=30003-30003msec

Disk stats (read/write):
  sdb: ios=0/9611, merge=0/0, ticks=0/29745, in_queue=29755, util=99.77%

1D5S-read

1D5S-read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.30
Starting 1 process

1D5S-read: (groupid=0, jobs=1): err= 0: pid=2169: Sun Oct 16 19:40:16 2022
  read: IOPS=2332, BW=9331KiB/s (9555kB/s)(273MiB/30001msec)
    clat (usec): min=217, max=89909, avg=426.88, stdev=913.42
     lat (usec): min=217, max=89909, avg=427.06, stdev=913.43
    clat percentiles (usec):
     |  1.00th=[  233],  5.00th=[  243], 10.00th=[  251], 20.00th=[  262],
     | 30.00th=[  273], 40.00th=[  302], 50.00th=[  404], 60.00th=[  429],
     | 70.00th=[  449], 80.00th=[  478], 90.00th=[  545], 95.00th=[  709],
     | 99.00th=[  988], 99.50th=[ 1106], 99.90th=[11207], 99.95th=[14615],
     | 99.99th=[47449]
   bw (  KiB/s): min= 6072, max=13584, per=99.92%, avg=9323.15, stdev=1714.58, samples=59
   iops        : min= 1518, max= 3396, avg=2330.78, stdev=428.66, samples=59
  lat (usec)   : 250=9.16%, 500=76.01%, 750=10.73%, 1000=3.20%
  lat (msec)   : 2=0.66%, 4=0.01%, 10=0.11%, 20=0.09%, 50=0.02%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.89%, sys=2.93%, ctx=69984, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=69984,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=9331KiB/s (9555kB/s), 9331KiB/s-9331KiB/s (9555kB/s-9555kB/s), io=273MiB (287MB), run=30001-30001msec

Disk stats (read/write):
  sdc: ios=69680/8, merge=0/6, ticks=28861/439, in_queue=29372, util=99.75%

1D5S-readwrite

1D5S-readwrite: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.30
Starting 1 process

1D5S-readwrite: (groupid=0, jobs=1): err= 0: pid=2182: Sun Oct 16 19:45:27 2022
  read: IOPS=279, BW=1117KiB/s (1144kB/s)(32.7MiB/30001msec)
    clat (usec): min=160, max=13797, avg=518.72, stdev=303.07
     lat (usec): min=160, max=13797, avg=518.94, stdev=303.11
    clat percentiles (usec):
     |  1.00th=[  184],  5.00th=[  219], 10.00th=[  265], 20.00th=[  318],
     | 30.00th=[  363], 40.00th=[  453], 50.00th=[  515], 60.00th=[  562],
     | 70.00th=[  611], 80.00th=[  676], 90.00th=[  758], 95.00th=[  873],
     | 99.00th=[ 1156], 99.50th=[ 1205], 99.90th=[ 1385], 99.95th=[ 3326],
     | 99.99th=[13829]
   bw (  KiB/s): min=  752, max= 1532, per=99.60%, avg=1113.24, stdev=226.97, samples=59
   iops        : min=  188, max=  383, avg=278.15, stdev=56.81, samples=59
  write: IOPS=280, BW=1120KiB/s (1147kB/s)(32.8MiB/30001msec); 0 zone resets
    clat (usec): min=1857, max=193042, avg=3048.46, stdev=3943.70
     lat (usec): min=1857, max=193042, avg=3048.72, stdev=3943.71
    clat percentiles (usec):
     |  1.00th=[  1991],  5.00th=[  2114], 10.00th=[  2180], 20.00th=[  2245],
     | 30.00th=[  2311], 40.00th=[  2376], 50.00th=[  2442], 60.00th=[  2507],
     | 70.00th=[  2606], 80.00th=[  2737], 90.00th=[  3490], 95.00th=[  5342],
     | 99.00th=[ 13042], 99.50th=[ 17957], 99.90th=[ 57410], 99.95th=[ 62129],
     | 99.99th=[193987]
   bw (  KiB/s): min=  752, max= 1477, per=99.97%, avg=1120.68, stdev=190.84, samples=59
   iops        : min=  188, max=  369, avg=280.02, stdev=47.75, samples=59
  lat (usec)   : 250=4.41%, 500=19.11%, 750=21.18%, 1000=3.60%
  lat (msec)   : 2=2.14%, 4=44.80%, 10=3.85%, 20=0.67%, 50=0.16%
  lat (msec)   : 100=0.05%, 250=0.02%
  cpu          : usr=0.33%, sys=0.80%, ctx=16784, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=8381,8403,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1117KiB/s (1144kB/s), 1117KiB/s-1117KiB/s (1144kB/s-1144kB/s), io=32.7MiB (34.3MB), run=30001-30001msec
  WRITE: bw=1120KiB/s (1147kB/s), 1120KiB/s-1120KiB/s (1147kB/s-1147kB/s), io=32.8MiB (34.4MB), run=30001-30001msec

Disk stats (read/write):
  sdc: ios=8347/8366, merge=0/0, ticks=4210/25396, in_queue=29606, util=99.76%

1D5S-write

1D5S-write: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.30
Starting 1 process

1D5S-write: (groupid=0, jobs=1): err= 0: pid=2172: Sun Oct 16 19:40:46 2022
  write: IOPS=332, BW=1331KiB/s (1363kB/s)(39.0MiB/30001msec); 0 zone resets
    clat (usec): min=1845, max=198964, avg=3003.52, stdev=4017.65
     lat (usec): min=1845, max=198964, avg=3003.78, stdev=4017.65
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    3], 90.00th=[    4], 95.00th=[    6],
     | 99.00th=[   14], 99.50th=[   19], 99.90th=[   56], 99.95th=[   66],
     | 99.99th=[  199]
   bw (  KiB/s): min=  712, max= 1656, per=99.72%, avg=1327.32, stdev=231.93, samples=59
   iops        : min=  178, max=  414, avg=331.83, stdev=57.98, samples=59
  lat (msec)   : 2=0.86%, 4=90.39%, 10=6.73%, 20=1.54%, 50=0.36%
  lat (msec)   : 100=0.09%, 250=0.02%
  cpu          : usr=0.23%, sys=0.46%, ctx=9981, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,9981,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1331KiB/s (1363kB/s), 1331KiB/s-1331KiB/s (1363kB/s-1363kB/s), io=39.0MiB (40.9MB), run=30001-30001msec

Disk stats (read/write):
  sdc: ios=0/9944, merge=0/0, ticks=0/29755, in_queue=29766, util=99.78%

CEPH.md

Ceph Setup

Node Setup

• 6 Nodes
  • A: E3-1230v3, 32GB
  • B: E3-1230v3, 32GB
  • C: E3-1230v3, 32GB
  • D: i5-6500T, 32GB
  • E: i5-6500T, 24GB
  • F: i5-3210M, 16GB
• 3 OSD Server Each HDD gets 50G block.db-Device on consumer LVM-STRIPE over 2 SATA SSDs (no protection whatsoever)
  • A: 4x 2TB Consumer HDD (misc, Toshiba, WD)
  • B: 4x 2TB Consumer HDD (misc, Toshiba, WD)
  • C: 6x 1TB Consumer HDD (misc, Toshiba, WD)
• Networking: OVS LACP Mode balance-tcp over two Gigabit ports

Ceph RBD Pool for Proxmox VMs is replicated, size 2, min 1

rbd create default/vm-X-disk-1 --size 10G
rbd create default/vm-X-disk-2 --size 10G --stripe-count 5 --stripe-unit 4M

Stripe-Count calculated as: 20 OSDs, with replica 2 10 remain, half that

VM

Alpine Linux 3.16.2, installed xfsprogs, fio

CPU: Set to IvyBridge, running on server B, with 4 cores

Memory: 2GB