Skip to content

Instantly share code, notes, and snippets.

@mdcallag
Created November 18, 2022 23:44
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mdcallag/bcfcc8213357ab36b9077934c35977fa to your computer and use it in GitHub Desktop.
Save mdcallag/bcfcc8213357ab36b9077934c35977fa to your computer and use it in GitHub Desktop.
Server:
* Explained at http://smalldatum.blogspot.com/2018/12/new-small-servers-for-performance.html
* Intel NUC with 4 cores, hyperthread disabled, 16G RAM
* Samsung 970 EVO
* Ubuntu 22.04, 5.15.0-52-generic kernel
* fio used 8 files, 20G per file for dir and buf with XFS filesystem
Example command lines, repeated for numjobs in 1 2 4 8 16 32 48 64
raw = no filesystem
fio --filename=/dev/nvme0n1 --direct=1 --rw=randread --bs=4096 --ioengine=psync --iodepth=1 --runtime=600 --numjobs=1 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --eta-interval=1 --randseed=1668657773 --readonly --eta=always
dir = O_DIRECT
fio --filename=/data/m/fio.in.1:/data/m/fio.in.2:/data/m/fio.in.3:/data/m/fio.in.4:/data/m/fio.in.5:/data/m/fio.in.6:/data/m/fio.in.7:/data/m/fio.in.8 --filesize=20480m --rw=randread --bs=4096 --ioengine=psync --iodepth=1 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --eta-interval=1 --randrepeat=0 --norandommap --invalidate=0 --eta=always --direct=1 --numjobs=1 --runtime=600
buf = buffered IO
fio --filename=/data/m/fio.in.1:/data/m/fio.in.2:/data/m/fio.in.3:/data/m/fio.in.4:/data/m/fio.in.5:/data/m/fio.in.6:/data/m/fio.in.7:/data/m/fio.in.8 --filesize=20480m --rw=randread --bs=4096 --ioengine=psync --iodepth=1 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --eta-interval=1 --randrepeat=0 --norandommap --invalidate=0 --eta=always --direct=0 --fadvise_hint=random --numjobs=1 --runtime=600
IOPs per fio
1 2 4 8 16 32 48 64 <- njobs
24470 49785 97255 163767 298737 439166 499544 498873 fio.4kb.raw.psync
9715 19123 38309 86112 143291 207611 237557 252481 fio.4kb.dir.psync
9242 19478 42212 92064 148756 223649 255541 272594 fio.4kb.buf.psync
-
22873 44656 85435 139814 229258 316238 351337 366243 fio.8kb.raw.psync
9116 17612 33060 66826 97391 121228 129733 133908 fio.8kb.dir.psync
8675 17029 36450 71135 105196 131658 141951 146795 fio.8kb.buf.psync
response time per fio - usecs
1 2 4 8 16 32 48 64 <- njobs
40.2 39.5 40.4 47.9 52.7 71.8 95.0 127.0 fio.4kb.raw.psync
100.8 102.2 102.6 92.2 111.0 153.5 201.4 252.9 fio.4kb.dir.psync
105.7 100.8 93.2 86.3 106.8 142.4 187.0 234.1 fio.4kb.buf.psync
-
43.2 43.3 46.2 56.4 68.8 100.2 135.7 173.7 fio.8kb.raw.psync
107.9 111.5 119.2 119.0 163.5 263.3 369.4 477.5 fio.8kb.dir.psync
113.4 115.4 108.4 111.7 151.4 242.4 337.6 435.4 fio.8kb.buf.psync
vmstat us+sy
1 2 4 8 16 32 48 64 <- njobs
3.2 6.0 12.2 28.8 59.6 93.1 98.9 99.9 fio.4kb.raw.psync
5.8 11.3 21.0 14.8 28.8 46.8 54.9 58.2 fio.4kb.dir.psync
7.4 15.1 23.6 22.5 44.4 67.4 80.0 84.2 fio.4kb.buf.psync
-
3.2 5.9 11.5 25.1 48.8 73.4 83.4 87.7 fio.8kb.raw.psync
5.6 11.2 20.0 12.3 21.0 27.6 29.4 31.7 fio.8kb.dir.psync
8.3 17.6 24.0 22.9 37.0 50.9 54.3 57.1 fio.8kb.buf.psync
cpu_usecs per operation
1 2 4 8 16 32 48 64 <- njobs
6.0 5.9 5.9 8.7 7.8 8.2 7.9 8.0 fio.4kb.raw.psync
28.1 28.4 25.1 8.0 8.9 9.4 9.3 9.2 fio.4kb.dir.psync
38.9 36.1 25.3 10.9 13.6 12.1 12.5 12.3 fio.4kb.buf.psync
-
6.4 6.2 6.3 8.8 8.7 8.9 9.1 9.2 fio.8kb.raw.psync
29.2 30.1 28.2 8.6 9.6 9.9 9.9 10.3 fio.8kb.dir.psync
43.1 45.3 28.5 13.7 14.6 16.4 15.9 15.9 fio.8kb.buf.psync
iostat r_await in usecs
1 2 4 8 16 32 48 64 <- njobs
40 39 40 40 45 50 60 60 fio.4kb.raw.psync
80 80 85 90 100 148 190 241 fio.4kb.dir.psync
81 81 81 90 100 140 180 230 fio.4kb.buf.psync
-
40 40 40 50 60 90 120 160 fio.8kb.raw.psync
90 90 100 110 160 260 360 470 fio.8kb.dir.psync
90 90 100 110 150 250 350 452 fio.8kb.buf.psync
iostat reads per query
1 2 4 8 16 32 48 64 <- njobs
~1 ~1 ~1 ~1 ~1 ~1 ~1 ~1 fio.4kb.raw.psync
~1 ~1 ~1 ~1 ~1 ~1 ~1 ~1 fio.4kb.dir.psync
~1 ~1 ~1 ~1 ~1 ~1 ~1 ~1 fio.4kb.buf.psync
-
~1 ~1 ~1 ~1 ~1 ~1 ~1 ~1 fio.8kb.raw.psync
~1 ~1 ~1 ~1 ~1 ~1 ~1 ~1 fio.8kb.dir.psync
~1 ~1 ~1 ~1 ~1 ~1 ~1 ~1 fio.8kb.buf.psync
iostat rareq-sz in KB
1 2 4 8 16 32 48 64 <- njobs
4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 fio.4kb.raw.psync
4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 fio.4kb.dir.psync
4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 fio.4kb.buf.psync
-
8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0 fio.8kb.raw.psync
8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0 fio.8kb.dir.psync
7.8 8.0 8.0 8.0 8.0 8.0 8.0 8.0 fio.8kb.buf.psync
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment