Skip to content

Instantly share code, notes, and snippets.

@rzarzynski

rzarzynski/20210210_crimson_alien_op_num_threads.md Secret

Last active February 17, 2021 13:20
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save rzarzynski/c851212f4b0baef4097a6087533ba17b to your computer and use it in GitHub Desktop.
Save rzarzynski/c851212f4b0baef4097a6087533ba17b to your computer and use it in GitHub Desktop.
Download ZIP
Raw
20210210_crimson_alien_op_num_threads.md

The impact of crimson_alien_op_num_threads

Conclusions

  • 16 threads is too much; it does not provide peak performance.
  • 2 threads is also nonoptimal.
  • For the tested hardware the sweet spot is somewhere around 4.
  • The difference between the sweet spot and other arragnements can be significant – around 25% in comparison to both 16 and 2 threads.

4 KB random reads

16 threads – 25848 IOPS

Total time run:       30.0419
Total reads made:     388284
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   50.4872
Average IOPS:         12924
Stddev IOPS:          251.975
Max IOPS:             13427
Min IOPS:             12024
Average Latency(s):   0.00123477
Max latency(s):       0.0429529
Total time run:       30.0419
Min latency(s):       0.000108876
Total reads made:     388264
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   50.4846
Average IOPS:         12924
Stddev IOPS:          258.218
Max IOPS:             13417
Min IOPS:             11979
Average Latency(s):   0.00123357
Max latency(s):       0.0437524
Min latency(s):       9.9422e-05

MGR=0 OSD=1 MDS=0 ../src/vstart.sh -l -b -n --crimson -o  "crimson_alien_op_num_threads=16"

8 threads – 26728 IOPS

Total time run:       30.0007
Total reads made:     401203
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   52.2388
Average IOPS:         13373
Stddev IOPS:          330.127
Max IOPS:             13995
Min IOPS:             12696
Average Latency(s):   0.00119383
Max latency(s):       0.00964449
Min latency(s):       0.000255265
   30      14    401215    401201   52.2317   49.9258 0.000633079  0.00119382
[rzarzynski@o06 build]$ Total time run:       30.0412
Total reads made:     401215
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   52.1699
Average IOPS:         13355
Stddev IOPS:          342.358
Max IOPS:             13987
Min IOPS:             12696
Average Latency(s):   0.00119522
Max latency(s):       0.0415767
Min latency(s):       0.000263606

MGR=0 OSD=1 MDS=0 ../src/vstart.sh -l -b -n --crimson -o  "crimson_alien_op_num_threads=8"

4 threads – 33212 IOPS

Total time run:       30.0416
Total reads made:     498885
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   64.869
Average IOPS:         16606
Stddev IOPS:          583.958
Max IOPS:             17414
Min IOPS:             15485
Average Latency(s):   0.000960445
Max latency(s):       0.0424199
Min latency(s):       0.00034168
Total time run:       30.0417
Total reads made:     498896
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   64.8703
Average IOPS:         16606
Stddev IOPS:          585.999
Max IOPS:             17413
Min IOPS:             15441
Average Latency(s):   0.00096087
Max latency(s):       0.0422955
Min latency(s):       0.000357209

MGR=0 OSD=1 MDS=0 ../src/vstart.sh -l -b -n --crimson -o  "crimson_alien_op_num_threads=4"

2 threads – 26532

Total time run:       30.0011
Total reads made:     398276
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   51.857
Average IOPS:         13275
Stddev IOPS:          1169.04
Max IOPS:             15152
Min IOPS:             10546
Average Latency(s):   0.00120364
Max latency(s):       0.00942078
Min latency(s):       0.000484212
Total time run:       30.0409
Total reads made:     398264
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   51.7867
Average IOPS:         13257
Stddev IOPS:          1192.36
Max IOPS:             15152
Min IOPS:             10543
Average Latency(s):   0.00120466
Max latency(s):       0.0420888
Min latency(s):       0.000393485

MGR=0 OSD=1 MDS=0 ../src/vstart.sh -l -b -n --crimson -o  "crimson_alien_op_num_threads=2"

1 thread – 13266 IOPS

Total time run:       30.0022
Total reads made:     199025
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   25.9128
Average IOPS:         6633
Stddev IOPS:          429.952
Max IOPS:             7356
Min IOPS:             5843
Average Latency(s):   0.00241028
Max latency(s):       0.0113926
Min latency(s):       0.000538047
Total time run:       30.0024
Total reads made:     199033
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   25.9137
Average IOPS:         6633
Stddev IOPS:          429.763
Max IOPS:             7356
Min IOPS:             5837
Average Latency(s):   0.00241056
Max latency(s):       0.00989819
Min latency(s):       0.000635115

MGR=0 OSD=1 MDS=0 ../src/vstart.sh -l -b -n --crimson -o  "crimson_alien_op_num_threads=1"

Envinroment

  • Commit ID: b4d9cc45d6ff1ea5382954dece424128b478d6f7.
  • Host: o06.front.sepia.ceph.com.

Commands

Deployment

[rzarzynski@o06 build]$ MGR=0 OSD=1 MDS=0 ../src/vstart.sh -l -b -n --crimson -o  "crimson_alien_op_num_threads=<num>"
  ...

Provisiong

[rzarzynski@o06 build]$ bin/ceph -c ceph.conf osd pool create test-pool 128 128 && bin/ceph -c ceph.conf osd pool set test-pool size 1 --yes-i-really-mean-it && bin/ceph -c ceph.conf osd pool set test-pool min_size 1
  ...
[rzarzynski@o06 build]$ bin/rados bench -p test-pool 120 write -b 4K --no-cleanup -c ceph.conf
  ...

Testing

[rzarzynski@o06 build]$ bin/rados bench -p test-pool 30 rand -c ceph.conf & bin/rados bench -p test-pool 30 rand -c ceph.conf
 ...
@rzarzynski
Copy link
Author

rzarzynski commented Feb 17, 2021 •
edited

Update: 6 threads – 28441 IOPS

Total time run:       30.0013
Total reads made:     426923
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   55.5866
Average IOPS:         14230
Stddev IOPS:          519.837
Max IOPS:             14911
Min IOPS:             13237
Average Latency(s):   0.00112084
Max latency(s):       0.00629479
Min latency(s):       0.000311102
Total time run:       30.0418
Total reads made:     426943
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   55.5141
Average IOPS:         14211
Stddev IOPS:          533.308
Max IOPS:             14917
Min IOPS:             13224
Average Latency(s):   0.00112225
Max latency(s):       0.0427185
Min latency(s):       0.000321506

MGR=0 OSD=1 MDS=0 ../src/vstart.sh -l -b -n --crimson -o  "crimson_alien_op_num_threads=6"

Update: 3 threads – 31595 IOPS

Total time run:       30.041
Total reads made:     465962
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   60.5894
Average IOPS:         15510
Stddev IOPS:          739.423
Max IOPS:             17370
Min IOPS:             14009
Average Latency(s):   0.00102658
Max latency(s):       0.0415566
Min latency(s):       0.000301449
Total time run:       30.041
Total reads made:     483233
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   62.835
Average IOPS:         16085
Stddev IOPS:          528.691
Max IOPS:             17365
Min IOPS:             14973
Average Latency(s):   0.000991553
Max latency(s):       0.0418695
Min latency(s):       0.000278118

MGR=0 OSD=1 MDS=0 ../src/vstart.sh -l -b -n --crimson -o  "crimson_alien_op_num_threads=3"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment