- 4 KB random reads with two
rados benchinstances over - single OSD instance
- deployed on developer's laptop
- with minimal content stored in
memstore/cyanstore.
>>> 191584887006. / (721754+710988)
133719.04153434464
>>> 83805740629. / (840753+839448)
49878.40182751945
This was quick & rough testing. It has been made on current devel branch of rzarzynski/ceph which bases on e4be7ee (master from Fri Dec 20 10:13:37 2019 +0100).
build$ git describe
v15.0.0-8732-g4ae0dc7
build$ MDS=0 MGR=1 OSD=1 MON=1 ../src/vstart.sh -n --without-dashboard --memstore -X -o "memstore_device_bytes=16777216" --nolockdep
...
build$ bin/ceph osd pool create test-pool 128 128 && bin/ceph osd pool set test-pool size 1 && bin/ceph osd pool set test-pool min_size 1
...
build$ bin/rados bench -p test-pool 1 write -b 4096 --no-cleanup
...
build$ perf stat -p `pgrep -u ${UID} ceph-osd` & bin/rados bench -p test-pool 30 rand & bin/rados bench -p test-pool 30 rand ; sleep 1; killall -INT perf
[1] 13994
[2] 13995
2020-01-13T13:08:02.403+0100 7f154722fa40 -1 WARNING: all dangerous and experimental features are enabled.
2020-01-13T13:08:02.403+0100 7f9a21d0fa40 -1 WARNING: all dangerous and experimental features are enabled.
2020-01-13T13:08:02.407+0100 7f154722fa40 -1 WARNING: all dangerous and experimental features are enabled.
2020-01-13T13:08:02.407+0100 7f9a21d0fa40 -1 WARNING: all dangerous and experimental features are enabled.
2020-01-13T13:08:02.407+0100 7f154722fa40 -1 WARNING: all dangerous and experimental features are enabled.
2020-01-13T13:08:02.407+0100 7f9a21d0fa40 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 11 11 0 0 0 - 0
1 16 23529 23513 91.834 91.8477 0.000651826 0.000674327
1 16 22933 22917 89.4948 89.5195 0.000560601 0.000691812
2 16 48139 48123 93.9791 96.1328 0.000551525 0.000659053
2 15 46744 46729 91.2513 93.0156 0.000600909 0.000678769
3 15 72637 72622 94.5498 95.6992 0.000526213 0.000655018
3 16 70465 70449 91.7171 92.6562 0.00074539 0.000675323
4 16 97116 97100 94.8145 95.6172 0.000799483 0.00065323
4 16 94314 94298 92.076 93.1602 0.000697524 0.000672717
5 16 121660 121644 95.0249 95.875 0.000596772 0.000651845
5 16 118416 118400 92.4889 94.1484 0.000654014 0.000669748
6 16 145780 145764 94.8892 94.2188 0.000502759 0.000652753
6 15 142384 142369 92.6768 93.6289 0.000620542 0.000668396
7 15 170214 170199 94.9678 95.4492 0.00064981 0.00065221
7 16 166418 166402 92.8475 93.8789 0.000644634 0.000667181
8 16 194752 194736 95.0767 95.8477 0.000549491 0.000651488
8 16 190331 190315 92.9169 93.4102 0.000707438 0.000666686
9 15 218993 218978 95.0333 94.6953 0.000552587 0.000651767
9 15 214087 214072 92.9032 92.8008 0.000641616 0.000666777
10 15 243273 243258 95.0135 94.8438 0.000729009 0.000651904
10 16 237872 237856 92.9027 92.9062 0.000862878 0.000666775
11 15 267074 267059 94.8273 92.9727 0.000653102 0.000653183
11 16 261348 261332 92.793 91.7031 0.000663187 0.000667562
12 16 291098 291082 94.7443 93.8398 0.000714927 0.000653739
12 16 285094 285078 92.789 92.7578 0.000696678 0.000667598
13 16 315036 315020 94.6485 93.5078 0.000276414 0.000654408
13 16 308927 308911 92.8122 93.0977 0.000921096 0.000667429
14 15 338985 338970 94.5697 93.5547 0.00056719 0.00065494
14 16 332731 332715 92.824 92.9844 0.000805716 0.000667332
15 16 362982 362966 94.5134 93.7344 0.000560417 0.000655339
15 16 356285 356269 92.769 92.0078 0.000900324 0.00066773
16 15 386685 386670 94.3929 92.5938 0.000872149 0.000656174
16 15 379704 379689 92.6882 91.4844 0.000591255 0.000668338
17 16 410638 410622 94.3436 93.5625 0.000532456 0.000656509
17 16 403248 403232 92.6452 91.9648 0.000743686 0.000668646
18 16 434769 434753 94.3386 94.2617 0.000698493 0.00065655
18 16 426853 426837 92.6205 92.207 0.00074412 0.000668828
19 16 458685 458669 94.2895 93.4219 0.000475539 0.000656904
19 16 450550 450534 92.6171 92.5664 0.000439915 0.000668852
2020-01-13T13:08:22.419365+0100 min lat: 0.000149753 max lat: 0.0110956 avg lat: 0.000657487
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 482401 482385 94.2066 92.6406 0.000656559 0.000657487
2020-01-13T13:08:22.421140+0100 min lat: 0.000108124 max lat: 0.0087571 avg lat: 0.000668797
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 474294 474278 92.6233 92.75 0.000566698 0.000668797
21 16 506405 506389 94.185 93.7656 0.000733651 0.000657639
21 16 498080 498064 92.6366 92.9141 0.000687501 0.000668702
22 16 530403 530387 94.1646 93.7422 0.000709908 0.000657777
22 15 521606 521591 92.6029 91.9023 0.000610151 0.000668947
23 16 554385 554369 94.1432 93.6797 0.00053764 0.000657916
23 16 545202 545186 92.5836 92.168 0.000786732 0.000669078
24 16 578227 578211 94.1007 93.1328 0.000728895 0.000658219
24 16 568949 568933 92.5907 92.7617 0.000923209 0.000669021
25 16 602061 602045 94.0604 93.1016 0.000721368 0.000658506
25 16 592666 592650 92.5926 92.6445 0.000574483 0.000669013
26 16 625928 625912 94.0279 93.2305 0.000865093 0.000658729
26 16 616305 616289 92.5821 92.3398 0.000935574 0.00066908
27 15 649826 649811 94.0027 93.3555 0.000616437 0.000658908
27 16 640044 640028 92.5873 92.7305 0.000596739 0.000669025
28 16 673756 673740 93.9832 93.4727 0.000579403 0.000659041
28 16 663894 663878 92.6074 93.1641 0.000603496 0.000668889
29 16 697857 697841 93.9882 94.1445 0.000737833 0.000659005
29 16 687338 687322 92.5716 91.5781 0.00054358 0.000669146
Total time run: 30.0007
Total reads made: 721754
Read size: 4096
Object size: 4096
Bandwidth (MB/sec): 93.9761
Average IOPS: 24057
Stddev IOPS: 282.341
Max IOPS: 24610
Min IOPS: 23513
Average Latency(s): 0.000659097
Max latency(s): 0.0110956
Min latency(s): 0.000149753
Total time run: 30.0003
Total reads made: 710988
Read size: 4096
Object size: 4096
Bandwidth (MB/sec): 92.5757
Average IOPS: 23699
Stddev IOPS: 224.325
Max IOPS: 24102
Min IOPS: 22917
Average Latency(s): 0.000669122
Max latency(s): 0.0087571
Min latency(s): 0.000108124
[2]+ Done bin/rados bench -p test-pool 30 rand
Performance counter stats for process id '13830':
72696,762039 task-clock (msec) # 2,362 CPUs utilized
913702 context-switches # 0,013 M/sec
198207 cpu-migrations # 0,003 M/sec
1704 page-faults # 0,023 K/sec
191584887006 cycles # 2,635 GHz
101617824099 instructions # 0,53 insn per cycle
19658622946 branches # 270,420 M/sec
328630857 branch-misses # 1,67% of all branches
30,779104553 seconds time elapsed
[1]+ Interrupt perf stat -p `pgrep -u ${UID} ceph-osd`
build$ MDS=0 MGR=1 OSD=1 MON=1 ../src/vstart.sh -n --without-dashboard --memstore -X -o "memstore_device_bytes=16777216" --nolockdep --crimson --nodaemon --redirect-output
...
build$ bin/ceph osd pool create test-pool 128 128 && bin/ceph osd pool set test-pool size 1 && bin/ceph osd pool set test-pool min_size 1
...
build$ bin/rados bench -p test-pool 1 write -b 4096 --no-cleanup
...
build$ perf stat -p `pgrep -u ${UID} crimson-osd` & bin/rados bench -p test-pool 30 rand & bin/rados bench -p test-pool 30 rand ; sleep 1; killall -INT perf
[1] 13087
[2] 13088
2020-01-13T13:06:43.647+0100 7f941b89ea40 -1 WARNING: all dangerous and experimental features are enabled.
2020-01-13T13:06:43.651+0100 7fb097caba40 -1 WARNING: all dangerous and experimental features are enabled.
2020-01-13T13:06:43.651+0100 7f941b89ea40 -1 WARNING: all dangerous and experimental features are enabled.
2020-01-13T13:06:43.655+0100 7f941b89ea40 -1 WARNING: all dangerous and experimental features are enabled.
2020-01-13T13:06:43.655+0100 7fb097caba40 -1 WARNING: all dangerous and experimental features are enabled.
2020-01-13T13:06:43.655+0100 7fb097caba40 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 16 16 0 0 0 - 0
1 16 28261 28245 110.305 110.332 0.000420313 0.000562056
1 15 27792 27777 108.452 108.504 0.000570422 0.000571558
2 16 56170 56154 109.658 109.02 0.00075461 0.000565304
2 16 55563 55547 108.46 108.477 0.00064393 0.000571536
3 15 84872 84857 110.476 112.121 0.000576251 0.000561021
3 16 83918 83902 109.224 110.762 0.000554372 0.000567576
4 16 113191 113175 110.509 110.617 0.000744813 0.000560797
4 15 111933 111918 109.275 109.438 0.000647519 0.000567304
5 16 140271 140255 109.561 105.781 0.00042758 0.00056579
5 15 139575 139560 109.014 107.977 0.000616637 0.000568586
6 16 168556 168540 109.714 110.488 0.000655561 0.000565023
6 16 167668 167652 109.132 109.734 0.000512408 0.000567937
7 16 195283 195267 108.954 104.402 0.000376429 0.000569019
7 16 194316 194300 108.411 104.094 0.000611379 0.000571823
8 16 223522 223506 109.123 110.309 0.000627115 0.000568112
8 16 222154 222138 108.452 108.742 0.000698593 0.000571622
9 16 252053 252037 109.38 111.449 0.000628799 0.000566792
9 15 250787 250772 108.828 111.852 0.00054371 0.000569629
10 15 279430 279415 109.136 106.945 0.000623571 0.000568093
10 16 277527 277511 108.39 104.449 0.000548561 0.00057192
11 16 308553 308537 109.555 113.758 0.000251312 0.00056589
11 16 306446 306430 108.805 112.965 0.000555802 0.000569704
12 16 336772 336756 109.611 110.23 0.000615323 0.000565602
12 16 334772 334756 108.958 110.648 0.000546309 0.000568897
13 16 364774 364758 109.592 109.383 0.000525704 0.000565673
13 15 362758 362743 108.985 109.324 0.000573407 0.000568723
14 16 392049 392033 109.374 106.543 0.000649921 0.00056681
14 16 390106 390090 108.83 106.824 0.00044337 0.000569566
15 16 419665 419649 109.273 107.875 0.000788269 0.000567337
15 16 417951 417935 108.826 108.77 0.000497266 0.000569613
16 16 448267 448251 109.426 111.727 0.000498347 0.00056655
16 16 446867 446851 109.083 112.953 0.000536066 0.000568273
17 16 476482 476466 109.472 110.215 0.000541026 0.000566284
17 16 474516 474500 109.019 108.004 0.000526912 0.000568541
18 16 504686 504670 109.51 110.172 0.000524527 0.000566071
18 15 502605 502590 109.058 109.727 0.00041388 0.000568329
19 16 532442 532426 109.453 108.422 0.000875226 0.000566362
19 16 529941 529925 108.938 106.777 0.000580201 0.000568964
2020-01-13T13:07:03.671957+0100 min lat: 0.000218156 max lat: 0.00213245 avg lat: 0.000566596
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 15 560238 560223 109.409 108.582 0.000307219 0.000566596
2020-01-13T13:07:03.674742+0100 min lat: 0.000162726 max lat: 0.00255079 avg lat: 0.000569325
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 15 557493 557478 108.872 107.629 0.000578193 0.000569325
21 16 588017 588001 109.365 108.508 0.000456627 0.000566837
21 16 585847 585831 108.961 110.754 0.00062144 0.000568867
22 16 616433 616417 109.439 111 0.000572106 0.000566445
22 15 614438 614423 109.084 111.688 0.000537598 0.000568214
23 16 645127 645111 109.554 112.086 0.000566797 0.000565855
23 16 642729 642713 109.146 110.508 0.000487386 0.000567893
24 16 672566 672550 109.454 107.184 0.000470955 0.000566369
24 16 670302 670286 109.086 107.707 0.00066905 0.000568222
25 15 699557 699542 109.293 105.438 0.000550335 0.000567224
25 16 697374 697358 108.952 105.75 0.00054325 0.00056892
26 16 727817 727801 109.335 110.387 0.000466037 0.000567012
26 15 726201 726186 109.092 112.609 0.000471902 0.000568167
27 16 756089 756073 109.376 110.438 0.00055702 0.000566807
27 16 754382 754366 109.128 110.078 0.000611213 0.000567979
28 15 783055 783040 109.231 105.34 0.000531028 0.000567569
28 16 781407 781391 109.001 105.566 0.000581264 0.000568642
29 16 811661 811645 109.317 111.738 0.000487951 0.000567111
29 16 810303 810287 109.134 112.875 0.000558577 0.000567935
30 5 840753 840748 109.463 113.684 0.000470745 0.000566309
30 6 839448 839442 109.292 113.887 0.000281943 0.000567064
Total time run: 30.0432
Total reads made: 840753
Read size: 4096
Object size: 4096
Bandwidth (MB/sec): 109.316
Average IOPS: 27984
Stddev IOPS: 623.549
Max IOPS: 29122
Min IOPS: 26727
Average Latency(s): 0.000566564
Max latency(s): 0.0436095
Min latency(s): 0.00017236
Total time run: 30.0405
Total reads made: 839448
Read size: 4096
Object size: 4096
Bandwidth (MB/sec): 109.156
Average IOPS: 27943
Stddev IOPS: 659.764
Max IOPS: 29155
Min IOPS: 26648
Average Latency(s): 0.000567352
Max latency(s): 0.0409942
Min latency(s): 0.000162726
[2]+ Done bin/rados bench -p test-pool 30 rand
Performance counter stats for process id '12844':
31293,761946 task-clock (msec) # 0,982 CPUs utilized
127639 context-switches # 0,004 M/sec
0 cpu-migrations # 0,000 K/sec
1 page-faults # 0,000 K/sec
83805740629 cycles # 2,678 GHz
88768702451 instructions # 1,06 insn per cycle
17182701775 branches # 549,078 M/sec
93131963 branch-misses # 0,54% of all branches
31,857793427 seconds time elapsed