rados bench: Rough perf testing of crimson-osd: ceph-osd
4 KB random reads and writes with two rados bench instances over
single OSD instance
deployed on purley server by vstart.sh
with minimal content stored in memstore / cyanstore.
master (commit ID caa58fce7a3c6a16d86ec2bf075a204e1e2b836c)write:
>>> 36,776,283,088. / (55535+55278) = 331876.9737
read:
>>> 72,550,517,369. / (461601+459538) = 78761.7475
write:
>>> 100,788,960,240. / (199517+198868) = 252993.8633
read:
>>>98,664,996,544. / (1158630+1161345) = 42528.4740
write: 0.000535897 0.000538354
read: 0.00103087 0.00103735
write: 0.000147802 0.000148525
read: 0.000411279 0.000410725
crimson Build configuration ./do_cmake.sh -DWITH_SEASTAR=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_FLAGS="-fno-omit-frame-pointer" -DWITH_TESTS=OFF
[build]$ MDS=0 MGR=0 OSD=1 MON=1 ../src/vstart.sh -n --without-dashboard --memstore -X -o "memstore_device_bytes=2516582400" --nolockdep --crimson --nodaemon --redirect-output
[build]$ bin/ceph osd pool create test-pool 128 128 && bin/ceph osd pool set --yes-i-really-mean-it test-pool size 1 && bin/ceph osd pool --yes-i-really-mean-it set test-pool min_size 1
...
crimson 4 KB random write, two rados bench instances [build]$ perf stat -p `pgrep -u ${UID} crimson-osd` & bin/rados bench -p test-pool -b 4096 -t 1 30 write --no-cleanup & bin/rados bench -p test-pool -b 4096 -t 1 30 write --no-cleanup ; sleep 1; killall -INT perf
[1] 39934
[2] 39935
2021-02-02T16:20:00.940-0800 7fd722af7c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T16:20:00.940-0800 7f8862308c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T16:20:00.944-0800 7fd722af7c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T16:20:00.944-0800 7fd722af7c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T16:20:00.944-0800 7f8862308c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T16:20:00.944-0800 7f8862308c00 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
Maintaining 1 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects
Object prefix: benchmark_data_otccldstore05_39937
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 1 1 0 0 0 - 0
hints = 1
Maintaining 1 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects
Object prefix: benchmark_data_otccldstore05_39935
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 1 6858 6857 26.7795 26.7852 0.000135591 0.000143715
1 1 6865 6864 26.811 26.8125 0.000137046 0.000143354
2 1 13311 13310 25.9915 25.207 0.000176132 0.000148048
2 1 13321 13320 26.0134 25.2188 0.000151368 0.000147108
3 1 20020 20019 26.0623 26.207 0.000157719 0.000147513
3 1 20023 20022 26.0679 26.1797 0.000136185 0.000146881
4 0 26489 26489 25.8645 25.2734 0.000145839 0.000148607
4 1 26494 26493 25.8697 25.2773 0.000151335 0.000148186
5 0 33352 33352 26.0529 26.8086 0.000137008 0.000147484
5 1 33359 33358 26.0585 26.8164 0.000139062 0.000147246
6 1 40251 40250 26.201 26.9453 0.00013494 0.000146615
6 0 40258 40258 26.2071 26.9531 0.00013908 0.000146494
7 1 46997 46996 26.2218 26.3516 0.000137849 0.000146479
7 1 47006 47005 26.228 26.3555 0.000128593 0.000146446
8 1 53715 53714 26.2237 26.2422 0.000133854 0.00014646
8 1 53726 53725 26.2304 26.25 0.000134123 0.000146484
9 1 60499 60498 26.2538 26.5 0.000133578 0.00014628
9 1 60507 60506 26.2588 26.4883 0.00014911 0.000146345
10 0 67381 67381 26.3167 26.8867 0.000142781 0.000145918
10 1 67392 67391 26.3219 26.8945 0.000150602 0.000146014
11 0 74263 74263 26.3677 26.8828 0.000151716 0.000145625
11 1 74271 74270 26.3715 26.8711 0.000156115 0.000145759
12 1 81295 81294 26.4587 27.4648 0.000133035 0.000145109
12 1 81291 81290 26.4586 27.4219 0.000138809 0.000145287
13 1 88249 88248 26.5127 27.1641 0.000138263 0.000144805
13 1 88228 88227 26.5074 27.0977 0.000136347 0.000145026
14 1 95164 95163 26.548 27.0117 0.000152804 0.000144605
14 1 95133 95132 26.5403 26.9727 0.000154237 0.000144836
15 1 101961 101960 26.5479 26.5508 0.000166962 0.000144602
15 1 101916 101915 26.5371 26.4961 0.000158134 0.000144849
16 0 108872 108872 26.5758 27 0.000132911 0.000144445
16 0 108809 108809 26.5614 26.9297 0.000133826 0.000144708
17 1 115696 115695 26.58 26.6523 0.000139372 0.000144419
17 1 115607 115606 26.5605 26.5508 0.000141601 0.000144707
18 1 122606 122605 26.6026 26.9922 0.000141074 0.000144292
18 1 122506 122505 26.5818 26.9492 0.000150915 0.000144582
19 0 129526 129526 26.6251 27.0352 0.000134035 0.000144167
19 1 129400 129399 26.5999 26.9297 0.000140062 0.000144478
2021-02-02T16:20:20.954142-0800 min lat: 0.000120798 max lat: 0.00142121 avg lat: 0.000144053
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 1 136449 136448 26.6456 27.0391 0.000136418 0.000144053
2021-02-02T16:20:20.954555-0800 min lat: 0.000126022 max lat: 0.00144533 avg lat: 0.000144376
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 1 136301 136300 26.6176 26.957 0.000142347 0.000144376
21 1 143364 143363 26.6627 27.0117 0.000140698 0.000143957
21 1 143190 143189 26.6313 26.9102 0.000134956 0.000144297
22 1 150026 150025 26.6334 26.0234 0.000145935 0.000144116
22 1 149825 149824 26.5986 25.918 0.000154121 0.000144481
23 0 156025 156025 26.4944 23.4375 0.000145871 0.000144885
23 1 155715 155714 26.4423 23.0078 0.000142902 0.000145357
24 1 161850 161849 26.3382 22.75 0.00016766 0.000145758
24 1 161468 161467 26.2768 22.4727 0.000143757 0.000146293
25 1 167680 167679 26.1954 22.7734 0.000144442 0.000146566
25 1 167182 167181 26.1184 22.3203 0.000176898 0.000147201
26 1 173544 173543 26.0688 22.9062 0.000142073 0.000147289
26 1 173023 173022 25.9912 22.8164 0.000176879 0.000147938
27 1 180082 180081 26.049 25.5391 0.000140195 0.000147399
27 0 179517 179517 25.9681 25.3711 0.000136034 0.000148079
28 0 186650 186650 26.0349 25.6602 0.00017829 0.000147481
28 0 186052 186052 25.9522 25.5273 0.000145767 0.000148177
29 1 193201 193200 26.0193 25.5859 0.00013518 0.000147569
29 1 192558 192557 25.9334 25.4102 0.000136675 0.000148292
Total time run: 30
Total writes made: 199517
Write size: 4096
Object size: 4096
Bandwidth (MB/sec): 25.9788
Stddev Bandwidth: 1.38058
Max bandwidth (MB/sec): 27.4648
Min bandwidth (MB/sec): 22.75
Average IOPS: 6650
Stddev IOPS: 353.43
Max IOPS: 7031
Min IOPS: 5824
Average Latency(s): 0.000147802
Stddev Latency(s): 5.15113e-05
Max latency(s): 0.00143561
Min latency(s): 0.000120798
Total time run: 30.0001
Total writes made: 198868
Write size: 4096
Object size: 4096
Bandwidth (MB/sec): 25.8942
Stddev Bandwidth: 1.46887
Max bandwidth (MB/sec): 27.4219
Min bandwidth (MB/sec): 22.3203
Average IOPS: 6628
Stddev IOPS: 376.03
Max IOPS: 7020
Min IOPS: 5714
Average Latency(s): 0.000148525
Stddev Latency(s): 5.69993e-05
Max latency(s): 0.00152783
Min latency(s): 0.000126022
[2]+ Done bin/rados bench -p test-pool -b 4096 -t 1 30 write --no-cleanup
chunmei@otccldstore05:~/ceph/build$
Performance counter stats for process id '39810':
29352.930488 task-clock (msec) # 0.955 CPUs utilized
119,428 context-switches # 0.004 M/sec
0 cpu-migrations # 0.000 K/sec
70,729 page-faults # 0.002 M/sec
100,788,960,240 cycles # 3.434 GHz
80,528,458,778 instructions # 0.80 insn per cycle
15,610,314,742 branches # 531.815 M/sec
122,922,213 branch-misses # 0.79% of all branches
30.746024477 seconds time elapsed
...
crimson 4 KB random reads, two rados bench instances [build]$ perf stat -p `pgrep -u ${UID} crimson-osd` & bin/rados bench -p test-pool 30 rand & bin/rados bench -p test-pool 30 rand ; sleep 1; killall -INT perf
[1] 39986
[2] 39987
2021-02-02T16:21:37.112-0800 7f24ad7f4c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T16:21:37.112-0800 7f7a11615c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T16:21:37.116-0800 7f24ad7f4c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T16:21:37.116-0800 7f24ad7f4c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T16:21:37.116-0800 7f7a11615c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T16:21:37.116-0800 7f7a11615c00 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 38783 38767 151.404 151.434 0.000434829 0.000409514
1 16 39069 39053 152.519 152.551 0.000488869 0.000406507
2 15 77478 77463 151.271 151.156 0.000473835 0.000409977
2 15 78034 78019 152.354 152.211 0.00041396 0.000407137
3 16 116253 116237 151.329 151.461 0.000381632 0.000409875
3 16 116921 116905 152.196 151.898 0.000780833 0.000407598
4 15 155126 155111 151.457 151.852 0.000494543 0.000409532
4 16 155745 155729 152.058 151.656 0.000464185 0.00040798
5 16 194378 194362 151.828 153.324 0.000416376 0.000408546
5 16 195114 195098 152.401 153.785 0.000408675 0.000407064
6 16 233589 233573 152.047 153.168 0.000386339 0.000407959
6 16 234559 234543 152.677 154.082 0.000451645 0.000406331
7 16 272933 272917 152.279 153.688 0.000330336 0.000407338
7 15 274167 274152 152.965 154.723 0.000484009 0.000405582
8 15 312435 312420 152.53 154.309 0.000394622 0.000406677
8 15 313799 313784 153.193 154.812 0.00034397 0.000404979
9 16 351698 351682 152.621 153.367 0.000464582 0.000406439
9 15 353250 353235 153.291 154.105 0.000412464 0.000404724
10 16 390947 390931 152.688 153.316 0.000441945 0.000406261
10 15 392519 392504 153.298 153.395 0.000407073 0.000404705
11 15 430239 430224 152.759 153.488 0.00043792 0.000406074
11 15 431782 431767 153.302 153.371 0.000368664 0.000404697
12 15 469594 469579 152.838 153.73 0.000475848 0.000405862
12 15 471197 471182 153.355 153.965 0.00033606 0.000404558
13 15 508808 508793 152.862 153.18 0.000317192 0.000405805
13 16 510557 510541 153.383 153.746 0.000474849 0.000404486
14 15 547939 547924 152.86 152.855 0.000428351 0.00040581
14 15 549904 549889 153.404 153.703 0.000381534 0.000404433
15 15 587085 587070 152.862 152.914 0.00042661 0.000405808
15 15 589108 589093 153.385 153.141 0.000394445 0.000404488
16 15 626248 626233 152.867 152.98 0.000337947 0.000405794
16 16 628355 628339 153.378 153.305 0.000414045 0.000404505
17 16 665498 665482 152.892 153.316 0.000322718 0.000405729
17 16 667653 667637 153.384 153.508 0.000412333 0.000404493
18 15 704493 704478 152.86 152.328 0.000477544 0.000405817
18 15 706989 706974 153.398 153.66 0.000416076 0.000404459
19 16 743468 743452 152.826 152.242 0.000363097 0.000405907
19 16 746160 746144 153.376 153.008 0.00046651 0.000404515
2021-02-02T16:21:57.127925-0800 min lat: 0.000178904 max lat: 0.00141762 avg lat: 0.000405993
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 15 782439 782424 152.795 152.234 0.000418111 0.000405993
2021-02-02T16:21:57.128530-0800 min lat: 0.000179936 max lat: 0.00162397 avg lat: 0.000404667
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 785144 785128 153.32 152.281 0.000378989 0.000404667
21 15 821587 821572 152.799 152.922 0.000317974 0.000405982
21 16 824177 824161 153.279 152.473 0.000400007 0.000404779
22 15 860617 860602 152.783 152.461 0.000345059 0.000406027
22 16 863328 863312 153.262 152.934 0.000512376 0.000404824
23 16 899573 899557 152.755 152.168 0.000413342 0.000406104
23 16 902328 902312 153.221 152.344 0.000446102 0.000404933
24 16 938635 938619 152.747 152.586 0.000334008 0.000406124
24 15 941371 941356 153.191 152.516 0.000380415 0.000405016
25 16 977584 977568 152.722 152.145 0.000441015 0.000406193
25 15 980539 980524 153.183 153 0.000411608 0.000405039
26 15 1016610 1016595 152.71 152.449 0.000448545 0.000406224
26 16 1019542 1019526 153.15 152.352 0.00043459 0.000405126
27 15 1055690 1055675 152.707 152.656 0.00044772 0.000406234
27 16 1058812 1058796 153.158 153.398 0.000388131 0.000405104
28 15 1090523 1090508 152.112 136.066 0.000487566 0.000407832
28 16 1091979 1091963 152.315 129.559 0.000764186 0.000407344
29 15 1124410 1124395 151.431 132.371 0.000469259 0.000409682
29 15 1122944 1122929 151.233 120.961 0.000507124 0.000410267
30 12 1161345 1161333 151.192 144.289 0.000318963 0.000410271
Total time run: 30.0002
Total reads made: 1158630
Read size: 4096
Object size: 4096
Bandwidth (MB/sec): 150.863
Average IOPS: 38620
Stddev IOPS: 1877.53
Max IOPS: 39632
Min IOPS: 30966
Average Latency(s): 0.000411279
Max latency(s): 0.00162397
Min latency(s): 0.000179936
Total time run: 30.0443
Total reads made: 1161345
Read size: 4096
Object size: 4096
Bandwidth (MB/sec): 150.994
Average IOPS: 38654
Stddev IOPS: 1266.31
Max IOPS: 39503
Min IOPS: 33887
Average Latency(s): 0.000410725
Max latency(s): 0.0444151
Min latency(s): 0.000178904
[2]+ Done bin/rados bench -p test-pool 30 rand
chunmei@otccldstore05:~/ceph/build$
Performance counter stats for process id '39810':
29346.590569 task-clock (msec) # 0.955 CPUs utilized
119,145 context-switches # 0.004 M/sec
0 cpu-migrations # 0.000 K/sec
495 page-faults # 0.017 K/sec
98,664,996,544 cycles # 3.362 GHz
125,469,617,643 instructions # 1.27 insn per cycle
24,283,227,140 branches # 827.463 M/sec
114,369,506 branch-misses # 0.47% of all branches
30.727531284 seconds time elapsed
...
./do_cmake.sh -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_FLAGS="-fno-omit-frame-pointer" -DWITH_TESTS=OFF
build$ MDS=0 MGR=0 OSD=1 MON=1 ../src/vstart.sh -n --without-dashboard --memstore -X -o "memstore_device_bytes=2516582400" --nolockdep --nodaemon --redirect-output
build$ bin/ceph osd pool create test-pool 128 128 && bin/ceph osd pool set --yes-i-really-mean-it test-pool size 1 && bin/ceph osd pool --yes-i-really-mean-it set test-pool min_size 1
...
ceph 4 KB random writes, two rados bench instances [build]$ perf stat -p `pgrep -u ${UID} ceph-osd` & bin/rados bench -p test-pool -b 4096 -t 1 30 write --no-cleanup & bin/rados bench -p test-pool -b 4096 -t 1 30 write --no-cleanup ; sleep 1; killall -INT perf
[1] 47836
[2] 47837
2021-02-02T15:50:36.032-0800 7f1905697c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T15:50:36.032-0800 7f09804a5c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T15:50:36.036-0800 7f1905697c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T15:50:36.036-0800 7f1905697c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T15:50:36.036-0800 7f09804a5c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T15:50:36.036-0800 7f09804a5c00 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
Maintaining 1 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects
Object prefix: benchmark_data_otccldstore05_47837
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
hints = 1
Maintaining 1 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects
Object prefix: benchmark_data_otccldstore05_47839
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 1 2317 2316 9.04514 9.04688 0.000573643 0.00042754
1 1 2378 2377 9.28461 9.28516 0.000502196 0.000417849
2 1 4376 4375 8.54362 8.04297 0.000489525 0.000453206
2 1 4497 4496 8.7805 8.27734 0.000438683 0.000442029
3 1 6329 6328 8.23849 7.62891 0.000512919 0.000470018
3 1 6467 6466 8.41837 7.69531 0.000465999 0.000461176
4 0 8199 8199 8.00585 7.30859 0.000481134 0.00048364
4 1 8317 8316 8.12008 7.22656 0.000448702 0.000477721
5 1 10220 10219 7.98268 7.89062 0.000502096 0.000485106
5 1 10235 10234 7.99424 7.49219 0.000417071 0.000485066
6 1 12117 12116 7.88713 7.41016 0.000407215 0.000491093
6 1 11977 11976 7.79576 6.80469 0.000680963 0.000497429
7 1 14008 14007 7.81551 7.38672 0.00045931 0.000495654
7 1 13729 13728 7.65956 6.84375 0.000658635 0.000506305
8 1 15904 15903 7.7642 7.40625 0.000419634 0.000498997
8 1 15484 15483 7.5589 6.85547 0.000454074 0.00051294
9 1 17691 17690 7.67698 6.98047 0.000485083 0.000504706
9 1 17227 17226 7.4754 6.80859 0.000489152 0.000518575
10 1 19473 19472 7.60529 6.96094 0.000651021 0.000509473
10 1 19004 19003 7.42188 6.94141 0.000463674 0.000522282
11 1 21307 21306 7.56507 7.16406 0.000627841 0.000512209
11 1 20790 20789 7.38129 6.97656 0.0004806 0.000525177
12 1 23142 23141 7.5319 7.16797 0.00044218 0.00051447
12 1 22553 22552 7.33998 6.88672 0.000454319 0.000528031
13 1 24912 24911 7.48427 6.91406 0.000507445 0.00051771
13 1 24368 24367 7.32065 7.08984 0.000429137 0.000529325
14 1 26713 26712 7.45211 7.03516 0.000698279 0.000520017
14 1 26199 26198 7.30853 7.15234 0.000565426 0.000530172
15 1 28496 28495 7.41954 6.96484 0.000613316 0.000522365
15 1 28009 28008 7.29257 7.07031 0.000531106 0.000531333
16 1 30334 30333 7.4045 7.17969 0.00055885 0.000523436
16 1 29859 29858 7.28836 7.22656 0.000608489 0.000531552
17 1 32196 32195 7.39671 7.27344 0.000726058 0.000523989
17 1 31680 31679 7.27799 7.11328 0.000496459 0.000532327
18 0 34024 34024 7.38265 7.14453 0.000700221 0.000525029
18 1 33518 33517 7.27245 7.17969 0.000635252 0.000532728
19 1 35920 35919 7.38361 7.40234 0.000870028 0.000525007
19 1 35374 35373 7.2712 7.25 0.000672184 0.000532826
2021-02-02T15:50:56.045271-0800 min lat: 0.000262273 max lat: 0.0106407 avg lat: 0.000525623
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 1 37764 37763 7.37452 7.20312 0.000805922 0.000525623
2021-02-02T15:50:56.049228-0800 min lat: 0.000256481 max lat: 0.0102007 avg lat: 0.000532913
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 1 37228 37227 7.26968 7.24219 0.000430906 0.000532913
21 1 39578 39577 7.36069 7.08594 0.000495532 0.000526555
21 1 39072 39071 7.26643 7.20312 0.000458494 0.000533125
22 1 41397 41396 7.34904 7.10547 0.000502652 0.000527396
22 1 40936 40935 7.26705 7.28125 0.000515167 0.000533111
23 1 43194 43193 7.33466 7.01953 0.000613976 0.000528452
23 1 42735 42734 7.25657 7.02734 0.000469327 0.000533907
24 1 44942 44941 7.31351 6.82812 0.000466191 0.000529992
24 1 44506 44505 7.24242 6.91797 0.000625847 0.000534969
25 1 46690 46689 7.29405 6.82812 0.000597494 0.0005314
25 1 46276 46275 7.22924 6.91406 0.000462946 0.000535955
26 1 48528 48527 7.2896 7.17969 0.000477062 0.000531724
26 1 48116 48115 7.22759 7.1875 0.000418401 0.000536081
27 0 50371 50371 7.28635 7.20312 0.000481265 0.000531956
27 1 49998 49997 7.23213 7.35156 0.000430613 0.000535756
28 1 52145 52144 7.27343 6.92578 0.000729977 0.000532856
28 1 51785 51784 7.2231 6.98047 0.000617918 0.000536408
29 0 53814 53814 7.24753 6.52344 0.000693434 0.000534719
29 1 53491 53490 7.20378 6.66406 0.000434546 0.00053784
Total time run: 30.0001
Total writes made: 55535
Write size: 4096
Object size: 4096
Bandwidth (MB/sec): 7.23109
Stddev Bandwidth: 0.464195
Max bandwidth (MB/sec): 9.04688
Min bandwidth (MB/sec): 6.52344
Average IOPS: 1851
Stddev IOPS: 118.834
Max IOPS: 2316
Min IOPS: 1670
Average Latency(s): 0.000535897
Stddev Latency(s): 0.0001355
Max latency(s): 0.0106407
Min latency(s): 0.000262273
Total time run: 30.0003
Total writes made: 55278
Write size: 4096
Object size: 4096
Bandwidth (MB/sec): 7.19758
Stddev Bandwidth: 0.507433
Max bandwidth (MB/sec): 9.28516
Min bandwidth (MB/sec): 6.66406
Average IOPS: 1842
Stddev IOPS: 129.903
Max IOPS: 2377
Min IOPS: 1706
Average Latency(s): 0.000538354
Stddev Latency(s): 0.000147474
Max latency(s): 0.0102007
Min latency(s): 0.000256481
[2]+ Done bin/rados bench -p test-pool -b 4096 -t 1 30 write --no-cleanup
chunmei@otccldstore05:~/ceph/build$
Performance counter stats for process id '47668':
38310.684494 task-clock (msec) # 1.247 CPUs utilized
447,958 context-switches # 0.012 M/sec
64 cpu-migrations # 0.002 K/sec
635,907 page-faults # 0.017 M/sec
36,776,283,088 cycles # 0.960 GHz
21,950,341,932 instructions # 0.60 insn per cycle
4,190,360,795 branches # 109.378 M/sec
192,770,531 branch-misses # 4.60% of all branches
30.716882104 seconds time elapsed
...
ceph 4 KB random reads, two rados bench instances [build]$ perf stat -p `pgrep -u ${UID} ceph-osd` & bin/rados bench -p test-pool 30 rand & bin/rados bench -p test-pool 30 rand ; sleep 1; killall -INT perf
[1] 47888
[2] 47889
2021-02-02T15:53:00.668-0800 7fabf3a48c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T15:53:00.668-0800 7f857a572c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T15:53:00.672-0800 7fabf3a48c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T15:53:00.672-0800 7fabf3a48c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T15:53:00.672-0800 7f857a572c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T15:53:00.672-0800 7f857a572c00 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 5 5 0 0 0 - 0
1 15 14120 14105 55.0829 55.0977 0.00124998 0.0011246
1 16 14202 14186 55.3997 55.4141 0.00106816 0.0011191
2 15 28663 28648 55.9407 56.8086 0.00108013 0.00110791
2 16 28675 28659 55.9614 56.5352 0.00101631 0.00110838
3 16 43631 43615 56.7783 58.4648 0.00105792 0.00109206
3 15 43165 43150 56.1714 56.6055 0.00127528 0.00110469
4 16 59328 59312 57.9103 61.3164 0.00120609 0.00107102
4 16 57783 57767 56.4006 57.0977 0.00115594 0.00110046
5 15 75002 74987 58.5723 61.2305 0.00102998 0.00105916
5 16 71847 71831 56.1061 54.9375 0.00108092 0.00110622
6 15 89678 89663 58.3634 57.3281 0.00105864 0.00106304
6 16 85463 85447 55.6183 53.1875 0.00134099 0.00111605
7 16 105084 105068 58.6209 60.1758 0.000497663 0.00105843
7 16 100246 100230 55.9209 57.7461 0.00104362 0.00111004
8 16 121625 121609 59.3686 64.6133 0.00101204 0.00104512
8 16 116767 116751 56.9962 64.5352 0.000916468 0.0010889
9 16 136995 136979 59.4418 60.0391 0.00111779 0.00104387
9 15 132766 132751 57.6067 62.5 0.000920086 0.00107698
10 16 150999 150983 58.967 54.7031 0.00100336 0.00105229
10 16 147741 147725 57.6946 58.4922 0.00102699 0.00107505
11 16 166909 166893 59.2553 62.1484 0.0010835 0.00104723
11 16 163712 163696 58.1206 62.3867 0.000991223 0.00106691
12 16 182773 182757 59.4805 61.9688 0.00129142 0.00104328
12 16 179865 179849 58.5348 63.0977 0.000917637 0.0010592
13 16 199056 199040 59.797 63.6055 0.00113963 0.00103781
13 15 196006 195991 58.8817 63.0547 0.000937929 0.00105287
14 15 213739 213724 59.6222 57.3594 0.00098426 0.00104082
14 16 211250 211234 58.9284 59.543 0.000970763 0.00105199
15 16 227536 227520 59.2406 63.6172 0.000963362 0.00104641
15 15 229777 229762 59.8232 62.6484 0.000911851 0.00103735
16 15 243941 243926 59.5431 64.0859 0.0010081 0.00104109
16 15 245719 245704 59.9757 62.2734 0.000909791 0.00103474
17 16 260048 260032 59.741 62.9141 0.000985296 0.00103762
17 16 261760 261744 60.1326 62.6562 0.000969663 0.0010321
18 16 274132 274116 59.478 55.0156 0.00110702 0.00104217
18 16 276558 276542 60.0028 57.8047 0.000976702 0.00103432
19 16 289664 289648 59.5403 60.6719 0.00100672 0.00104106
19 16 292812 292796 60.1859 63.4922 0.0010654 0.0010312
2021-02-02T15:53:20.684927-0800 min lat: 0.000237794 max lat: 0.00621949 avg lat: 0.001038
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 305800 305784 59.7144 63.0312 0.00103086 0.001038
2021-02-02T15:53:20.685197-0800 min lat: 0.000233678 max lat: 0.00521386 avg lat: 0.00102867
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 308988 308972 60.3354 63.1875 0.000986608 0.00102867
21 16 321784 321768 59.8433 62.4375 0.00111894 0.00103575
21 16 324978 324962 60.436 62.4609 0.00101579 0.00102696
22 15 336834 336819 59.7951 58.793 0.000899899 0.00103655
22 16 339681 339665 60.299 57.4336 0.00102852 0.00102932
23 16 352308 352292 59.8228 60.4414 0.00103828 0.001036
23 15 354618 354603 60.2139 58.3516 0.00143221 0.00103081
24 16 368041 368025 59.8904 61.457 0.000991831 0.00103478
24 15 370398 370383 60.2729 61.6406 0.000877069 0.00102979
25 15 384146 384131 60.0108 62.9141 0.00090133 0.00103265
25 16 386088 386072 60.313 61.2852 0.00114774 0.00102913
26 16 399984 399968 60.0817 61.8633 0.00104868 0.00103139
26 15 399825 399810 60.0569 53.6641 0.00137533 0.00103351
27 16 415168 415152 60.0528 59.3125 0.000977912 0.00103183
27 16 414838 414822 60.0041 58.6406 0.0010264 0.00103441
28 16 430983 430967 60.1141 61.7773 0.00102559 0.00103075
28 15 430080 430065 59.9873 59.543 0.000951289 0.0010347
29 15 447354 447339 60.2461 63.9531 0.000930648 0.00102845
29 16 445825 445809 60.0391 61.5 0.00106371 0.00103382
Total time run: 30.0007
Total reads made: 461601
Read size: 4096
Object size: 4096
Bandwidth (MB/sec): 60.1029
Average IOPS: 15386
Stddev IOPS: 832.099
Max IOPS: 16521
Min IOPS: 13616
Average Latency(s): 0.00103087
Max latency(s): 0.00621949
Min latency(s): 0.000237794
Total time run: 30.0006
Total reads made: 459538
Read size: 4096
Object size: 4096
Bandwidth (MB/sec): 59.8344
Average IOPS: 15317
Stddev IOPS: 742.104
Max IOPS: 16541
Min IOPS: 13738
Average Latency(s): 0.00103735
Max latency(s): 0.00569449
Min latency(s): 0.000233678
[2]+ Done bin/rados bench -p test-pool 30 rand
chunmei@otccldstore05:~/ceph/build$
Performance counter stats for process id '47668':
73196.398540 task-clock (msec) # 2.381 CPUs utilized
453,224 context-switches # 0.006 M/sec
71 cpu-migrations # 0.001 K/sec
444,128 page-faults # 0.006 M/sec
72,550,517,369 cycles # 0.991 GHz
61,102,232,881 instructions # 0.84 insn per cycle
11,803,742,743 branches # 161.261 M/sec
171,681,752 branch-misses # 1.45% of all branches
30.736441254 seconds time elapsed
...
