Skip to content

Instantly share code, notes, and snippets.

@liu-chunmei

liu-chunmei/gist:4fd88fd0ff56d6849439a2df329aa80e

Last active February 10, 2021 06:04
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save liu-chunmei/4fd88fd0ff56d6849439a2df329aa80e to your computer and use it in GitHub Desktop.
Save liu-chunmei/4fd88fd0ff56d6849439a2df329aa80e to your computer and use it in GitHub Desktop.
Download ZIP
crimson-osd performance testing : rados bench (ceph osd bind to 1 cpu)
Raw
gistfile3.md

rados bench: Rough perf testing of crimson-osd: ceph-osd

Scenario

  • 4 KB random reads and writes with two rados bench instances over
  • single OSD instance
  • deployed on purley server by vstart.sh
  • with minimal content stored in memstore / cyanstore.

version

master (commit ID caa58fce7a3c6a16d86ec2bf075a204e1e2b836c)

The cycles-per-op metric

ceph-osd

write:
>>> 143,135,708,110 / (244201+244425) = 292935.1039  
read: 
>>> 205,183,497,074 / (1168374+1168101) = 87817.5444

crimson-osd

write: 99,117,715,396 / (282236+301107) = 169913.2678
>>> 
read: 
>>> 99,704,254,646 /(1143911+1141542) = 43625.5983

The avg-lat metric

ceph-osd

write:  0.000243567  0.000243454
read: 0.000407572    0.00040759

crimson-osd

write:  0.000210333 0.000197312
read:  0.000416625 0.000417968

crimson Build configuration

./do_cmake.sh -DWITH_SEASTAR=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_FLAGS="-fno-omit-frame-pointer" -DWITH_TESTS=OFF

crimson Deployment

[build]$ MDS=0 MGR=0 OSD=1 MON=1 ../src/vstart.sh -n --without-dashboard --memstore -X -o "memstore_device_bytes=2516582400" --nolockdep --crimson --nodaemon --redirect-output            

[build]$ bin/ceph osd pool create test-pool 128 128 && bin/ceph osd pool set --yes-i-really-mean-it test-pool size 1 && bin/ceph osd pool --yes-i-really-mean-it set test-pool min_size 1 
...

crimson 4 KB random write, two rados bench instances

[build]$ perf stat -p `pgrep -u ${UID} crimson-osd` & bin/rados bench -p test-pool -b 4096 -t 2 30 write  --no-cleanup & bin/rados bench -p test-pool -b 4096 -t 2 30 write  --no-cleanup ; sleep 1; killall -INT perf
[1] 35640
[2] 35641
2021-02-02T23:19:28.535-0800 7fb558e2bc00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:19:28.535-0800 7fe7f39d2c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:19:28.535-0800 7fb558e2bc00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:19:28.535-0800 7fb558e2bc00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:19:28.539-0800 7fe7f39d2c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:19:28.539-0800 7fe7f39d2c00 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
Maintaining 2 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects
Object prefix: benchmark_data_otccldstore05_35641
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
hints = 1
Maintaining 2 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects
Object prefix: benchmark_data_otccldstore05_35643
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1       2     10722     10720   41.8723    41.875 0.000172965 0.000184479
    1       2     11949     11947   46.6588    46.668 0.000147128 0.000165361
    2       1     21020     21019   41.0486   40.2305 0.000167733 0.000188207
    2       2     23418     23416   45.7261   44.8008 0.000157021 0.000168783
    3       2     32175     32173   41.8866   43.5703 0.000166672 0.000184352
    3       2     35445     35443   46.1416   46.9805 0.000186083 0.000167242
    4       2     42764     42762    41.754   41.3633 0.000166139 0.000184918
    4       1     46801     46800   45.6961   44.3633 0.000154328 0.000168895
    5       2     53752     53750   41.9861   42.9219 0.000163787 0.000183863
    5       2     58389     58387   45.6079   45.2617  0.00015325 0.000169221
    6       2     64880     64878    42.232   43.4688 0.000172084 0.000182767
    6       2     70294     70292   45.7559   46.5039 0.000159273 0.000168655
    7       2     75684     75682   42.2268   42.2031 0.000164277 0.000182779
    7       1     82136     82135    45.827   46.2617 0.000149423 0.000168403
    8       2     86618     86616   42.2863   42.7109 0.000172922  0.00018251
    8       2     94124     94122   45.9506   46.8242 0.000153396 0.000167942
    9       2     97351     97349   42.2453   41.9258 0.000279716 0.000182684
    9       2    105784    105782   45.9047   45.5469 0.000186075 0.000168111
   10       2    105801    105799    41.321   33.0078 0.000532779 0.000186818
   10       2    114730    114728   44.8081   34.9453  0.00021164 0.000172274
   11       2    114136    114134   40.5238   32.5586 0.000577008 0.000190542
   11       2    123484    123482   43.8428   34.1953 0.000197904 0.000176112
   12       2    123983    123981   40.3517   38.4648 0.000160289 0.000191362
   12       2    133829    133827   43.5561   40.4102 0.000150877 0.000177284
   13       2    134954    134952   40.5437   42.8555 0.000165407 0.000190441
   13       2    145598    145596   43.7413   45.9727 0.000491612 0.000176522
   14       1    145450    145449    40.576   41.0039 0.000182196 0.000190285
   14       2    156455    156453   43.6456   42.4102 0.000156846 0.000176915
   15       2    154673    154671    40.272   36.0234 0.000204763 0.000191735
   15       1    166273    166272   43.2924   38.3555 0.000180707 0.000178375
   16       2    163433    163431   39.8933   34.2188 0.000196281 0.000193579
   16       2    175512    175510   42.8415   36.0859 0.000177219 0.000180273
   17       2    172475    172473   39.6239   35.3203 0.000198137 0.000194906
   17       2    185107    185105   42.5257   37.4805 0.000175174 0.000181627
   18       2    181229    181227    39.322   34.1953 0.000211465 0.000196425
   18       2    194417    194415   42.1832   36.3672 0.000197436 0.000183116
   19       1    189846    189845   39.0239   33.6641 0.000218819 0.000197943
   19       2    203578    203576   41.8462   35.7852  0.00020033 0.000184607
2021-02-02T23:19:48.548477-0800 min lat: 0.000113862 max lat: 0.00163842 avg lat: 0.000199377
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20       2    198417    198415   38.7462   33.4766 0.000187681 0.000199377
2021-02-02T23:19:48.549936-0800 min lat: 0.000104047 max lat: 0.00207444 avg lat: 0.000186102
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20       2    212588    212586   41.5133   35.1953 0.000216091 0.000186102
   21       1    206906    206905     38.48   33.1641 0.000182585 0.000200773
   21       1    221526    221525   41.1988    34.918  0.00017214 0.000187537
   22       2    215471    215469   38.2511   33.4531 0.000228251 0.000201987
   22       2    230735    230733   40.9607   35.9688  0.00018928 0.000188639
   23       2    223992    223990   38.0349   33.2852 0.000216848 0.000203147
   23       2    239755    239753   40.7115   35.2344  0.00018359 0.000189808
   24       1    232192    232191   37.7847   32.0352 0.000185913 0.000204508
   24       2    248314    248312    40.408   33.4336 0.000194988 0.000191248
   25       2    240603    240601   37.5872   32.8516 0.000209535 0.000205594
   25       1    257096    257095   40.1638   34.3086 0.000555264 0.000192421
   26       2    249140    249138   37.4239   33.3477 0.000231503 0.000206497
   26       2    266108    266106   39.9726   35.1992 0.000810707 0.000193352
   27       2    257526    257524   37.2508   32.7578 0.000211358 0.000207469
   27       2    274825    274823    39.753   34.0508 0.000187724 0.000194431
   28       2    265737    265735   37.0658   32.0742 0.000210402 0.000208517
   28       2    283512    283510    39.545   33.9336 0.000203711 0.000195464
   29       2    273944    273942   36.8929   32.0586 0.000212178 0.000209505
   29       2    292245    292243   39.3575   34.1133 0.000193788 0.000196405
Total time run:         30
Total writes made:      282236
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     36.7494
Stddev Bandwidth:       4.43604
Max bandwidth (MB/sec): 43.5703
Min bandwidth (MB/sec): 32.0352
Average IOPS:           9407
Stddev IOPS:            1135.63
Max IOPS:               11154
Min IOPS:               8201
Average Latency(s):     0.000210333
Stddev Latency(s):      0.000103254
Max latency(s):         0.00195279
Min latency(s):         0.000113862
   30       1    301107    301106   39.1994   34.6211 0.000189335 0.000197171
Total time run:         30.0425
Total writes made:      301107
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     39.1511
Stddev Bandwidth:       5.19071
Max bandwidth (MB/sec): 46.9805
Min bandwidth (MB/sec): 33.4336
Average IOPS:           10022
Stddev IOPS:            1328.82
Max IOPS:               12027
Min IOPS:               8559
Average Latency(s):     0.000197312
Stddev Latency(s):      0.000129292
Max latency(s):         0.0426281
Min latency(s):         0.000104047
[2]+  Done                    bin/rados bench -p test-pool -b 4096 -t 2 30 write --no-cleanup

 Performance counter stats for process id '35547':

      29375.155704      task-clock (msec)         #    0.926 CPUs utilized
           119,202      context-switches          #    0.004 M/sec
                 0      cpu-migrations            #    0.000 K/sec
           180,856      page-faults               #    0.006 M/sec
    99,117,715,396      cycles                    #    3.374 GHz
    92,457,474,184      instructions              #    0.93  insn per cycle
    18,215,928,043      branches                  #  620.113 M/sec
       121,770,620      branch-misses             #    0.67% of all branches

      31.735040725 seconds time elapsed
...

crimson 4 KB random reads, two rados bench instances

[build]$ perf stat -p `pgrep -u ${UID} crimson-osd` & bin/rados bench -p test-pool 30 rand & bin/rados bench -p test-pool 30 rand ; sleep 1; killall -INT perf 
[1] 35691
[2] 35692
2021-02-02T23:22:05.051-0800 7f3b4ee12c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:22:05.051-0800 7f8c242c4c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:22:05.055-0800 7f3b4ee12c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:22:05.055-0800 7f3b4ee12c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:22:05.059-0800 7f8c242c4c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:22:05.059-0800 7f8c242c4c00 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       1         1         0         0         0           -           0
    1      15     36683     36668   143.209   143.234 0.000348953 0.000433213
    1      16     36577     36561   142.777   142.816 0.000441971 0.000434593
    2      16     74407     74391    145.27   147.355 0.000489552 0.000427119
    2      16     74225     74209   144.909   147.062 0.000298399 0.000428308
    3      16    111972    111956   145.751   146.738 0.000441017 0.000425789
    3      15    111605    111590   145.274    146.02 0.000383916 0.000427302
    4      16    150423    150407   146.857   150.199 0.000378026 0.000422598
    4      16    149919    149903   146.367    149.66 0.000429449 0.000424115
    5      15    188924    188909    147.56   150.398 0.000331789 0.000420572
    5      16    188286    188270   147.065   149.871 0.000389396 0.000422103
    6      16    227213    227197    147.89   149.562 0.000383662  0.00041963
    6      15    226532    226517    147.45   149.402  0.00043966 0.000421004
    7      15    265472    265457    148.11   149.453 0.000420781 0.000418996
    7      15    264768    264753   147.719   149.359 0.000320962 0.000420251
    8      15    303698    303683   148.258    149.32 0.000424099 0.000418584
    8      16    302933    302917   147.886   149.078  0.00024179 0.000419761
    9      16    341964    341948    148.39   149.473 0.000460273 0.000418204
    9      16    340942    340926   147.948   148.473 0.000449695 0.000419602
   10      15    380173    380158   148.475   149.258 0.000419906 0.000417968
   10      16    378994    378978   148.015   148.641 0.000314514  0.00041942
   11      15    418314    418299   148.519   148.988 0.000383033 0.000417844
   11      16    417028    417012   148.063    148.57 0.000404144 0.000419285
   12      15    456334    456319   148.517   148.516 0.000436463  0.00041785
   12      15    454941    454926   148.064   148.102 0.000413468 0.000419284
   13      16    494336    494320   148.509   148.441 0.000417157 0.000417856
   13      15    492875    492860   148.071    148.18 0.000408611  0.00041927
   14      16    532850    532834   148.646   150.445 0.000424519 0.000417483
   14      16    531079    531063   148.152    149.23 0.000425438 0.000419037
   15      15    571106    571091   148.697   149.441 0.000394399 0.000417337
   15      15    569314    569299    148.23   149.359 0.000491081 0.000418812
   16      15    609429    609414   148.758   149.699 0.000422966 0.000417164
   16      16    607680    607664    148.33   149.863 0.000418088 0.000418531
   17      16    647804    647788   148.824   149.898 0.000471822 0.000416979
   17      16    645955    645939   148.398   149.512 0.000389351 0.000418336
   18      16    686109    686093   148.867   149.629 0.000478512 0.000416852
   18      15    684024    684009   148.414   148.711 0.000485036 0.000418291
   19      15    724296    724281   148.882   149.172  0.00044907  0.00041681
   19      15    722225    722210   148.455   149.223 0.000838097 0.000418178
2021-02-02T23:22:25.068368-0800 min lat: 0.000185996 max lat: 0.00190639 avg lat: 0.000416842
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      15    762361    762346   148.871   148.691 0.000451543 0.000416842
2021-02-02T23:22:25.073948-0800 min lat: 0.000183385 max lat: 0.0020365 avg lat: 0.000418175
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      15    760238    760223   148.456   148.488 0.000423487 0.000418175
   21      16    800506    800490   148.876       149 0.000429141 0.000416824
   21      15    798376    798361   148.479   148.977 0.000438838  0.00041811
   22      15    838708    838693   148.892    149.23 0.000479304 0.000416779
   22      16    836636    836620   148.522   149.449 0.000413635 0.000417989
   23      15    876929    876914   148.908   149.301  0.00036257 0.000416733
   23      15    874816    874801   148.548   149.145 0.000724564 0.000417916
   24      16    915112    915096   148.917   149.148 0.000463212 0.000416707
   24      15    912980    912965   148.569   149.078 0.000485282 0.000417859
   25      15    953309    953294   148.928   149.211 0.000363178 0.000416676
   25      16    951126    951110   148.585   149.004 0.000384527 0.000417813
   26      15    991399    991384   148.922   148.789  0.00060119  0.00041669
   26      15    989224    989209   148.593   148.824 0.000436352 0.000417792
   27      15   1029570   1029555   148.928   149.105 0.000377319 0.000416675
   27      15   1027371   1027356   148.608   149.012 0.000439245 0.000417753
   28      15   1067700   1067685   148.928   148.945 0.000415469 0.000416674
   28      16   1065387   1065371   148.603   148.496 0.000389348 0.000417766
   29      16   1105831   1105815   148.928   148.945 0.000332489 0.000416675
   29      15   1103435   1103420   148.603   148.629 0.000470944 0.000417766
Total time run:       30.0002
Total reads made:     1143911
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   148.946
Average IOPS:         38130
Stddev IOPS:          342.847
Max IOPS:             38514
Min IOPS:             36668
Average Latency(s):   0.000416625
Max latency(s):       0.00190639
Min latency(s):       0.000185996
   30       8   1141542   1141534   148.611   148.883 0.000301912 0.000417672
Total time run:       30.0426
Total reads made:     1141542
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   148.427
Average IOPS:         37997
Stddev IOPS:          345.232
Max IOPS:             38367
Min IOPS:             36561
Average Latency(s):   0.000417968
Max latency(s):       0.042654
Min latency(s):       0.000183385
[2]+  Done                    bin/rados bench -p test-pool 30 rand
chunmei@otccldstore05:~/ceph/build$
 Performance counter stats for process id '35547':

      29355.680114      task-clock (msec)         #    0.955 CPUs utilized
           119,286      context-switches          #    0.004 M/sec
                 0      cpu-migrations            #    0.000 K/sec
             3,047      page-faults               #    0.104 K/sec
    99,704,254,646      cycles                    #    3.396 GHz
   123,024,458,709      instructions              #    1.23  insn per cycle
    23,819,502,162      branches                  #  811.410 M/sec
       113,496,463      branch-misses             #    0.48% of all branches

      30.725163741 seconds time elapsed
...

ceph options configuration

ms_async_op_threads = 1

osd_op_num_threads_per_shard = 1

osd_op_num_shards =1

ceph Build configuration

./do_cmake.sh -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_FLAGS="-fno-omit-frame-pointer" -DWITH_TESTS=OFF

ceph Deployment

build$ MDS=0 MGR=0 OSD=1 MON=1 ../src/vstart.sh -n --without-dashboard --memstore -X -o "memstore_device_bytes=2516582400" --nolockdep --nodaemon --redirect-output

build$ bin/ceph osd pool create test-pool 128 128 && bin/ceph osd pool set --yes-i-really-mean-it test-pool size 1 && bin/ceph osd pool --yes-i-really-mean-it set test-pool min_size 1
build$ taskset -cp 0 <ceph_osd Pid> 
...

ceph 4 KB random writes, two rados bench instances

[build]$ perf stat -p `pgrep -u ${UID} ceph-osd` & bin/rados bench -p test-pool -b 4096 -t 2 30 write  --no-cleanup & bin/rados bench -p test-pool -b 4096 -t 2 30 write  --no-cleanup ; sleep 1; killall -INT perf
[1] 35107
[2] 35108
2021-02-02T23:05:25.035-0800 7fd270a77c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:05:25.035-0800 7f8fa6606c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:05:25.039-0800 7fd270a77c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:05:25.039-0800 7f8fa6606c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:05:25.039-0800 7fd270a77c00 -1 WARNING: all dangerous and experimental features are enabled.2021-02-02T23:05:25.039-0800 7f8fa6606c00 -1 WARNING: all dangerous and experimental features are enabled.

hints = 1
Maintaining 2 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects
hints = 1
Object prefix: benchmark_data_otccldstore05_35110
Maintaining 2 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects
Object prefix: benchmark_data_otccldstore05_35108
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    0       0         0         0         0         0           -           0
    1       2      8764      8762   34.2245   34.2266 0.000230533 0.000226223
    1       2      8765      8763    34.228   34.2305 0.000238954 0.000226378
    2       2     17617     17615   34.4012    34.582 0.000253242 0.000225088
    2       1     17624     17623   34.4161   34.6094 0.000230366 0.000225122
    3       2     26608     26606   34.6397   35.1211 0.000215777 0.000223459
    3       2     26616     26614   34.6494   35.1211  0.00021594 0.000223556
    4       2     35525     35523    34.687    34.832 0.000218877 0.000223173
    4       2     35532     35530   34.6932   34.8281 0.000206358 0.000223267
    5       2     44622     44620   34.8559   35.5352 0.000230731  0.00022208
    5       1     44630     44629   34.8624    35.543 0.000232921 0.000222167
    6       1     53351     53350   34.7296   34.1016 0.000332273 0.000222906
    6       2     53360     53358   34.7344   34.0977 0.000324601  0.00022297
    7       2     62434     62432   34.8358   35.4766  0.00020295 0.000222224
    7       2     62445     62443   34.8416   35.4883 0.000204535  0.00022225
    8       2     71641     71639   34.9765   35.9648 0.000242869 0.000221325
    8       2     71647     71645   34.9791   35.9453 0.000188274 0.000221351
    9       2     80753     80751   35.0447   35.5938 0.000210213  0.00022089
    9       2     80758     80756   35.0466   35.5898 0.000220958 0.000220911
   10       1     90031     90030   35.1644   36.2461 0.000205608 0.000220101
   10       2     90026     90024   35.1619   36.2031 0.000186574 0.000220167
   11       2     99172     99170   35.2131   35.7031 0.000205781 0.000219782
   11       2     99155     99153   35.2068   35.6602 0.000222838 0.000219875
   12       2    108052    108050    35.169   34.6875 0.000218707 0.000220039
   12       1    108036    108035   35.1639   34.6953 0.000222211 0.000220137
   13       2    115745    115743    34.775   30.0508 0.000276147 0.000222543
   13       2    115780    115778   34.7853   30.2461 0.000265084 0.000222547
   14       2    124694    124692   34.7878    34.957  0.00020055 0.000222459
   14       2    124725    124723   34.7961   34.9414  0.00020544  0.00022247
   15       1    133842    133841   34.8509   35.7383 0.000209918 0.000222063
   15       2    133874    133872   34.8585   35.7383 0.000238309  0.00022207
   16       1    143118    143117   34.9372   36.2344 0.000208224 0.000221516
   16       2    143132    143130   34.9398   36.1641 0.000216755 0.000221548
   17       2    152029    152027   34.9292   34.8047 0.000252092 0.000221572
   17       2    152041    152039   34.9312   34.8008 0.000222415 0.000221611
   18       2    159374    159372   34.5824   28.6914 0.000243701 0.000223811
   18       2    159361    159359   34.5789   28.5938 0.000254687 0.000223893
   19       2    166606    166604   34.2489     28.25 0.000240992 0.000226001
   19       2    166612    166610   34.2494   28.3242 0.000244693 0.000226067
2021-02-02T23:05:45.050721-0800 min lat: 0.00014215 max lat: 0.00644484 avg lat: 0.000228184
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20       2    173705    173703   33.9229   27.7305   0.0012266 0.000228184
2021-02-02T23:05:45.051177-0800 min lat: 0.000139647 max lat: 0.00638065 avg lat: 0.000228172
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20       1    173777    173776   33.9364   27.9922 0.000232231 0.000228172
   21       2    180846    180844   33.6356   27.8945 0.000297002 0.000230146
   21       2    180963    180961   33.6566   28.0664 0.000207891 0.000230089
   22       2    188136    188134   33.4009   28.4766 0.000229901  0.00023178
   22       2    188260    188258   33.4222   28.5039 0.000244517 0.000231715
   23       2    195270    195268   33.1602   27.8672 0.000203067 0.000233478
   23       2    195387    195385   33.1794   27.8398 0.000230537 0.000233426
   24       2    202349    202347   32.9306   27.6523  0.00107865   0.0002351
   24       2    202448    202446    32.946    27.582  0.00117275 0.000235062
   25       2    209478    209476   32.7272   27.8477 0.000236424 0.000236598
   25       2    209596    209594    32.745   27.9219 0.000236082 0.000236548
   26       2    216610    216608   32.5398   27.8594 0.000244188  0.00023797
   26       2    216730    216728   32.5572   27.8672 0.000236294 0.000237924
   27       2    223755    223753   32.3682   27.9102 0.000252229 0.000239247
   27       2    223854    223852    32.382   27.8281 0.000226992 0.000239233
   28       2    230763    230761   32.1898    27.375 0.000237344  0.00024058
   28       2    230854    230852   32.2019   27.3438 0.000207399 0.000240584
   29       1    237609    237608   32.0019   26.7461 0.000234497 0.000241999
   29       2    237747    237745   32.0198   26.9258 0.000232733 0.000241965
Total time run:         30.0002
Total writes made:      244201
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     31.7968
Stddev Bandwidth:       3.71723
Max bandwidth (MB/sec): 36.2461
Min bandwidth (MB/sec): 26.7461
Average IOPS:           8139
Stddev IOPS:            951.61
Max IOPS:               9279
Min IOPS:               6847
Average Latency(s):     0.000243567
Stddev Latency(s):      0.000152039
Max latency(s):         0.00644484
Min latency(s):         0.00014215
Total time run:         30.0001
Total writes made:      244425
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     31.826
Stddev Bandwidth:       3.68654
Max bandwidth (MB/sec): 36.2031
Min bandwidth (MB/sec): 26.9258
Average IOPS:           8147
Stddev IOPS:            943.754
Max IOPS:               9268
Min IOPS:               6893
Average Latency(s):     0.000243454
Stddev Latency(s):      0.000150773
Max latency(s):         0.00638065
Min latency(s):         0.000139647
[2]+  Done                    bin/rados bench -p test-pool -b 4096 -t 2 30 write --no-cleanup

 Performance counter stats for process id '34782':

      53442.519939      task-clock (msec)         #    1.741 CPUs utilized
         1,216,273      context-switches          #    0.023 M/sec
                21      cpu-migrations            #    0.000 K/sec
         2,674,276      page-faults               #    0.050 M/sec
   143,135,708,110      cycles                    #    2.678 GHz
    93,577,040,270      instructions              #    0.65  insn per cycle
    18,006,643,439      branches                  #  336.935 M/sec
       218,208,792      branch-misses             #    1.21% of all branches

      30.694539647 seconds time elapsed
...

ceph 4 KB random reads, two rados bench instances

[build]$ perf stat -p `pgrep -u ${UID} ceph-osd` & bin/rados bench -p test-pool 30 rand & bin/rados bench -p test-pool 30 rand ; sleep 1; killall -INT perf 
[1] 35157
[2] 35158
2021-02-02T23:06:59.551-0800 7f8820c99c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:06:59.555-0800 7f28cd32dc00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:06:59.555-0800 7f8820c99c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:06:59.555-0800 7f8820c99c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:06:59.559-0800 7f28cd32dc00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:06:59.559-0800 7f28cd32dc00 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       5         5         0         0         0           -           0
    1      15     33242     33227   129.768   129.793 0.000430011 0.000477143
    1      15     33777     33762   131.849   131.883 0.000390749 0.000469644
    2      16     69083     69067   134.872       140 0.000500188 0.000459272
    2      16     69650     69634   135.972   140.125 0.000431492 0.000455544
    3      16    105630    105614   137.494   142.762 0.000462931 0.000450609
    3      15    106719    106704   138.906   144.805 0.000413589 0.000446038
    4      15    143864    143849   140.453   149.355 0.000387884 0.000441216
    4      16    144971    144955   141.528   149.418 0.000430534 0.000437842
    5      16    182257    182241   142.353   149.969 0.000434419 0.000435383
    5      15    183408    183393   143.249   150.148 0.000381309 0.000432631
    6      16    221053    221037   143.883   151.547 0.000423929 0.000430791
    6      15    222252    222237    144.66   151.734 0.000368114 0.000428437
    7      15    259698    259683   144.892   150.961 0.000407835 0.000427811
    7      15    260987    260972   145.607   151.309  0.00043704 0.000425667
    8      15    298623    298608   145.785   152.051 0.000393809 0.000425214
    8      15    299988    299973   146.446   152.348 0.000442188 0.000423241
    9      15    337677    337662   146.536   152.555 0.000431467 0.000423051
    9      16    339127    339111   147.159   152.883 0.000394981 0.000421209
   10      16    376320    376304   146.975   150.945 0.000380652  0.00042179
   10      15    378827    378812   147.949   155.082 0.000372934 0.000418973
   11      15    415491    415476   147.523   153.016 0.000414767 0.000420237
   11      15    418416    418401   148.556   154.645 0.000369089 0.000417273
   12      15    454832    454817   148.035   153.676 0.000402417   0.0004188
   12      15    457906    457891   149.029   154.258 0.000399457 0.000415958
   13      16    494219    494203   148.481   153.852  0.00036001  0.00041755
   13      15    497377    497362   149.424   154.184 0.000413895 0.000414863
   14      16    533708    533692   148.893   154.254 0.000395998 0.000416404
   14      16    536899    536883   149.776   154.379  0.00038892 0.000413894
   15      16    573337    573321   149.285   154.801 0.000370388 0.000415319
   15      16    576041    576025   149.982   152.898 0.000364202 0.000413328
   16      15    612880    612865   149.609   154.469 0.000395392 0.000414433
   16      16    615714    615698   150.293   154.973 0.000408898 0.000412483
   17      16    652511    652495   149.913   154.805 0.000465567 0.000413595
   17      15    655465    655450   150.585   155.281 0.000417754 0.000411691
   18      15    692158    692143   150.188   154.875 0.000409808 0.000412847
   18      15    695224    695209   150.846   155.309 0.000383669 0.000410987
   19      16    731833    731817    150.44   154.977 0.000379053 0.000412164
   19      15    735073    735058   151.098    155.66 0.000394891 0.000410308
2021-02-02T23:07:19.566690-0800 min lat: 0.000215955 max lat: 0.00447428 avg lat: 0.000411654
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      15    771310    771295   150.627   154.211 0.000521035 0.000411654
2021-02-02T23:07:19.569697-0800 min lat: 0.000190867 max lat: 0.00445151 avg lat: 0.000409807
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      15    774715    774700   151.284   154.852 0.000368313 0.000409807
   21      16    810650    810634   150.771   153.668 0.000390475 0.000411264
   21      16    814174    814158   151.419   154.133  0.00038017 0.000409447
   22      16    849853    849837   150.878   153.137  0.00037567 0.000410972
   22      15    853459    853444    151.51   153.461 0.000396834 0.000409199
   23      15    889116    889101   150.986   153.375 0.000409038  0.00041068
   23      16    892809    892793   151.605   153.707 0.000364089 0.000408944
   24      15    927810    927795   150.992   151.148 0.000446852 0.000410664
   24      15    931522    931507   151.588   151.227 0.000419309 0.000408986
   25      15    966565    966550   151.008   151.387 0.000406156 0.000410621
   25      16    970266    970250   151.577    151.34 0.000401452 0.000409013
   26      15   1005493   1005478   151.048   152.062  0.00039465 0.000410511
   26      15   1009197   1009182   151.596   152.078 0.000347194 0.000408963
   27      16   1045905   1045889   151.299   157.855 0.000366469 0.000409821
   27      16   1049615   1049599   151.827   157.879 0.000342113 0.000408327
   28      15   1086368   1086353    151.54   158.062 0.000361262 0.000409163
   28      16   1090059   1090043   152.046   157.984 0.000381364  0.00040773
   29      15   1126818   1126803   151.763   158.008 0.000353864 0.000408557
   29      16   1129297   1129281   152.088   153.273 0.000373011 0.000407609
Total time run:       30.0002
Total reads made:     1168374
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   152.131
Average IOPS:         38945
Stddev IOPS:          1455.63
Max IOPS:             40464
Min IOPS:             33227
Average Latency(s):   0.000407572
Max latency(s):       0.00447428
Min latency(s):       0.000198671
Total time run:       30.0002
Total reads made:     1168101
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   152.095
Average IOPS:         38936
Stddev IOPS:          1347.87
Max IOPS:             40444
Min IOPS:             33762
Average Latency(s):   0.00040759
Max latency(s):       0.00445151
Min latency(s):       0.000190867
[2]+  Done                    bin/rados bench -p test-pool 30 rand

 Performance counter stats for process id '34782':

      64628.669812      task-clock (msec)         #    2.102 CPUs utilized
            75,827      context-switches          #    0.001 M/sec
                 9      cpu-migrations            #    0.000 K/sec
         1,016,138      page-faults               #    0.016 M/sec
   205,183,497,074      cycles                    #    3.175 GHz
   160,090,403,816      instructions              #    0.78  insn per cycle
    30,983,267,800      branches                  #  479.404 M/sec
       203,952,144      branch-misses             #    0.66% of all branches

      30.751388356 seconds time elapsed
...
Raw
gistfile4.md

The cycles-per-op metric

ceph-osd

write: 164,925,256,438 /(290146+290153) = 284207.3766
>>>   
read: 204,124,491,811 /(1232861+1235965) = 82680.7931
>>>

crimson-osd

write: 104,166,645,448 /(506508+493477) = 104168.2079
>>> 
read: 98,634,481,323/(1475104+1449760) = 33722.7581
>>>

The avg-lat metric

ceph-osd

write:  0.00661583  0.00661569
read: 0.0015544   0.00155049

crimson-osd

write:   0.00379233    0.00388976
read:  0.00129928    0.00132214

crimson 4 KB random write, two rados bench instances

[build]$ perf stat -p `pgrep -u ${UID} crimson-osd` & bin/rados bench -p test-pool -b 4096 -t 64 30 write  --no-cleanup & bin/rados bench -p test-pool -b 4096 -t 64 30 write  --no-cleanup ; sleep 1; killall -INT perf
[1] 36207
[2] 36208
2021-02-02T23:37:31.259-0800 7f530a28ec00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:37:31.259-0800 7fa333e15c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:37:31.259-0800 7f530a28ec00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:37:31.263-0800 7f530a28ec00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:37:31.263-0800 7fa333e15c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:37:31.263-0800 7fa333e15c00 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
Maintaining 64 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects
Object prefix: benchmark_data_otccldstore05_36210
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
hints = 1
Maintaining 64 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects
Object prefix: benchmark_data_otccldstore05_36208
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      64     19455     19391   75.7415   75.7461  0.00222923   0.0032899
    1      64     19051     18987   74.1554    74.168  0.00193011  0.00336455
    2      64     37998     37934   74.0833   72.4336  0.00294475   0.0033693
    2      63     37880     37817   73.8495   73.5547  0.00437796  0.00337864
    3      63     56231     56168   73.1273   71.2266  0.00514697  0.00341386
    3      63     55440     55377   72.0939   68.5938  0.00373401  0.00346256
    4      64     75880     75816     74.03     76.75  0.00305963   0.0033734
    4      64     75340     75276   73.5004   77.7305  0.00380927  0.00339601
    5      63     96364     96301   75.2249   80.0195  0.00414043  0.00331977
    5      64     94779     94715   73.9853   75.9336  0.00321792  0.00337513
    6      63    116261    116198   75.6392   77.7227  0.00436526  0.00330208
    6      64    113674    113610   73.9545   73.8086   0.0055421  0.00337714
    7      64    136486    136422   76.1175        79  0.00427948  0.00328174
    7      64    133331    133267   74.3579   76.7852  0.00300364  0.00335892
    8      64    153231    153167   74.7777   65.4102  0.00752909  0.00334044
    8      64    149121    149057   72.7725   61.6797  0.00312374    0.003432
    9      64    166795    166731   72.3551   52.9844  0.00314613  0.00345235
    9      64    162764    162700   70.6075    53.293  0.00283412  0.00353747
   10      63    184373    184310   71.9853    68.668  0.00303618  0.00347013
   10      64    179822    179758   70.2095   66.6328  0.00543156   0.0035569
   11      64    200473    200409   71.1572   62.8867  0.00430595  0.00351051
   11      64    195301    195237    69.323   60.4648  0.00460503  0.00360305
   12      63    215965    215902   70.2696   60.5195   0.0034791  0.00355491
   12      64    210280    210216   68.4217   58.5117  0.00422514   0.0036506
   13      63    231522    231459    69.538   60.7695  0.00515941  0.00359234
   13      64    225254    225190   67.6575   58.4922  0.00235896  0.00369156
   14      64    246749    246685   68.8186   59.4766  0.00707112  0.00362993
   14      64    238993    238929   66.6577    53.668   0.0037801  0.00374722
   15      63    263187    263124    68.511   64.2148  0.00529191  0.00364624
   15      63    254514    254451   66.2553   60.6328  0.00389001  0.00377013
   16      64    279806    279742   68.2855   64.9141   0.0035166  0.00365839
   16      63    270710    270647   66.0681   63.2656  0.00301839  0.00378121
   17      64    296595    296531   68.1258    65.582  0.00466228  0.00366682
   17      64    286567    286503   65.8247   61.9375  0.00476633  0.00379478
   18      64    313295    313231   67.9646   65.2344  0.00211792  0.00367602
   18      64    303197    303133   65.7763   64.9609  0.00352966  0.00379811
   19      63    329812    329749    67.783   64.5234  0.00354795   0.0036858
   19      64    319568    319504   65.6799   63.9492  0.00333439  0.00380356
2021-02-02T23:37:51.273012-0800 min lat: 0.00113995 max lat: 0.0185743 avg lat: 0.00369281
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      64    346477    346413    67.648   65.0938  0.00403669  0.00369281
2021-02-02T23:37:51.278288-0800 min lat: 0.00126083 max lat: 0.0183174 avg lat: 0.00381584
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      64    335310    335246   65.4701   61.4922  0.00353469  0.00381584
   21      63    363341    363278   67.5631   65.8789  0.00286182  0.00369784
   21      64    350786    350722   65.2308   60.4531  0.00290771  0.00382977
   22      64    379493    379429   67.3593   63.0898  0.00299851  0.00370899
   22      64    366606    366542   65.0744   61.7969  0.00572297  0.00383891
   23      64    395773    395709   67.1951   63.5938  0.00667096  0.00371782
   23      64    382593    382529     64.96   62.4492   0.0050787  0.00384572
   24      64    411953    411889   67.0284   63.2031  0.00850847  0.00372737
   24      64    398472    398408   64.8376   62.0273  0.00468001    0.003853
   25      64    428506    428442   66.9332   64.6602   0.0026309  0.00373262
   25      63    414296    414233   64.7166   61.8164  0.00334548  0.00386038
   26      64    444607    444543   66.7775   62.8945  0.00414895  0.00374131
   26      64    430254    430190   64.6246    62.332  0.00294565  0.00386576
   27      64    460538    460474   66.6087   62.2305  0.00351254  0.00375089
   27      63    446513    446450   64.5833   63.5156  0.00209933  0.00386843
   28      63    476407    476344   66.4434   61.9922  0.00505479  0.00376009
   28      64    462544    462480   64.5128   62.6172  0.00228703  0.00387262
   29      64    491044    490980   66.1234   57.1719  0.00356206  0.00377847
   29      64    477764    477700    64.338   59.4531  0.00820323  0.00388281
   30      44    506508    506464   65.9351   60.4844  0.00352197  0.00378874
   30      16    493477    493461   64.2454   61.5664  0.00174512  0.00388853
Total time run:         30.0438
Total writes made:      506508
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     65.8554
Stddev Bandwidth:       6.56255
Max bandwidth (MB/sec): 80.0195
Min bandwidth (MB/sec): 52.9844
Average IOPS:           16858
Stddev IOPS:            1680.01
Max IOPS:               20485
Min IOPS:               13564
Average Latency(s):     0.00379233
Stddev Latency(s):      0.0013496
Max latency(s):         0.0462679
Min latency(s):         0.00113995
Total time run:         30.0413
Total writes made:      493477
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     64.1665
Stddev Bandwidth:       6.41792
Max bandwidth (MB/sec): 77.7305
Min bandwidth (MB/sec): 53.293
Average IOPS:           16426
Stddev IOPS:            1642.99
Max IOPS:               19899
Min IOPS:               13643
Average Latency(s):     0.00388976
Stddev Latency(s):      0.00133641
Max latency(s):         0.0419026
Min latency(s):         0.00126083
[2]+  Done                    bin/rados bench -p test-pool -b 4096 -t 64 30 write --no-cleanup

 Performance counter stats for process id '36086':

      29374.847691      task-clock (msec)         #    0.926 CPUs utilized
           119,173      context-switches          #    0.004 M/sec
                 0      cpu-migrations            #    0.000 K/sec
           253,061      page-faults               #    0.009 M/sec
   104,166,645,448      cycles                    #    3.546 GHz
   126,203,558,227      instructions              #    1.21  insn per cycle
    25,343,630,468      branches                  #  862.766 M/sec
       130,516,359      branch-misses             #    0.51% of all branches

      31.737525793 seconds time elapsed
...

crimson 4 KB random reads, two rados bench instances

[build]$ perf stat -p `pgrep -u ${UID} crimson-osd` & bin/rados bench -p test-pool -t 64 30 rand & bin/rados bench -p test-pool -t 64 30 rand ; sleep 1; killall -INT perf 
[1] 36259
[2] 36260
2021-02-02T23:42:09.487-0800 7fb05d06cc00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:42:09.491-0800 7f2303ceac00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:42:09.491-0800 7fb05d06cc00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:42:09.491-0800 7fb05d06cc00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:42:09.495-0800 7f2303ceac00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:42:09.495-0800 7f2303ceac00 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       4         4         0         0         0           -           0
    1      63     47824     47761   186.531   186.566  0.00123794  0.00133531
    1      64     48728     48664   190.045   190.094  0.00133721  0.00131089
    2      63     95994     95931   187.334   188.164  0.00130538  0.00133039
    2      64     97699     97635   190.649   191.293  0.00158145  0.00130746
    3      63    144052    143989   187.456   187.727  0.00132308  0.00133003
    3      63    146003    145940   189.981   188.691   0.0013312  0.00131245
    4      64    192048    191984   187.457    187.48  0.00221173  0.00133012
    4      63    194216    194153   189.561   188.332  0.00133183  0.00131566
    5      63    240215    240152   187.593   188.156  0.00132053  0.00132927
    5      64    242590    242526   189.434   188.957  0.00104856  0.00131657
    6      63    289127    289064   188.168   191.062  0.00126892  0.00132533
    6      64    291801    291737   189.894    192.23  0.00131397   0.0013135
    7      63    338071    338008   188.597   191.188  0.00132733  0.00132238
    7      63    340896    340833   190.159   191.781  0.00107945  0.00131174
    8      63    387034    386971   188.926   191.262   0.0015054  0.00132012
    8      63    389859    389796   190.293   191.262  0.00123939  0.00131084
    9      63    435958    435895   189.166   191.109  0.00157384  0.00131847
    9      63    438944    438881   190.449   191.738  0.00109411   0.0013098
   10      63    485816    485753   189.723   194.758  0.00133419  0.00131464
   10      63    487242    487179   190.268   188.664  0.00130167  0.00131103
   11      63    536389    536326   190.432   197.551   0.0010766  0.00130986
   11      63    536821    536758   190.573   193.668  0.00131066  0.00130897
   12      63    586869    586806   190.993   197.188  0.00152052  0.00130604
   12      64    586197    586133   190.762   192.871  0.00115201  0.00130769
   13      64    636907    636843   191.335   195.457  0.00104798  0.00130371
   13      63    634782    634719   190.685   189.789  0.00128489  0.00130823
   14      63    686760    686697   191.577   194.742  0.00128431  0.00130215
   14      63    682392    682329   190.346   185.977  0.00123851  0.00131059
   15      64    736538    736474   191.767   194.441  0.00142635  0.00130083
   15      63    730039    729976   190.062   186.121   0.0013422  0.00131253
   16      64    786205    786141   191.906   194.012  0.00107362  0.00129991
   16      63    777741    777678   189.827   186.336  0.00133812  0.00131418
   17      64    835869    835805   192.028       194  0.00162353  0.00129911
   17      63    825556    825493   189.646   186.777  0.00157969  0.00131544
   18      63    885481    885418   192.124   193.801  0.00168112  0.00129848
   18      63    873229    873166   189.455   186.223  0.00129197  0.00131681
   19      64    935166    935102   192.225   194.078  0.00141979  0.00129785
   19      63    920817    920754   189.266   185.891   0.0015066  0.00131813
2021-02-02T23:42:29.503870-0800 min lat: 0.000692873 max lat: 0.0038417 avg lat: 0.00129743
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      64    984719    984655   192.291   193.566 0.000895737  0.00129743
2021-02-02T23:42:29.510425-0800 min lat: 0.000851125 max lat: 0.00497497 avg lat: 0.00131906
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      64    968601    968537   189.134   186.652  0.00127531  0.00131906
   21      63   1034292   1034229   192.353   193.648  0.00194582  0.00129698
   21      63   1016420   1016357   189.021   186.797  0.00142075  0.00131987
   22      63   1083668   1083605   192.376   192.875  0.00165159  0.00129687
   22      64   1064359   1064295    188.94   187.258  0.00112972  0.00132045
   23      63   1133188   1133125   192.421   193.438  0.00108632   0.0012966
   23      63   1112272   1112209   188.862   187.164  0.00129014    0.001321
   24      64   1182580   1182516   192.441   192.934  0.00143023  0.00129648
   24      64   1160302   1160238   188.809   187.613  0.00108774  0.00132139
   25      64   1231952   1231888   192.456   192.859 0.000997911  0.00129637
   25      63   1208300   1208237   188.756   187.496  0.00134746  0.00132175
   26      64   1281472   1281408   192.493   193.438 0.000836297  0.00129615
   26      64   1256119   1256055   188.679   186.789  0.00129903   0.0013223
   27      64   1331152   1331088    192.55   194.062  0.00179048  0.00129576
   27      63   1303560   1303497   188.554    185.32  0.00131626  0.00132318
   28      63   1378880   1378817   192.331   186.441   0.0013296  0.00129724
   28      64   1353328   1353264   188.762   194.402   0.0016153  0.00132174
   29      63   1426929   1426866    192.17   187.691  0.00126315  0.00129834
   29      64   1402680   1402616     188.9   192.781  0.00140974  0.00132079
   30       7   1475104   1475097   192.044   188.402  0.00113559  0.00129907
   30      14   1449760   1449746    188.74   184.102  0.00217992  0.00132174
Total time run:       30.0439
Total reads made:     1475104
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   191.79
Average IOPS:         49098
Stddev IOPS:          802.726
Max IOPS:             50573
Min IOPS:             47729
Average Latency(s):   0.00129928
Max latency(s):       0.0439156
Min latency(s):       0.000678091
Total time run:       30.0424
Total reads made:     1449760
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   188.504
Average IOPS:         48257
Stddev IOPS:          716.867
Max IOPS:             49767
Min IOPS:             47130
Average Latency(s):   0.00132214
Max latency(s):       0.042529
Min latency(s):       0.000688252
[2]+  Done                    bin/rados bench -p test-pool -t 64 30 rand

 Performance counter stats for process id '36086':

      29376.546524      task-clock (msec)         #    0.926 CPUs utilized
           119,155      context-switches          #    0.004 M/sec
                 0      cpu-migrations            #    0.000 K/sec
               667      page-faults               #    0.023 K/sec
    98,634,481,323      cycles                    #    3.358 GHz
   142,348,956,479      instructions              #    1.44  insn per cycle
    27,641,219,549      branches                  #  940.928 M/sec
       103,601,927      branch-misses             #    0.37% of all branches

      31.738048132 seconds time elapsed

ceph 4 KB random write, two rados bench instances

[build]$ perf stat -p `pgrep -u ${UID} ceph-osd` & bin/rados bench -p test-pool -b 4096 -t 64 30 write  --no-cleanup & bin/rados bench -p test-pool -b 4096 -t 64 30 write  --no-cleanup ; sleep 1; killall -INT perf
[1] 36877
[2] 36878
2021-02-02T23:53:25.239-0800 7fa125230c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:53:25.239-0800 7f22ee31cc00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:53:25.239-0800 7f22ee31cc00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:53:25.243-0800 7f22ee31cc00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:53:25.243-0800 7fa125230c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:53:25.243-0800 7fa125230c00 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
Maintaining 64 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects
Object prefix: benchmark_data_otccldstore05_36878
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
hints = 1
Maintaining 64 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects
Object prefix: benchmark_data_otccldstore05_36880
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      63     11007     10944   42.7473     42.75  0.00592751  0.00582764
    1      63     11017     10954   42.7808   42.7891  0.00600747  0.00582096
    2      64     21663     21599   42.1801   41.6211  0.00599997  0.00591515
    2      64     21673     21609   42.1963   41.6211  0.00604362  0.00591187
    3      64     32101     32037   41.7085   40.7734  0.00623109  0.00598507
    3      64     32111     32047   41.7187   40.7734  0.00618822  0.00598316
    4      64     42556     42492   41.4895   40.8398  0.00641929  0.00601825
    4      63     42564     42501   41.4963   40.8359  0.00633418  0.00601698
    5      64     52713     52649   41.1253   39.6758  0.00690936  0.00607086
    5      64     52694     52630   41.1089   39.5664   0.0103767  0.00607022
    6      64     63089     63025   41.0251   40.5312   0.0059363   0.0060883
    6      64     63100     63036    41.031   40.6484  0.00594451   0.0060873
    7      64     73798     73734   41.1393    41.832  0.00596273  0.00607178
    7      64     73809     73745   41.1444    41.832  0.00600603  0.00607098
    8      64     84457     84393   41.2006   41.6367  0.00609538   0.0060631
    8      63     84467     84404    41.205   41.6367  0.00600076  0.00606247
    9      64     94934     94870   41.1692   40.9258  0.00613115  0.00606795
    9      64     94944     94880   41.1727   40.9219  0.00611903  0.00606744
   10      63    105477    105414   41.1702   41.1875  0.00603642  0.00606807
   10      64    105488    105424   41.1733   41.1875  0.00606493  0.00606763
   11      64    115002    114938    40.809   37.2031  0.00655117  0.00612193
   11      63    115010    114947   40.8115   37.1992  0.00622616  0.00612157
   12      63    125476    125413   40.8174    40.918  0.00605329  0.00612086
   12      64    125487    125423   40.8201   40.9219  0.00603918  0.00612053
   13      64    136021    135957   40.8453   41.1875  0.00602587   0.0061168
   13      64    136031    135967   40.8477   41.1875  0.00605888  0.00611651
   14      63    146550    146487   40.8653   41.1328  0.00610848   0.0061139
   14      64    146561    146497   40.8676   41.1328  0.00605671  0.00611365
   15      64    156351    156287   40.6926   38.2812  0.00719883  0.00613961
   15      64    156346    156282   40.6908   38.2227  0.00778252  0.00613958
   16      64    165494    165430    40.381   35.7148  0.00590939  0.00618737
   16      64    165488    165424   40.3791   35.7109  0.00772153  0.00618732
   17      64    174528    174464   40.0811   35.2891  0.00804223  0.00623345
   17      63    174539    174476   40.0836   35.3594  0.00669641  0.00623349
   18      64    183497    183433   39.8005   35.0352  0.00746773   0.0062777
   18      63    183499    183436    39.801        35  0.00747284  0.00627775
   19      63    192452    192389   39.5468   34.9844  0.00587607  0.00631828
   19      63    192460    192397   39.5483   35.0039  0.00606239  0.00631808
2021-02-02T23:53:45.252657-0800 min lat: 0.00248417 max lat: 0.0144783 avg lat: 0.0063598
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      64    201230    201166   39.2835   34.2852  0.00975165   0.0063598
2021-02-02T23:53:45.256019-0800 min lat: 0.00292495 max lat: 0.0148166 avg lat: 0.00636024
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      64    201239    201175   39.2852   34.2891  0.00885681  0.00636024
   21      64    210057    209993   39.0545   34.4805  0.00607417  0.00639802
   21      64    210063    209999   39.0555   34.4688  0.00640609  0.00639788
   22      64    218843    218779    38.839   34.3203  0.00691431  0.00643338
   22      64    218842    218778   38.8388    34.293  0.00796055  0.00643345
   23      64    227642    227578   38.6445   34.3711  0.00699205  0.00646607
   23      64    227638    227574   38.6439   34.3594  0.00685595  0.00646594
   24      64    236525    236461   38.4798   34.6992  0.00582162  0.00649375
   24      64    236511    236447   38.4777   34.6602  0.00801169  0.00649371
   25      64    245036    244972   38.2702   33.2461  0.00770847  0.00652932
   25      64    245046    244982    38.272   33.3398  0.00560016  0.00652921
   26      64    253986    253922   38.1427   34.9609  0.00777447  0.00655109
   26      64    253994    253930   38.1442   34.9531  0.00680869   0.0065511
   27      64    263118    263054   38.0509   35.6719  0.00664626  0.00656674
   27      64    263121    263057   38.0517   35.6523  0.00766737  0.00656679
   28      64    272148    272084   37.9515   35.2734  0.00590657  0.00658436
   28      64    272143    272079   37.9513   35.2422  0.00686271  0.00658428
   29      64    281247    281183   37.8682    35.543  0.00677561  0.00659879
   29      63    281252    281189   37.8696   35.5859  0.00581171  0.00659872
   30      17    290146    290129   37.7706   34.9453  0.00681676  0.00661581
   30       5    290153    290148   37.7736   34.9961  0.00553188  0.00661571
Total time run:         30.0068
Total writes made:      290146
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     37.7709
Stddev Bandwidth:       3.15844
Max bandwidth (MB/sec): 42.75
Min bandwidth (MB/sec): 33.2461
Average IOPS:           9669
Stddev IOPS:            808.561
Max IOPS:               10944
Min IOPS:               8511
Average Latency(s):     0.00661583
Stddev Latency(s):      0.00103069
Max latency(s):         0.0167771
Min latency(s):         0.00248417
Total time run:         30.0051
Total writes made:      290153
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     37.7739
Stddev Bandwidth:       3.15766
Max bandwidth (MB/sec): 42.7891
Min bandwidth (MB/sec): 33.3398
Average IOPS:           9670
Stddev IOPS:            808.361
Max IOPS:               10954
Min IOPS:               8535
Average Latency(s):     0.00661569
Stddev Latency(s):      0.00102874
Max latency(s):         0.0167917
Min latency(s):         0.00292495
[2]+  Done                    bin/rados bench -p test-pool -b 4096 -t 64 30 write --no-cleanup

 Performance counter stats for process id '36746':

      59099.078074      task-clock (msec)         #    1.861 CPUs utilized
         1,235,772      context-switches          #    0.021 M/sec
                37      cpu-migrations            #    0.001 K/sec
         3,116,079      page-faults               #    0.053 M/sec
   164,925,256,438      cycles                    #    2.791 GHz
   109,495,487,594      instructions              #    0.66  insn per cycle
    21,126,345,603      branches                  #  357.473 M/sec
       238,399,921      branch-misses             #    1.13% of all branches

      31.752842483 seconds time elapsed
...

ceph 4 KB random reads, two rados bench instances

[build]$ perf stat -p `pgrep -u ${UID} ceph-osd` & bin/rados bench -p test-pool -t 64 30 rand & bin/rados bench -p test-pool -t 64 30 rand ; sleep 1; killall -INT perf 
[1] 36925
[2] 36926
2021-02-02T23:57:58.095-0800 7f8952b34c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:57:58.099-0800 7fb08f656c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:57:58.099-0800 7f8952b34c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:57:58.099-0800 7f8952b34c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:57:58.103-0800 7fb08f656c00 -1 WARNING: all dangerous and experimental features are enabled.
2021-02-02T23:57:58.103-0800 7fb08f656c00 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       5         5         0         0         0           -           0
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       4         4         0         0         0           -           0
    1      64     37422     37358   145.895    145.93   0.0015292  0.00170796
    1      64     37478     37414   146.105   146.148  0.00162079  0.00170554
    2      64     77590     77526   151.391   156.906  0.00157334  0.00164707
    2      63     77623     77560   151.447    156.82   0.0015224  0.00164649
    3      63    117403    117340   152.763   155.523  0.00165219  0.00163264
    3      63    116853    116790   152.035   153.242  0.00164369  0.00164046
    4      64    156638    156574   152.883   153.258  0.00160396  0.00163153
    4      63    156060    155997   152.307   153.152  0.00155508  0.00163766
    5      63    195834    195771   152.926   153.113  0.00161211  0.00163118
    5      64    195287    195223   152.486   153.227  0.00164209  0.00163581
    6      63    236069    236006   153.628   157.168  0.00155318  0.00162381
    6      64    235554    235490   153.282   157.293  0.00158836  0.00162738
    7      64    276464    276400   154.219   157.789  0.00159175  0.00161766
    7      64    275982    275918   153.941   157.922  0.00162229  0.00162046
    8      64    317245    317181   154.852   159.301  0.00153832   0.0016111
    8      64    316751    316687   154.601   159.254  0.00150436  0.00161357
    9      64    358130    358066   155.389   159.707  0.00156665  0.00160546
    9      63    357562    357499   155.133   159.422  0.00446289  0.00160789
   10      63    398980    398917   155.804   159.574  0.00156894  0.00160131
   10      63    398505    398442    155.61   159.934  0.00163243  0.00160313
   11      64    440179    440115   156.268    160.93  0.00158123  0.00159658
   11      64    439292    439228   155.945    159.32  0.00149868   0.0015997
   12      63    481370    481307   156.652   160.906  0.00160362  0.00159268
   12      63    480461    480398   156.349    160.82  0.00155583  0.00159558
   13      63    522531    522468   156.969   160.785  0.00154856  0.00158949
   13      63    521650    521587   156.696   160.895  0.00155259  0.00159206
   14      64    563734    563670   157.252   160.945  0.00154409  0.00158665
   14      64    562818    562754   156.988   160.809  0.00150181  0.00158911
   15      64    605274    605210   157.584   162.266   0.0015057  0.00158329
   15      63    604375    604312   157.342   162.336  0.00157485  0.00158555
   16      64    646809    646745   157.874   162.246  0.00149343  0.00158038
   16      63    645939    645876   157.654   162.359  0.00155197  0.00158243
   17      64    688202    688138   158.097   161.691  0.00150578  0.00157813
   17      64    687863    687799   158.011   163.762  0.00153279  0.00157886
   18      63    729326    729263   158.237   160.645  0.00155426  0.00157673
   18      63    729839    729776   158.341   163.973  0.00150249   0.0015756
   19      63    771438    771375   158.565     164.5  0.00148848  0.00157348
   19      64    771956    771892   158.664   164.516  0.00149321  0.00157241
2021-02-02T23:58:18.111239-0800 min lat: 0.000788995 max lat: 0.00572504 avg lat: 0.0015703
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      64    813684    813620   158.886    165.02  0.00138983   0.0015703
2021-02-02T23:58:18.114787-0800 min lat: 0.000809026 max lat: 0.00581566 avg lat: 0.00156924
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      64    814239    814175   158.988   165.168  0.00147641  0.00156924
   21      63    855880    855817   159.168   164.832  0.00152483  0.00156753
   21      63    856475    856412   159.272   164.988   0.0014484  0.00156646
   22      64    898226    898162    159.45    165.41  0.00143295  0.00156477
   22      64    898813    898749   159.548   165.379  0.00147253  0.00156377
   23      63    940280    940217   159.659   164.277  0.00147085  0.00156273
   23      63    940927    940864   159.762   164.512  0.00153744  0.00156169
   24      63    981845    981782   159.771   162.363  0.00151196  0.00156164
   24      64    983330    983266   160.006   165.633  0.00146478  0.00155933
   25      64   1023959   1023895   159.959   164.504  0.00154949  0.00155981
   25      63   1025462   1025399   160.188   164.582  0.00157022  0.00155757
   26      63   1066133   1066070   160.142   164.746    0.001487  0.00155804
   26      63   1067671   1067608   160.367   164.879  0.00147542  0.00155585
   27      64   1108234   1108170     160.3   164.453  0.00148019  0.00155651
   27      63   1109788   1109725    160.52    164.52  0.00147287  0.00155438
   28      63   1149893   1149830   160.386   162.734  0.00152504  0.00155569
   28      64   1151956   1151892   160.669   164.715   0.0014877  0.00155295
   29      64   1191085   1191021   160.403   160.902  0.00149505  0.00155553
   29      63   1194160   1194097   160.813   164.863  0.00150406  0.00155157
Total time run:       30.0015
Total reads made:     1232861
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   160.521
Average IOPS:         41093
Stddev IOPS:          1122.5
Max IOPS:             42345
Min IOPS:             37358
Average Latency(s):   0.0015544
Max latency(s):       0.00572504
Min latency(s):       0.000788995
Total time run:       30.0009
Total reads made:     1235965
Read size:            4096
Object size:          4096
Bandwidth (MB/sec):   160.928
Average IOPS:         41197
Stddev IOPS:          1223.92
Max IOPS:             42402
Min IOPS:             37414
Average Latency(s):   0.00155049
Max latency(s):       0.00581566
Min latency(s):       0.000809026
[2]+  Done                    bin/rados bench -p test-pool -t 64 30 rand

 Performance counter stats for process id '36746':

      63650.276940      task-clock (msec)         #    2.070 CPUs utilized
            73,086      context-switches          #    0.001 M/sec
                35      cpu-migrations            #    0.001 K/sec
         1,421,685      page-faults               #    0.022 M/sec
   204,124,491,811      cycles                    #    3.207 GHz
   167,755,438,822      instructions              #    0.82  insn per cycle
    32,480,647,599      branches                  #  510.299 M/sec
       213,143,680      branch-misses             #    0.66% of all branches

      30.745869089 seconds time elapsed
...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment