rzarzynski/20190613_ifb_0d94a3a7a8b6eeee8d16cd5e937811c67471f6a1_summary.md Secret

## 20190613_ifb_0d94a3a7a8b6eeee8d16cd5e937811c67471f6a1_summary.md

      
    Raw
  

              20190613_ifb_0d94a3a7a8b6eeee8d16cd5e937811c67471f6a1_summary.md
            
          
    Evaluation of input_buffer_factory in crimson-osd

Summary

Average write bandwidth (MB/s) for single rados bench load generator.


chunk size
0d94a3a7a8b6eeee8d16cd5e937811c67471f6a1
reference


4 KB
84.71
84.65


8 KB
175.56
170.67


32 KB
648.03
604.519


128 KB
2185.79
1623.47


512 KB
3904.72
2156.67


4 MB
3708.27
2072.82


Modifications to crimson-osd

Writing in CyanStore has been bypassed to not exhaust RAM in 10s runs, so the test is focused solely the network subsystem.
The performance bonus for having properly aligned buffers isn't visible as crimson-osd doesn't offer a persistent store at this stage. Vanilla CyanStore just buries data in RAM.
diff --git a/src/crimson/os/cyan_store.cc b/src/crimson/os/cyan_store.cc
index 2486c53..30f4507 100644
--- a/src/crimson/os/cyan_store.cc
+++ b/src/crimson/os/cyan_store.cc
@@ -348,7 +348,7 @@ int CyanStore::_write(const coll_t& cid, const ghobject_t& oid,
   ObjectRef o = c->get_or_create_object(oid);
   if (len > 0) {
     const ssize_t old_size = o->get_size();
-    o->write(offset, bl);
+    //o->write(offset, bl);
     used_bytes += (o->get_size() - old_size);
   }

  
## 20190613_ifb_0d94a3a7a8b6eeee8d16cd5e937811c67471f6a1_testing.txt
rzarz@ubulap:/work/ceph-3/build$ bin/rados bench -p test-pool 10 write -b 4194304 --no-cleanup
2019-06-13 11:23:05.545 7feee2953a40 -1 WARNING: all dangerous and experimental features are enabled.
2019-06-13 11:23:05.549 7feee2953a40 -1 WARNING: all dangerous and experimental features are enabled.
2019-06-13 11:23:05.549 7feee2953a40 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_ubulap_14189
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       944       928    3711.5      3712    0.015791   0.0171283
    2      16      1875      1859   3717.53      3724   0.0157876   0.0171225
    3      16      2718      2702   3602.24      3372   0.0223342   0.0177049
    4      16      3646      3630   3629.59      3712   0.0156672   0.0176192
    5      16      4585      4569   3654.79      3756   0.0157205   0.0174887
    6      16      5525      5509   3672.26      3760   0.0158157   0.0173947
    7      16      6468      6452   3686.45      3772   0.0156713   0.0173261
    8      16      7398      7382    3690.6      3720   0.0156756   0.0173079
    9      16      8343      8327   3700.48      3780   0.0157431   0.0172794
   10      16      9285      9269   3707.19      3768   0.0149308   0.0172562
Total time run:         10.0165
Total writes made:      9286
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     3708.27
Stddev Bandwidth:       120.745
Max bandwidth (MB/sec): 3780
Min bandwidth (MB/sec): 3372
Average IOPS:           927
Stddev IOPS:            30.1863
Max IOPS:               945
Min IOPS:               843
Average Latency(s):     0.0172542
Stddev Latency(s):      0.00560234
Max latency(s):         0.112665
Min latency(s):         0.00254351
rzarz@ubulap:/work/ceph-3/build$ ^C
rzarz@ubulap:/work/ceph-3/build$ bin/rados bench -p test-pool 10 write -b 524288 --no-cleanup
2019-06-13 11:23:26.113 7ff44e169a40 -1 WARNING: all dangerous and experimental features are enabled.
2019-06-13 11:23:26.117 7ff44e169a40 -1 WARNING: all dangerous and experimental features are enabled.
2019-06-13 11:23:26.117 7ff44e169a40 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
Maintaining 16 concurrent writes of 524288 bytes to objects of size 524288 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_ubulap_14215
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16      7825      7809   3903.89    3904.5  0.00193001  0.00204488
    2      16     15632     15616    3903.5    3903.5  0.00207548  0.00204702
    3      16     23565     23549   3924.34    3966.5  0.00184462  0.00203738
    4      16     31395     31379   3921.89      3915  0.00188658  0.00203827
    5      16     39226     39210   3920.55    3915.5   0.0019284   0.0020393
    6      16     47046     47030   3918.71      3910  0.00192425   0.0020403
    7      16     54826     54810   3914.56      3890  0.00192067  0.00204285
    8      16     62543     62527   3907.51    3858.5  0.00155498  0.00204632
    9      16     70299     70283    3904.2      3878  0.00202009  0.00204824
   10       8     78105     78097   3904.44      3907  0.00200238  0.00204812
Total time run:         10.0013
Total writes made:      78105
Write size:             524288
Object size:            524288
Bandwidth (MB/sec):     3904.72
Stddev Bandwidth:       28.1662
Max bandwidth (MB/sec): 3966.5
Min bandwidth (MB/sec): 3858.5
Average IOPS:           7809
Stddev IOPS:            56.3324
Max IOPS:               7933
Min IOPS:               7717
Average Latency(s):     0.0020481
Stddev Latency(s):      0.00038033
Max latency(s):         0.00519594
Min latency(s):         0.000438046
rzarz@ubulap:/work/ceph-3/build$ bin/rados bench -p test-pool 10 write -b 131072 --no-cleanup
2019-06-13 11:24:54.901 7fd09bed8a40 -1 WARNING: all dangerous and experimental features are enabled.
2019-06-13 11:24:54.905 7fd09bed8a40 -1 WARNING: all dangerous and experimental features are enabled.
2019-06-13 11:24:54.905 7fd09bed8a40 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
Maintaining 16 concurrent writes of 131072 bytes to objects of size 131072 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_ubulap_14291
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16     17576     17560   2194.89      2195  0.00111331 0.000910235
    2      16     35069     35053   2190.67   2186.62 0.000589368 0.000912114
    3      15     52640     52625   2192.55    2196.5 0.000998841 0.000911447
    4      16     70291     70275   2195.91   2206.25 0.000496421  0.00091012
    5      16     87780     87764   2193.91   2186.12 0.000795104 0.000910919
    6      16    104988    104972   2186.73      2151 0.000424628 0.000913919
    7      16    122524    122508   2187.46      2192 0.000990421 0.000913642
    8      16    140163    140147    2189.6   2204.88 0.000891372 0.000912776
    9      16    157409    157393   2185.82   2155.75 0.000584024 0.000914337
Total time run:         10.0007
Total writes made:      174875
Write size:             131072
Object size:            131072
Bandwidth (MB/sec):     2185.79
Stddev Bandwidth:       19.7928
Max bandwidth (MB/sec): 2206.25
Min bandwidth (MB/sec): 2151
Average IOPS:           17486
Stddev IOPS:            158.343
Max IOPS:               17650
Min IOPS:               17208
Average Latency(s):     0.000914397
Stddev Latency(s):      0.000205522
Max latency(s):         0.00201977
Min latency(s):         0.000292885
rzarz@ubulap:/work/ceph-3/build$ bin/rados bench -p test-pool 10 write -b 32768 --no-cleanup
2019-06-13 11:25:53.193 7f6fdf2c8a40 -1 WARNING: all dangerous and experimental features are enabled.
2019-06-13 11:25:53.197 7f6fdf2c8a40 -1 WARNING: all dangerous and experimental features are enabled.
2019-06-13 11:25:53.201 7f6fdf2c8a40 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
Maintaining 16 concurrent writes of 32768 bytes to objects of size 32768 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_ubulap_14332
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      15     20659     20644   645.089   645.125 0.000753405 0.000773881
    2      16     41101     41085   641.906   638.781 0.000847279 0.000778115
    3      16     62136     62120   647.025   657.344 0.000819362 0.000771902
    4      16     83391     83375   651.309   664.219 0.000849632 0.000766948
    5      15    104124    104109   650.621   647.938 0.000898156 0.000767798
    6      16    124465    124449   648.113   635.625 0.000820268 0.000770732
    7      16    144655    144639   645.649   630.938 0.000703497 0.000773683
    8      16    165582    165566   646.682   653.969 0.000380198 0.000772449
    9      16    186574    186558   647.711       656 0.000359595  0.00077124
Total time run:         10.0005
Total writes made:      207382
Write size:             32768
Object size:            32768
Bandwidth (MB/sec):     648.037
Stddev Bandwidth:       11.1076
Max bandwidth (MB/sec): 664.219
Min bandwidth (MB/sec): 630.938
Average IOPS:           20737
Stddev IOPS:            355.443
Max IOPS:               21255
Min IOPS:               20190
Average Latency(s):     0.000770863
Stddev Latency(s):      9.25743e-05
Max latency(s):         0.00154499
Min latency(s):         0.000169398
rzarz@ubulap:/work/ceph-3/build$ bin/rados bench -p test-pool 10 write -b 8192 --no-cleanup
2019-06-13 11:26:37.125 7f5189b51a40 -1 WARNING: all dangerous and experimental features are enabled.
2019-06-13 11:26:37.129 7f5189b51a40 -1 WARNING: all dangerous and experimental features are enabled.
2019-06-13 11:26:37.129 7f5189b51a40 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
Maintaining 16 concurrent writes of 8192 bytes to objects of size 8192 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_ubulap_14374
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16     22729     22713   177.434   177.445 0.000707697 0.000703602
    2      15     45252     45237   176.693   175.969 0.000693497 0.000706538
    3      16     67474     67458   175.656   173.602 0.000702474  0.00071078
    4      16     90028     90012   175.788   176.203 0.000718799 0.000710311
    5      16    112678    112662   176.017   176.953 0.000729054 0.000709391
    6      15    135599    135584   176.524   179.078 0.000748394 0.000707352
    7      16    157964    157948   176.264   174.719 0.000582272 0.000708427
    8      16    180272    180256   176.014   174.281 0.000729156 0.000709478
    9      16    202420    202404   175.681   173.031 0.000696343 0.000710797
Total time run:         10.0006
Total writes made:      224739
Write size:             8192
Object size:            8192
Bandwidth (MB/sec):     175.568
Stddev Bandwidth:       1.96389
Max bandwidth (MB/sec): 179.078
Min bandwidth (MB/sec): 173.031
Average IOPS:           22472
Stddev IOPS:            251.378
Max IOPS:               22922
Min IOPS:               22148
Average Latency(s):     0.000711282
Stddev Latency(s):      6.05256e-05
Max latency(s):         0.00145628
Min latency(s):         0.000211509
rzarz@ubulap:/work/ceph-3/build$ bin/rados bench -p test-pool 10 write -b 4096 --no-cleanup
2019-06-13 11:27:18.129 7f20fc613a40 -1 WARNING: all dangerous and experimental features are enabled.
2019-06-13 11:27:18.137 7f20fc613a40 -1 WARNING: all dangerous and experimental features are enabled.
2019-06-13 11:27:18.137 7f20fc613a40 -1 WARNING: all dangerous and experimental features are enabled.
hints = 1
Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_ubulap_14403
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16     21814     21798    85.144   85.1484 0.000744237 0.000732946
    2      15     43469     43454   84.8636   84.5938 0.000748761 0.000735756
    3      15     65277     65262   84.9693   85.1875 0.000684072 0.000734823
    4      15     87037     87022   84.9752        85 0.000713883  0.00073481
    5      15    108083    108068   84.4202   82.2109 0.000741228 0.000739624
    6      15    129725    129710   84.4389   84.5391   0.0007119 0.000739496
    7      16    151114    151098   84.3102   83.5469 0.000702538 0.000740628
    8      15    172862    172847   84.3901    84.957 0.000722552 0.000739927
    9      16    194724    194708   84.5009   85.3945 0.000662215 0.000738947
Total time run:         10.0004
Total writes made:      216888
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     84.7186
Stddev Bandwidth:       1.0192
Max bandwidth (MB/sec): 85.3945
Min bandwidth (MB/sec): 82.2109
Average IOPS:           21687
Stddev IOPS:            260.914
Max IOPS:               21861
Min IOPS:               21046
Average Latency(s):     0.000737065
Stddev Latency(s):      9.21797e-05
Max latency(s):         0.00475167
Min latency(s):         0.00020717
chunk size	0d94a3a7a8b6eeee8d16cd5e937811c67471f6a1	reference
4 KB	84.71	84.65
8 KB	175.56	170.67
32 KB	648.03	604.519
128 KB	2185.79	1623.47
512 KB	3904.72	2156.67
4 MB	3708.27	2072.82
	rzarz@ubulap:/work/ceph-3/build$ bin/rados bench -p test-pool 10 write -b 4194304 --no-cleanup
	2019-06-13 11:23:05.545 7feee2953a40 -1 WARNING: all dangerous and experimental features are enabled.
	2019-06-13 11:23:05.549 7feee2953a40 -1 WARNING: all dangerous and experimental features are enabled.
	2019-06-13 11:23:05.549 7feee2953a40 -1 WARNING: all dangerous and experimental features are enabled.
	hints = 1
	Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
	Object prefix: benchmark_data_ubulap_14189
	sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
	0 0 0 0 0 0 - 0
	1 16 944 928 3711.5 3712 0.015791 0.0171283
	2 16 1875 1859 3717.53 3724 0.0157876 0.0171225
	3 16 2718 2702 3602.24 3372 0.0223342 0.0177049
	4 16 3646 3630 3629.59 3712 0.0156672 0.0176192
	5 16 4585 4569 3654.79 3756 0.0157205 0.0174887
	6 16 5525 5509 3672.26 3760 0.0158157 0.0173947
	7 16 6468 6452 3686.45 3772 0.0156713 0.0173261
	8 16 7398 7382 3690.6 3720 0.0156756 0.0173079
	9 16 8343 8327 3700.48 3780 0.0157431 0.0172794
	10 16 9285 9269 3707.19 3768 0.0149308 0.0172562
	Total time run: 10.0165
	Total writes made: 9286
	Write size: 4194304
	Object size: 4194304
	Bandwidth (MB/sec): 3708.27
	Stddev Bandwidth: 120.745
	Max bandwidth (MB/sec): 3780
	Min bandwidth (MB/sec): 3372
	Average IOPS: 927
	Stddev IOPS: 30.1863
	Max IOPS: 945
	Min IOPS: 843
	Average Latency(s): 0.0172542
	Stddev Latency(s): 0.00560234
	Max latency(s): 0.112665
	Min latency(s): 0.00254351
	rzarz@ubulap:/work/ceph-3/build$ ^C
	rzarz@ubulap:/work/ceph-3/build$ bin/rados bench -p test-pool 10 write -b 524288 --no-cleanup
	2019-06-13 11:23:26.113 7ff44e169a40 -1 WARNING: all dangerous and experimental features are enabled.
	2019-06-13 11:23:26.117 7ff44e169a40 -1 WARNING: all dangerous and experimental features are enabled.
	2019-06-13 11:23:26.117 7ff44e169a40 -1 WARNING: all dangerous and experimental features are enabled.
	hints = 1
	Maintaining 16 concurrent writes of 524288 bytes to objects of size 524288 for up to 10 seconds or 0 objects
	Object prefix: benchmark_data_ubulap_14215
	sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
	0 0 0 0 0 0 - 0
	1 16 7825 7809 3903.89 3904.5 0.00193001 0.00204488
	2 16 15632 15616 3903.5 3903.5 0.00207548 0.00204702
	3 16 23565 23549 3924.34 3966.5 0.00184462 0.00203738
	4 16 31395 31379 3921.89 3915 0.00188658 0.00203827
	5 16 39226 39210 3920.55 3915.5 0.0019284 0.0020393
	6 16 47046 47030 3918.71 3910 0.00192425 0.0020403
	7 16 54826 54810 3914.56 3890 0.00192067 0.00204285
	8 16 62543 62527 3907.51 3858.5 0.00155498 0.00204632
	9 16 70299 70283 3904.2 3878 0.00202009 0.00204824
	10 8 78105 78097 3904.44 3907 0.00200238 0.00204812
	Total time run: 10.0013
	Total writes made: 78105
	Write size: 524288
	Object size: 524288
	Bandwidth (MB/sec): 3904.72
	Stddev Bandwidth: 28.1662
	Max bandwidth (MB/sec): 3966.5
	Min bandwidth (MB/sec): 3858.5
	Average IOPS: 7809
	Stddev IOPS: 56.3324
	Max IOPS: 7933
	Min IOPS: 7717
	Average Latency(s): 0.0020481
	Stddev Latency(s): 0.00038033
	Max latency(s): 0.00519594
	Min latency(s): 0.000438046
	rzarz@ubulap:/work/ceph-3/build$ bin/rados bench -p test-pool 10 write -b 131072 --no-cleanup
	2019-06-13 11:24:54.901 7fd09bed8a40 -1 WARNING: all dangerous and experimental features are enabled.
	2019-06-13 11:24:54.905 7fd09bed8a40 -1 WARNING: all dangerous and experimental features are enabled.
	2019-06-13 11:24:54.905 7fd09bed8a40 -1 WARNING: all dangerous and experimental features are enabled.
	hints = 1
	Maintaining 16 concurrent writes of 131072 bytes to objects of size 131072 for up to 10 seconds or 0 objects
	Object prefix: benchmark_data_ubulap_14291
	sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
	0 0 0 0 0 0 - 0
	1 16 17576 17560 2194.89 2195 0.00111331 0.000910235
	2 16 35069 35053 2190.67 2186.62 0.000589368 0.000912114
	3 15 52640 52625 2192.55 2196.5 0.000998841 0.000911447
	4 16 70291 70275 2195.91 2206.25 0.000496421 0.00091012
	5 16 87780 87764 2193.91 2186.12 0.000795104 0.000910919
	6 16 104988 104972 2186.73 2151 0.000424628 0.000913919
	7 16 122524 122508 2187.46 2192 0.000990421 0.000913642
	8 16 140163 140147 2189.6 2204.88 0.000891372 0.000912776
	9 16 157409 157393 2185.82 2155.75 0.000584024 0.000914337
	Total time run: 10.0007
	Total writes made: 174875
	Write size: 131072
	Object size: 131072
	Bandwidth (MB/sec): 2185.79
	Stddev Bandwidth: 19.7928
	Max bandwidth (MB/sec): 2206.25
	Min bandwidth (MB/sec): 2151
	Average IOPS: 17486
	Stddev IOPS: 158.343
	Max IOPS: 17650
	Min IOPS: 17208
	Average Latency(s): 0.000914397
	Stddev Latency(s): 0.000205522
	Max latency(s): 0.00201977
	Min latency(s): 0.000292885
	rzarz@ubulap:/work/ceph-3/build$ bin/rados bench -p test-pool 10 write -b 32768 --no-cleanup
	2019-06-13 11:25:53.193 7f6fdf2c8a40 -1 WARNING: all dangerous and experimental features are enabled.
	2019-06-13 11:25:53.197 7f6fdf2c8a40 -1 WARNING: all dangerous and experimental features are enabled.
	2019-06-13 11:25:53.201 7f6fdf2c8a40 -1 WARNING: all dangerous and experimental features are enabled.
	hints = 1
	Maintaining 16 concurrent writes of 32768 bytes to objects of size 32768 for up to 10 seconds or 0 objects
	Object prefix: benchmark_data_ubulap_14332
	sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
	0 0 0 0 0 0 - 0
	1 15 20659 20644 645.089 645.125 0.000753405 0.000773881
	2 16 41101 41085 641.906 638.781 0.000847279 0.000778115
	3 16 62136 62120 647.025 657.344 0.000819362 0.000771902
	4 16 83391 83375 651.309 664.219 0.000849632 0.000766948
	5 15 104124 104109 650.621 647.938 0.000898156 0.000767798
	6 16 124465 124449 648.113 635.625 0.000820268 0.000770732
	7 16 144655 144639 645.649 630.938 0.000703497 0.000773683
	8 16 165582 165566 646.682 653.969 0.000380198 0.000772449
	9 16 186574 186558 647.711 656 0.000359595 0.00077124
	Total time run: 10.0005
	Total writes made: 207382
	Write size: 32768
	Object size: 32768
	Bandwidth (MB/sec): 648.037
	Stddev Bandwidth: 11.1076
	Max bandwidth (MB/sec): 664.219
	Min bandwidth (MB/sec): 630.938
	Average IOPS: 20737
	Stddev IOPS: 355.443
	Max IOPS: 21255
	Min IOPS: 20190
	Average Latency(s): 0.000770863
	Stddev Latency(s): 9.25743e-05
	Max latency(s): 0.00154499
	Min latency(s): 0.000169398
	rzarz@ubulap:/work/ceph-3/build$ bin/rados bench -p test-pool 10 write -b 8192 --no-cleanup
	2019-06-13 11:26:37.125 7f5189b51a40 -1 WARNING: all dangerous and experimental features are enabled.
	2019-06-13 11:26:37.129 7f5189b51a40 -1 WARNING: all dangerous and experimental features are enabled.
	2019-06-13 11:26:37.129 7f5189b51a40 -1 WARNING: all dangerous and experimental features are enabled.
	hints = 1
	Maintaining 16 concurrent writes of 8192 bytes to objects of size 8192 for up to 10 seconds or 0 objects
	Object prefix: benchmark_data_ubulap_14374
	sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
	0 0 0 0 0 0 - 0
	1 16 22729 22713 177.434 177.445 0.000707697 0.000703602
	2 15 45252 45237 176.693 175.969 0.000693497 0.000706538
	3 16 67474 67458 175.656 173.602 0.000702474 0.00071078
	4 16 90028 90012 175.788 176.203 0.000718799 0.000710311
	5 16 112678 112662 176.017 176.953 0.000729054 0.000709391
	6 15 135599 135584 176.524 179.078 0.000748394 0.000707352
	7 16 157964 157948 176.264 174.719 0.000582272 0.000708427
	8 16 180272 180256 176.014 174.281 0.000729156 0.000709478
	9 16 202420 202404 175.681 173.031 0.000696343 0.000710797
	Total time run: 10.0006
	Total writes made: 224739
	Write size: 8192
	Object size: 8192
	Bandwidth (MB/sec): 175.568
	Stddev Bandwidth: 1.96389
	Max bandwidth (MB/sec): 179.078
	Min bandwidth (MB/sec): 173.031
	Average IOPS: 22472
	Stddev IOPS: 251.378
	Max IOPS: 22922
	Min IOPS: 22148
	Average Latency(s): 0.000711282
	Stddev Latency(s): 6.05256e-05
	Max latency(s): 0.00145628
	Min latency(s): 0.000211509
	rzarz@ubulap:/work/ceph-3/build$ bin/rados bench -p test-pool 10 write -b 4096 --no-cleanup
	2019-06-13 11:27:18.129 7f20fc613a40 -1 WARNING: all dangerous and experimental features are enabled.
	2019-06-13 11:27:18.137 7f20fc613a40 -1 WARNING: all dangerous and experimental features are enabled.
	2019-06-13 11:27:18.137 7f20fc613a40 -1 WARNING: all dangerous and experimental features are enabled.
	hints = 1
	Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects
	Object prefix: benchmark_data_ubulap_14403
	sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
	0 0 0 0 0 0 - 0
	1 16 21814 21798 85.144 85.1484 0.000744237 0.000732946
	2 15 43469 43454 84.8636 84.5938 0.000748761 0.000735756
	3 15 65277 65262 84.9693 85.1875 0.000684072 0.000734823
	4 15 87037 87022 84.9752 85 0.000713883 0.00073481
	5 15 108083 108068 84.4202 82.2109 0.000741228 0.000739624
	6 15 129725 129710 84.4389 84.5391 0.0007119 0.000739496
	7 16 151114 151098 84.3102 83.5469 0.000702538 0.000740628
	8 15 172862 172847 84.3901 84.957 0.000722552 0.000739927
	9 16 194724 194708 84.5009 85.3945 0.000662215 0.000738947
	Total time run: 10.0004
	Total writes made: 216888
	Write size: 4096
	Object size: 4096
	Bandwidth (MB/sec): 84.7186
	Stddev Bandwidth: 1.0192
	Max bandwidth (MB/sec): 85.3945
	Min bandwidth (MB/sec): 82.2109
	Average IOPS: 21687
	Stddev IOPS: 260.914
	Max IOPS: 21861
	Min IOPS: 21046
	Average Latency(s): 0.000737065
	Stddev Latency(s): 9.21797e-05
	Max latency(s): 0.00475167
	Min latency(s): 0.00020717