ref-cycles in the RocksDB's abstraction over memcmp. It seems there is a lot of overhead related to indirecations (vtables, compositions) and deserialization. The PERF_COUNTER_ADD() (responsible for __tls_get_addr()) has been removed on hot path.

perf-histogram.0.txt

+   60,55%     0,00%  fio              libc-2.23.so                [.] __clone
+   60,52%     0,00%  fio              libpthread-2.23.so          [.] start_thread
+   60,51%     0,27%  fio              fio                         [.] thread_main
+   52,10%     0,30%  fio              fio                         [.] td_io_queue
+   51,62%     0,46%  fio              libfio_ceph_objectstore.so  [.] (anonymous namespace)::fio_ceph_os_queue
+   41,66%     0,37%  fio              libfio_ceph_objectstore.so  [.] BlueStore::queue_transactions
+   30,53%     0,22%  fio              libfio_ceph_objectstore.so  [.] BlueStore::_txc_add_transaction
+   22,39%     0,12%  fio              libfio_ceph_objectstore.so  [.] BlueStore::_write
+   21,95%     0,24%  fio              libfio_ceph_objectstore.so  [.] BlueStore::_do_write
+   15,73%     0,01%  fio              [kernel]                    [k] 0xffffffff988d603b
+   14,46%     0,86%  fio              libfio_ceph_objectstore.so  [.] BlueStore::_do_alloc_write
-   10,34%     0,21%  bstore_kv_sync   libfio_ceph_objectstore.so  [.] BlueStore::_kv_sync_thread
   - 10,13% BlueStore::_kv_sync_thread
      - 7,82% RocksDBStore::submit_transaction_sync
         - 7,80% RocksDBStore::submit_common
            - 7,78% rocksdb::DBImpl::Write
               - rocksdb::DBImpl::WriteImpl
                  - 5,97% rocksdb::WriteBatchInternal::InsertInto
                     - 5,96% rocksdb::WriteBatch::Iterate
                        - 5,67% rocksdb::MemTableInserter::PutCF
                           - 5,60% rocksdb::MemTable::Add
                              - 5,26% rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::Insert<false>
                                 - 4,59% rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::RecomputeSpliceLevels
                                    - rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::FindSpliceForLevel
                                       - rocksdb::MemTable::KeyComparator::operator()
                                          - 1,80% rocksdb::InternalKeyComparator::Compare
                                             - rocksdb::(anonymous namespace)::BytewiseComparatorImpl::Compare
                                                  __memcmp_sse4_1
                  + 1,70% rocksdb::DBImpl::WriteToWAL
      + 1,63% fuse_and_reorder_transactions
+   10,29%     0,00%  bstore_kv_sync   libfio_ceph_objectstore.so  [.] BlueStore::KVSyncThread::entry