Radoslaw Zarzynski rzarzynski
rzarzynski
/ perf.cycle_activity.stalls_l1d_miss.txt
Created
October 9, 2017 14:34
Ceph BlueStore KV infra: stalled cycles analysis
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Samples: 57K of event 'cycle_activity.stalls_l1d_miss', Event count (approx.): 114520171780, Thread | |
| Children Self Command Shared Object Symbol ▒ | |
| - 14,41% 0,30% bstore_kv_sync libfio_ceph_objectstore.so [.] BlueStore::_kv_sync_thread ◆ | |
| - 14,11% BlueStore::_kv_sync_thread ▒ | |
| - 12,32% RocksDBStore::submit_transaction ▒ | |
| - 12,14% RocksDBStore::submit_common ▒ | |
| - 12,07% rocksdb::DBImpl::Write ▒ | |
| - rocksdb::DBImpl::WriteImpl ▒ | |
| - 10,72% rocksdb::WriteBatchInternal::InsertInto ▒ | |
| - 10,68% rocksdb::WriteBatch::Iterate ▒ |
rzarzynski
/ gdbprof.wall-time.fio.randwrite.4k.txt
Last active
October 9, 2017 14:26
Profiling BlueStore with gdbprof
This file has been truncated, but you can view the full file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 | |
| Copyright (C) 2016 Free Software Foundation, Inc. | |
| License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> | |
| This is free software: you are free to change and redistribute it. | |
| There is NO WARRANTY, to the extent permitted by law. Type "show copying" | |
| and "show warranty" for details. | |
| This GDB was configured as "x86_64-linux-gnu". | |
| Type "show configuration" for configuration details. | |
| For bug reporting instructions, please see: | |
| <http://www.gnu.org/software/gdb/bugs/>. |
rzarzynski
/ perf-annotate.cache-misses.txt
Last active
September 28, 2017 13:43
perf shows that even accessing thread's own stack can be painful. Does RocksDB compare keys in the non-temporal way (MOVNT* of x86) to avoid massive cache pollution?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| rocksdb::MemTable::KeyComparator::operator() /work/ceph-3/build/lib/libfio_ceph_objectstore.so | |
| │ _ZN7rocksdb14GetVarint32PtrEPKcS1_Pj(): | |
| │ uint32_t* value); | |
| │ inline const char* GetVarint32Ptr(const char* p, | |
| │ const char* limit, | |
| │ uint32_t* value) { | |
| │ if (p < limit) { | |
| │ uint32_t result = *(reinterpret_cast<const unsigned char*>(p)); | |
| 0,43 │42b266: movzbl (%rsi),%edx | |
| │ _ZN7rocksdb22GetLengthPrefixedSliceEPKc(): |
rzarzynski
/ perf-histogram.0.txt
Last active
September 27, 2017 20:37
ref-cycles in the RocksDB's abstraction over memcmp. It seems there is a lot of overhead related to indirecations (vtables, compositions) and deserialization. The PERF_COUNTER_ADD() (responsible for __tls_get_addr()) has been removed on hot path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| + 60,55% 0,00% fio libc-2.23.so [.] __clone | |
| + 60,52% 0,00% fio libpthread-2.23.so [.] start_thread | |
| + 60,51% 0,27% fio fio [.] thread_main | |
| + 52,10% 0,30% fio fio [.] td_io_queue | |
| + 51,62% 0,46% fio libfio_ceph_objectstore.so [.] (anonymous namespace)::fio_ceph_os_queue | |
| + 41,66% 0,37% fio libfio_ceph_objectstore.so [.] BlueStore::queue_transactions | |
| + 30,53% 0,22% fio libfio_ceph_objectstore.so [.] BlueStore::_txc_add_transaction | |
| + 22,39% 0,12% fio libfio_ceph_objectstore.so [.] BlueStore::_write | |
| + 21,95% 0,24% fio libfio_ceph_objectstore.so [.] BlueStore::_do_write | |
| + 15,73% 0,01% fio [kernel] [k] 0xffffffff988d603b |
rzarzynski
/ perf-histogram.0.txt
Created
September 27, 2017 14:28
Branch mispredictions in KV transaction fusing & reordering that is built on top of the std::map and rocksdb::WriteBatch::Iterate
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| - 22,54% 0,06% bstore_kv_sync libfio_ceph_objectstore.so [.] BlueStore::_kv_sync_thread | |
| - 22,48% BlueStore::_kv_sync_thread | |
| + 14,55% RocksDBStore::submit_transaction_sync | |
| - 7,12% fuse_and_reorder_transactions | |
| + 5,71% rocksdb::WriteBatch::Iterate | |
| + 0,69% rocksdb::WriteBatch::Put | |
| + 22,50% 0,00% bstore_kv_sync libfio_ceph_objectstore.so [.] BlueStore::KVSyncThread::entry | |
| + 22,50% 0,00% bstore_kv_sync libpthread-2.23.so [.] start_thread | |
| + 22,38% 0,00% bstore_kv_sync libc-2.23.so [.] __clone | |
| + 15,39% 0,08% bstore_kv_sync libfio_ceph_objectstore.so [.] rocksdb::WriteBatch::Iterate |
rzarzynski
/ perf-annotate.txt
Last active
September 27, 2017 14:19
Cache misses in the RocksDB's abstraction over memcmp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| rocksdb::MemTable::KeyComparator::operator() /work/ceph-3/build/lib/libfio_ceph_objectstore.so | |
| │ _ZN7rocksdb14GetVarint32PtrEPKcS1_Pj(): | |
| │ uint32_t* value); | |
| │ inline const char* GetVarint32Ptr(const char* p, | |
| │ const char* limit, | |
| │ uint32_t* value) { | |
| │ if (p < limit) { | |
| │ uint32_t result = *(reinterpret_cast<const unsigned char*>(p)); | |
| 0,51 │ movzbl (%rsi),%edx | |
| │ _ZN7rocksdb22GetLengthPrefixedSliceEPKc(): |
NewerOlder