Skip to content

Instantly share code, notes, and snippets.

View rzarzynski's full-sized avatar

Radoslaw Zarzynski rzarzynski

50 followers · 3 following
  • Red Hat
  • Poznan, Poland
View GitHub Profile
@rzarzynski
rzarzynski / perf-annotate.txt
Last active September 27, 2017 14:19
Cache misses in the RocksDB's abstraction over memcmp
rocksdb::MemTable::KeyComparator::operator() /work/ceph-3/build/lib/libfio_ceph_objectstore.so
│ _ZN7rocksdb14GetVarint32PtrEPKcS1_Pj():
│ uint32_t* value);
│ inline const char* GetVarint32Ptr(const char* p,
│ const char* limit,
│ uint32_t* value) {
│ if (p < limit) {
│ uint32_t result = *(reinterpret_cast<const unsigned char*>(p));
0,51 │ movzbl (%rsi),%edx
│ _ZN7rocksdb22GetLengthPrefixedSliceEPKc():
@rzarzynski
rzarzynski / perf-histogram.0.txt
Created September 27, 2017 14:28
Branch mispredictions in KV transaction fusing & reordering that is built on top of the std::map and rocksdb::WriteBatch::Iterate
- 22,54% 0,06% bstore_kv_sync libfio_ceph_objectstore.so [.] BlueStore::_kv_sync_thread
- 22,48% BlueStore::_kv_sync_thread
+ 14,55% RocksDBStore::submit_transaction_sync
- 7,12% fuse_and_reorder_transactions
+ 5,71% rocksdb::WriteBatch::Iterate
+ 0,69% rocksdb::WriteBatch::Put
+ 22,50% 0,00% bstore_kv_sync libfio_ceph_objectstore.so [.] BlueStore::KVSyncThread::entry
+ 22,50% 0,00% bstore_kv_sync libpthread-2.23.so [.] start_thread
+ 22,38% 0,00% bstore_kv_sync libc-2.23.so [.] __clone
+ 15,39% 0,08% bstore_kv_sync libfio_ceph_objectstore.so [.] rocksdb::WriteBatch::Iterate
@rzarzynski
rzarzynski / perf-histogram.0.txt
Last active September 27, 2017 20:37
ref-cycles in the RocksDB's abstraction over memcmp. It seems there is a lot of overhead related to indirecations (vtables, compositions) and deserialization. The PERF_COUNTER_ADD() (responsible for __tls_get_addr()) has been removed on hot path.
+ 60,55% 0,00% fio libc-2.23.so [.] __clone
+ 60,52% 0,00% fio libpthread-2.23.so [.] start_thread
+ 60,51% 0,27% fio fio [.] thread_main
+ 52,10% 0,30% fio fio [.] td_io_queue
+ 51,62% 0,46% fio libfio_ceph_objectstore.so [.] (anonymous namespace)::fio_ceph_os_queue
+ 41,66% 0,37% fio libfio_ceph_objectstore.so [.] BlueStore::queue_transactions
+ 30,53% 0,22% fio libfio_ceph_objectstore.so [.] BlueStore::_txc_add_transaction
+ 22,39% 0,12% fio libfio_ceph_objectstore.so [.] BlueStore::_write
+ 21,95% 0,24% fio libfio_ceph_objectstore.so [.] BlueStore::_do_write
+ 15,73% 0,01% fio [kernel] [k] 0xffffffff988d603b
@rzarzynski
rzarzynski / perf-annotate.cache-misses.txt
Last active September 28, 2017 13:43
perf shows that even accessing thread's own stack can be painful. Does RocksDB compare keys in the non-temporal way (MOVNT* of x86) to avoid massive cache pollution?
rocksdb::MemTable::KeyComparator::operator() /work/ceph-3/build/lib/libfio_ceph_objectstore.so
│ _ZN7rocksdb14GetVarint32PtrEPKcS1_Pj():
│ uint32_t* value);
│ inline const char* GetVarint32Ptr(const char* p,
│ const char* limit,
│ uint32_t* value) {
│ if (p < limit) {
│ uint32_t result = *(reinterpret_cast<const unsigned char*>(p));
0,43 │42b266: movzbl (%rsi),%edx
│ _ZN7rocksdb22GetLengthPrefixedSliceEPKc():
@rzarzynski
rzarzynski / gdbprof.wall-time.fio.randwrite.4k.txt
Last active October 9, 2017 14:26
Profiling BlueStore with gdbprof
This file has been truncated, but you can view the full file.
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
@rzarzynski
rzarzynski / perf.cycle_activity.stalls_l1d_miss.txt
Created October 9, 2017 14:34
Ceph BlueStore KV infra: stalled cycles analysis
Samples: 57K of event 'cycle_activity.stalls_l1d_miss', Event count (approx.): 114520171780, Thread
Children Self Command Shared Object Symbol ▒
- 14,41% 0,30% bstore_kv_sync libfio_ceph_objectstore.so [.] BlueStore::_kv_sync_thread ◆
- 14,11% BlueStore::_kv_sync_thread ▒
- 12,32% RocksDBStore::submit_transaction ▒
- 12,14% RocksDBStore::submit_common ▒
- 12,07% rocksdb::DBImpl::Write ▒
- rocksdb::DBImpl::WriteImpl ▒
- 10,72% rocksdb::WriteBatchInternal::InsertInto ▒
- 10,68% rocksdb::WriteBatch::Iterate ▒
@rzarzynski
rzarzynski / with.txt
Created October 11, 2017 14:06
Performance tests of commit 92a28f033a5272b7dc2c5d726e67b6d09f6166ba
$ /work/fio/fio /work/fio/examples/rbd.fio
rbd_iodepth32: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=32
fio-3.0-32-g83070
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=17.4MiB/s][r=0,w=4467 IOPS][eta 00m:00s]
rbd_iodepth32: (groupid=0, jobs=1): err= 0: pid=28369: Wed Oct 11 15:16:59 2017
write: IOPS=4321, BW=16.9MiB/s (17.7MB/s)(4096MiB/242614msec)
slat (nsec): min=626, max=1643.6k, avg=2736.83, stdev=6738.13
clat (usec): min=692, max=254925, avg=7399.87, stdev=3855.18
lat (usec): min=694, max=254927, avg=7402.61, stdev=3854.96
@rzarzynski
rzarzynski / with_92a28f033a5272b7dc2c5d726e67b6d09f6166ba_rocksdb_enable_rmrange=true.txt
Last active October 12, 2017 14:33
Performance tests of commit 92a28f033a5272b7dc2c5d726e67b6d09f6166ba and the rocksdb_enable_rmrange configurable
$ /work/fio/fio /work/fio/examples/rbd.fio
rbd_iodepth32: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=32
fio-3.0-32-g83070
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=16.6MiB/s][r=0,w=4248 IOPS][eta 00m:00s]
rbd_iodepth32: (groupid=0, jobs=1): err= 0: pid=14177: Thu Oct 12 14:54:48 2017
write: IOPS=4211, BW=16.5MiB/s (17.3MB/s)(4096MiB/248960msec)
slat (nsec): min=637, max=1264.4k, avg=2790.40, stdev=7299.46
clat (usec): min=768, max=194612, avg=7593.42, stdev=3468.52
lat (usec): min=772, max=194614, avg=7596.21, stdev=3468.28
@rzarzynski
rzarzynski / after.txt
Created October 18, 2017 14:30
Testing boost::container::flat_map for BlueStore's coll_map
Samples: 702K of event 'cycles:p', Event count (approx.): 240519119891
Children Self Command Shared Object Symbol ◆
+ 62,05% 0,00% fio libc-2.23.so [.] __clone ▒
+ 62,05% 0,00% fio libpthread-2.23.so [.] start_thread ▒
+ 62,03% 0,22% fio fio [.] thread_main ▒
+ 52,81% 0,19% fio fio [.] td_io_queue ▒
+ 52,50% 0,38% fio libfio_ceph_objectstore.so [.] (anonymous namespace)::fio_ceph_os_queu▒
- 42,50% 0,17% fio libfio_ceph_objectstore.so [.] BlueStore::queue_transactions ▒
- 42,33% BlueStore::queue_transactions ▒
- 29,66% BlueStore::_txc_add_transaction
@rzarzynski
rzarzynski / reference_43b88419d4f87da514721848d28fe949cf0ecb9e.txt
Last active January 4, 2018 22:52
wip-bs-async-read-rebased: profilling of the sync read path
4 KiB randread over 1 GiB RBD volume provisioned with 4 MiB chunks
rbd_iodepth32: (groupid=0, jobs=1): err= 0: pid=11881: Thu Jan 4 23:25:27 2018
read: IOPS=25.3k, BW=98.0MiB/s (104MB/s)(5939MiB/60002msec)
slat (nsec): min=1000, max=7019.8k, avg=7565.57, stdev=28274.55
clat (usec): min=31, max=99654, avg=1254.29, stdev=778.99
lat (usec): min=128, max=99655, avg=1261.86, stdev=778.78
clat percentiles (usec):
| 1.00th=[ 351], 5.00th=[ 537], 10.00th=[ 660], 20.00th=[ 824],
| 30.00th=[ 947], 40.00th=[ 1045], 50.00th=[ 1139], 60.00th=[ 1254],