Skip to content

Instantly share code, notes, and snippets.

View rzarzynski's full-sized avatar

Radoslaw Zarzynski rzarzynski

50 followers · 3 following
  • Red Hat
  • Poznan, Poland
View GitHub Profile
@rzarzynski
rzarzynski / 20180119_wip-bs-async-read-delegation-shwq_bstore_aio_on_4KiB_randreads.txt
Created January 19, 2018 15:31
bstore_aio load during 4 KiB randreads
Thread: 120 (bstore_aio) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% KernelDevice::AioCompletionThread::entry
+ 100.00% KernelDevice::_aio_thread
+ 89.50% BlueStore::AioReadBatch::aio_finish
| + 85.30% BlueStore::PlainRegionReader::postprocess
| | + 63.30% BlueStore::RegionReader::_verify_csum
| | | + 63.30% bluestore_blob_t::verify_csum
@rzarzynski
rzarzynski / 20180119_wip-bs-async-read-delegation-shwq_bstore_aio_on_4MiB_randreads.txt
Last active January 19, 2018 15:23
bstore_aio load during 4 MiB reads & randreads
Thread: 120 (bstore_aio) - 1000 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% KernelDevice::AioCompletionThread::entry
+ 100.00% KernelDevice::_aio_thread
+ 55.00% aio_queue_t::get_next_completed
| + 54.70% __io_getevents_0_4
| + 0.10% io_getevents
+ 44.70% BlueStore::AioReadBatch::aio_finish
@rzarzynski
rzarzynski / wip-bs-async-delegation---rbd_randread_4kB_1G_bscache.txt
Created January 18, 2018 15:29
4k randreads over 1 GB RBD image - the cache hit scenario
rbd_iodepth32: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=32
fio-3.0-32-g83070-dirty
Starting 1 process
rbd_iodepth32: (groupid=0, jobs=1): err= 0: pid=32696: Tue Jan 16 13:25:51 2018
read: IOPS=29.1k, BW=114MiB/s (119MB/s)(6814MiB/60001msec)
slat (nsec): min=1004, max=872435, avg=7025.71, stdev=18582.37
clat (usec): min=141, max=9765, avg=1092.74, stdev=350.96
lat (usec): min=148, max=9790, avg=1099.76, stdev=349.95
clat percentiles (usec):
@rzarzynski
rzarzynski / hot spots during UNBUFFERED 4k randreads
Last active January 18, 2018 15:27
Hot spots:
* Locking in OSD::schedule_recovery_work() in bstore_aio:
The underlying recovery_gen_wq does extensive mutex dance around multiple
std::lists (_queue, to_process, to_finish). This includes memallocs under
coarse-grained lock. See:
+ https://github.com/ceph/ceph/blob/2a87e2cbe5438afa2410e866fccb12978c4c8092/src/common/WorkQueue.h#L236-L240
+ https://github.com/ceph/ceph/blob/2a87e2cbe5438afa2410e866fccb12978c4c8092/src/common/WorkQueue.h#L543-L545
+ https://github.com/ceph/ceph/blob/2a87e2cbe5438afa2410e866fccb12978c4c8092/src/common/WorkQueue.cc#L112-L130
+ https://github.com/ceph/ceph/blob/2a87e2cbe5438afa2410e866fccb12978c4c8092/src/common/WorkQueue.h#L192-L224
@rzarzynski
rzarzynski / gdbpmp_ref_master.txt
Last active January 9, 2018 15:27
Profiling unbuffered 4 KiB randreads on incerta
Gathering Samples........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Profil
@rzarzynski
rzarzynski / reference_43b88419d4f87da514721848d28fe949cf0ecb9e.txt
Last active January 5, 2018 00:57
wip-bs-async-read-rebased: profilling of the sync read path with buffering disabled
4 KiB randread over 1 GiB RBD volume provisioned with 4 MiB chunks
rbd_iodepth32: (groupid=0, jobs=1): err= 0: pid=13417: Fri Jan 5 01:18:41 2018
read: IOPS=22.9k, BW=89.4MiB/s (93.8MB/s)(5366MiB/60003msec)
slat (nsec): min=975, max=1999.6k, avg=7852.04, stdev=29403.40
clat (usec): min=92, max=239443, avg=1388.82, stdev=1764.49
lat (usec): min=201, max=239536, avg=1396.67, stdev=1764.73
clat percentiles (usec):
| 1.00th=[ 433], 5.00th=[ 611], 10.00th=[ 725], 20.00th=[ 881],
| 30.00th=[ 1004], 40.00th=[ 1106], 50.00th=[ 1221], 60.00th=[ 1336],
@rzarzynski
rzarzynski / reference_43b88419d4f87da514721848d28fe949cf0ecb9e.txt
Last active January 4, 2018 22:52
wip-bs-async-read-rebased: profilling of the sync read path
4 KiB randread over 1 GiB RBD volume provisioned with 4 MiB chunks
rbd_iodepth32: (groupid=0, jobs=1): err= 0: pid=11881: Thu Jan 4 23:25:27 2018
read: IOPS=25.3k, BW=98.0MiB/s (104MB/s)(5939MiB/60002msec)
slat (nsec): min=1000, max=7019.8k, avg=7565.57, stdev=28274.55
clat (usec): min=31, max=99654, avg=1254.29, stdev=778.99
lat (usec): min=128, max=99655, avg=1261.86, stdev=778.78
clat percentiles (usec):
| 1.00th=[ 351], 5.00th=[ 537], 10.00th=[ 660], 20.00th=[ 824],
| 30.00th=[ 947], 40.00th=[ 1045], 50.00th=[ 1139], 60.00th=[ 1254],
@rzarzynski
rzarzynski / after.txt
Created October 18, 2017 14:30
Testing boost::container::flat_map for BlueStore's coll_map
Samples: 702K of event 'cycles:p', Event count (approx.): 240519119891
Children Self Command Shared Object Symbol ◆
+ 62,05% 0,00% fio libc-2.23.so [.] __clone ▒
+ 62,05% 0,00% fio libpthread-2.23.so [.] start_thread ▒
+ 62,03% 0,22% fio fio [.] thread_main ▒
+ 52,81% 0,19% fio fio [.] td_io_queue ▒
+ 52,50% 0,38% fio libfio_ceph_objectstore.so [.] (anonymous namespace)::fio_ceph_os_queu▒
- 42,50% 0,17% fio libfio_ceph_objectstore.so [.] BlueStore::queue_transactions ▒
- 42,33% BlueStore::queue_transactions ▒
- 29,66% BlueStore::_txc_add_transaction
@rzarzynski
rzarzynski / with_92a28f033a5272b7dc2c5d726e67b6d09f6166ba_rocksdb_enable_rmrange=true.txt
Last active October 12, 2017 14:33
Performance tests of commit 92a28f033a5272b7dc2c5d726e67b6d09f6166ba and the rocksdb_enable_rmrange configurable
$ /work/fio/fio /work/fio/examples/rbd.fio
rbd_iodepth32: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=32
fio-3.0-32-g83070
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=16.6MiB/s][r=0,w=4248 IOPS][eta 00m:00s]
rbd_iodepth32: (groupid=0, jobs=1): err= 0: pid=14177: Thu Oct 12 14:54:48 2017
write: IOPS=4211, BW=16.5MiB/s (17.3MB/s)(4096MiB/248960msec)
slat (nsec): min=637, max=1264.4k, avg=2790.40, stdev=7299.46
clat (usec): min=768, max=194612, avg=7593.42, stdev=3468.52
lat (usec): min=772, max=194614, avg=7596.21, stdev=3468.28
@rzarzynski
rzarzynski / with.txt
Created October 11, 2017 14:06
Performance tests of commit 92a28f033a5272b7dc2c5d726e67b6d09f6166ba
$ /work/fio/fio /work/fio/examples/rbd.fio
rbd_iodepth32: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=32
fio-3.0-32-g83070
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=17.4MiB/s][r=0,w=4467 IOPS][eta 00m:00s]
rbd_iodepth32: (groupid=0, jobs=1): err= 0: pid=28369: Wed Oct 11 15:16:59 2017
write: IOPS=4321, BW=16.9MiB/s (17.7MB/s)(4096MiB/242614msec)
slat (nsec): min=626, max=1643.6k, avg=2736.83, stdev=6738.13
clat (usec): min=692, max=254925, avg=7399.87, stdev=3855.18
lat (usec): min=694, max=254927, avg=7402.61, stdev=3854.96