Benchmark for https://github.com/facebook/rocksdb/pull/5431

gistfile1.txt

-------------------------------------------------------------------------------------------------------------------------
Benchmark - seekrandom

Each experiment is run thrice.

Database size: 100M keys

--------Buffered IO--------

----Branch - Master (5795617fc45274c0b11dc1dbf3e8018e500ae72e)----

Fill database:
TEST_TMPDIR=/data/users/$USER/benchmarks/master ./db_bench -benchmarks=fillrandom -num=1000000000 -compression_type="none" -level_compaction_dynamic_level_bytes

Page cache cleared before each run using: sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"

Seek random:
TEST_TMPDIR=/data/users/$USER/benchmarks/master/ ./db_bench -benchmarks=seekrandom -num=100000000 -duration=60 -num=100000000 -use_existing_db -seek_nexts=10000 -readahead_size=524288 -statistics -histogram 

1. seekrandom : 29130.207 micros/op 34 ops/sec;   38.0 MB/s (2452 of 2999 found)
Microseconds per seek:
Count: 2999 Average: 29130.2614  StdDev: 3089.92
Min: 20296  Median: 28007.5064  Max: 47444
Percentiles: P50: 28007.51 P75: 31027.37 P99: 47444.00 P99.9: 47444.00 P99.99: 47444.00

2. seekrandom : 30997.611 micros/op 32 ops/sec;   35.7 MB/s (1640 of 1999 found)
Microseconds per seek:
Count: 1999 Average: 30997.6858  StdDev: 7003.97
Min: 20932  Median: 29018.2109  Max: 173156
Percentiles: P50: 29018.21 P75: 32530.83 P99: 49837.34 P99.9: 110060.00 P99.99: 173156.00 

3. seekrandom : 30193.831 micros/op 33 ops/sec;   36.6 MB/s (1640 of 1999 found)
Microseconds per seek:
Count: 1999 Average: 30193.9390  StdDev: 3642.81
Min: 21437  Median: 28590.0901  Max: 61031
Percentiles: P50: 28590.09 P75: 31891.74 P99: 49024.68 P99.9: 49948.69 P99.99: 61031.00


----Branch - PR 5431----

Fill database:
TEST_TMPDIR=/data/users/$USER/benchmarks/feature ./db_bench -benchmarks=fillrandom -num=1000000000 -compression_type="none" -level_compaction_dynamic_level_bytes

Page cache cleared before each run using: sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"

Seek random:
TEST_TMPDIR=/data/users/$USER/benchmarks/feature ./db_bench -benchmarks=seekrandom -num=100000000 -duration=60 -num=100000000 -use_existing_db -seek_nexts=10000 -readahead_size=524288 -statistics -histogram 

1. seekrandom : 29218.841 micros/op 34 ops/sec;   37.9 MB/s (2452 of 2999 found)
Microseconds per seek:
Count: 2999 Average: 29218.8990  StdDev: 3669.61
Min: 19983  Median: 28138.3377  Max: 85521
Percentiles: P50: 28138.34 P75: 31226.04 P99: 48697.03 P99.9: 65005.00 P99.99: 85521.00

2. seekrandom : 30022.525 micros/op 33 ops/sec;   36.8 MB/s (1640 of 1999 found)
Microseconds per seek:
Count: 1999 Average: 30022.6228  StdDev: 3389.50
Min: 20393  Median: 28427.4383  Max: 52375
Percentiles: P50: 28427.44 P75: 31657.31 P99: 48890.62 P99.9: 49941.64 P99.99: 52375.00

3. seekrandom : 31027.565 micros/op 32 ops/sec;   35.7 MB/s (1640 of 1999 found)
Microseconds per seek:
Count: 1999 Average: 31027.6973  StdDev: 4521.39
Min: 20825  Median: 29297.9361  Max: 76376
Percentiles: P50: 29297.94 P75: 32957.89 P99: 49478.88 P99.9: 68756.25 P99.99: 76376.00

------------------------------------------------------

--------Direct IO--------

TEST_TMPDIR=/data/users/$USER/benchmarks/master/ ./db_bench -use_direct_reads -benchmarks=seekrandom -num=100000000 -duration=60 -num=100000000 -use_existing_db -seek_nexts=10000 -readahead_size=524288 -statistics -histogram

----Branch - Master (5795617fc45274c0b11dc1dbf3e8018e500ae72e)----

1. seekrandom : 34863.928 micros/op 28 ops/sec;   31.7 MB/s (1640 of 1999 found)
Microseconds per seek:
Count: 1999 Average: 34864.0200  StdDev: 10501.50
Min: 25187  Median: 33634.9602  Max: 247134
Percentiles: P50: 33634.96 P75: 42096.86 P99: 69132.35 P99.9: 210020.00 P99.99: 246002.00

2. seekrandom : 34457.468 micros/op 29 ops/sec;   32.1 MB/s (1640 of 1999 found)
Microseconds per seek:
Count: 1999 Average: 34457.5638  StdDev: 7898.50
Min: 24637  Median: 32602.2179  Max: 216985
Percentiles: P50: 32602.22 P75: 41431.60 P99: 62512.50 P99.9: 150020.00 P99.99: 216985.00

3. seekrandom : 33701.524 micros/op 29 ops/sec;   32.8 MB/s (1640 of 1999 found)
Microseconds per seek:
Count: 1999 Average: 33701.5968  StdDev: 8480.60
Min: 24347  Median: 30931.3566  Max: 173838
Percentiles: P50: 30931.36 P75: 39246.92 P99: 66675.93 P99.9: 160010.00 P99.99: 173838.00

----Branch - PR 5431----
TEST_TMPDIR=/data/users/$USER/benchmarks/feature/ ./db_bench -use_direct_reads -benchmarks=seekrandom -num=100000000 -duration=60 -num=100000000 -use_existing_db -seek_nexts=10000 -readahead_size=524288 -statistics -histogram

1. seekrandom : 32208.814 micros/op 31 ops/sec;   34.3 MB/s (1640 of 1999 found)
Microseconds per seek:
Count: 1999 Average: 32208.9290  StdDev: 3156.53
Min: 24924  Median: 29961.2600  Max: 51803
Percentiles: P50: 29961.26 P75: 36268.70 P99: 49530.36 P99.9: 51803.00 P99.99: 51803.00

2. seekrandom : 31700.035 micros/op 31 ops/sec;   34.9 MB/s (1640 of 1999 found)
Microseconds per seek:
Count: 1999 Average: 31700.0875  StdDev: 3086.57
Min: 25139  Median: 29359.1031  Max: 51906
Percentiles: P50: 29359.10 P75: 33177.44 P99: 49391.99 P99.9: 50012.50 P99.99: 51906.00

3. seekrandom : 32534.459 micros/op 30 ops/sec;   34.0 MB/s (1640 of 1999 found)
Microseconds per seek:
Count: 1999 Average: 32534.5393  StdDev: 3574.78
Min: 23796  Median: 30304.0030  Max: 58120
Percentiles: P50: 30304.00 P75: 37459.96 P99: 49669.42 P99.9: 58120.00 P99.99: 58120.00

-------------------------------------------------------------------------------------------------------------------------

Benchmark - readrandomwriterandom (mainly to test compaction performance)

Database size: 10M keys

Approx 530 compactions

--------Buffered IO--------

----Branch - Master----

Command (changed read/write ratio from 90% to 50%):
TEST_TMPDIR=/dev/shm/benchmarks/master/ ./db_bench -benchmarks=readrandomwriterandom -num=10000000 -value_size=1024 -write_buffer_size=1048576 -compaction_readahead_size=2097152 -statistics -histogram 

readrandomwriterandom : 846.934 micros/op 1180 ops/sec; ( reads:5000000 writes:5000000 total:10000000 found:1064601)
Microseconds per read:
Count: 5000000 Average: 687.6952  StdDev: 382.63
Min: 18  Median: 650.0904  Max: 31386
Percentiles: P50: 650.09 P75: 966.36 P99: 1813.11 P99.9: 2633.27 P99.99: 6587.46


Microseconds per write:
Count: 5000000 Average: 1006.1731  StdDev: 684.73
Min: 19  Median: 49.8037  Max: 53203
Percentiles: P50: 49.80 P75: 70.88 P99: 19940.77 P99.9: 26057.62 P99.99: 32325.82

----Branch - PR 5431----

Command (changed read/write ratio from 90% to 50%):
TEST_TMPDIR=/dev/shm/benchmarks/feature/ ./db_bench -benchmarks=readrandomwriterandom -num=10000000 -value_size=1024 -write_buffer_size=1048576 -compaction_readahead_size=2097152 -statistics -histogram

1. readrandomwriterandom : 822.289 micros/op 1216 ops/sec; ( reads:5000000 writes:5000000 total:10000000 found:1064601)
Microseconds per read:
Count: 5000000 Average: 684.8000  StdDev: 390.97
Min: 16  Median: 645.9187  Max: 47676
Percentiles: P50: 645.92 P75: 961.99 P99: 1811.89 P99.9: 2661.58 P99.99: 8518.89

Microseconds per write:
Count: 5000000 Average: 959.7788  StdDev: 460.62
Min: 20  Median: 49.7593  Max: 44814
Percentiles: P50: 49.76 P75: 70.85 P99: 19592.31 P99.9: 25727.50 P99.99: 32297.72

------------------------------------------------------

--------Direct IO--------

----Branch - Master----

Command (changed read/write ratio from 90% to 50%):
TEST_TMPDIR=/data/users/vijaynadimpalli/benchmarks/master/ ./db_bench -benchmarks=readrandomwriterandom -num=10000000 -use_direct_reads -use_direct_io_for_flush_and_compaction -level_compaction_dynamic_level_bytes -value_size=1024 -write_buffer_size=1048576 -compaction_readahead_size=2097152 -statistics -histogram

Approx 1200 compactions

1.readrandomwriterandom : 1108.635 micros/op 902 ops/sec; ( reads:5000000 writes:5000000 total:10000000 found:1064601)
Microseconds per read:
Count: 5000000 Average: 2151.7394  StdDev: 750.50
Min: 17  Median: 1787.4503  Max: 268090
Percentiles: P50: 1787.45 P75: 2775.31 P99: 6552.71 P99.9: 13668.34 P99.99: 54444.44

Microseconds per write:
Count: 5000000 Average: 65.5296  StdDev: 61.00
Min: 24  Median: 60.4603  Max: 37400
Percentiles: P50: 60.46 P75: 73.11 P99: 167.73 P99.9: 734.41 P99.99: 1262.36

----Branch - PR 5431----

Command (changed read/write ratio from 90% to 50%):
TEST_TMPDIR=/data/users/vijaynadimpalli/benchmarks/feature/ ./db_bench -benchmarks=readrandomwriterandom -num=10000000 -use_direct_reads -use_direct_io_for_flush_and_compaction -level_compaction_dynamic_level_bytes -value_size=1024 -write_buffer_size=1048576 -compaction_readahead_size=2097152 -statistics -histogram

Approx 1200 compactions

1. readrandomwriterandom : 1092.494 micros/op 915 ops/sec; ( reads:5000000 writes:5000000 total:10000000 found:1064601)
Microseconds per read:
Count: 5000000 Average: 2121.5550  StdDev: 490.22
Min: 15  Median: 1755.6443  Max: 254067
Percentiles: P50: 1755.64 P75: 2725.14 P99: 6529.94 P99.9: 15125.08 P99.99: 67882.35


Microseconds per write:
Count: 5000000 Average: 63.4339  StdDev: 245.90
Min: 23  Median: 59.4693  Max: 391005
Percentiles: P50: 59.47 P75: 71.34 P99: 163.96 P99.9: 766.09 P99.99: 1278.79

strace

Ran strace -e pread64 -f -T -t ./db_compaction_test --gtest_filter=DBCompactionTest.PartialManualCompaction command to verify pread64 syscalls haven't changed before and after code change. 

The strace below is from master:

[pid 1444170] 15:59:08 pread64(9, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000084>
[pid 1444170] 15:59:08 pread64(9, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000074>
[pid 1444170] 15:59:08 pread64(9, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000094>
[pid 1444170] 15:59:08 pread64(9, "\0\v\3foo\2\7\0\0\0\0\0\0\0\270 \0\v\4foo\2\3\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000087>
[pid 1444170] 15:59:08 pread64(11, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000088>
[pid 1444170] 15:59:08 pread64(11, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000089>
[pid 1444170] 15:59:08 pread64(11, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000086>
[pid 1444170] 15:59:08 pread64(11, "\0\v\3foo\2\21\0\0\0\0\0\0\0\270 \0\v\4foo\2\r\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000103>
[pid 1444170] 15:59:08 pread64(12, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000070>
[pid 1444170] 15:59:08 pread64(12, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000069>
[pid 1444170] 15:59:08 pread64(12, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000059>
[pid 1444170] 15:59:08 pread64(12, "\0\v\3foo\2\33\0\0\0\0\0\0\0\270 \0\v\4foo\2\27\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000060>
[pid 1444170] 15:59:08 pread64(13, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000086>
[pid 1444170] 15:59:08 pread64(13, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000087>
[pid 1444170] 15:59:08 pread64(13, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000069>
[pid 1444170] 15:59:08 pread64(13, "\0\v\3foo\2%\0\0\0\0\0\0\0\270 \0\v\4foo\2!\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000085>
[pid 1444170] 15:59:08 pread64(14, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000068>
[pid 1444170] 15:59:08 pread64(14, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000065>
[pid 1444170] 15:59:08 pread64(14, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000076>
[pid 1444170] 15:59:08 pread64(14, "\0\v\3foo\2/\0\0\0\0\0\0\0\270 \0\v\4foo\2+\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000066>
[pid 1444170] 15:59:08 pread64(15, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000069>
[pid 1444170] 15:59:08 pread64(15, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000050>
[pid 1444170] 15:59:08 pread64(15, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000064>
[pid 1444170] 15:59:08 pread64(15, "\0\v\3foo\0029\0\0\0\0\0\0\0\270 \0\v\4foo\0025\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000067>
[pid 1444170] 15:59:09 pread64(16, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000097>
[pid 1444170] 15:59:09 pread64(16, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000079>
[pid 1444170] 15:59:09 pread64(16, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000067>
[pid 1444170] 15:59:09 pread64(16, "\0\v\3foo\2C\0\0\0\0\0\0\0\270 \0\v\4foo\2?\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000095>
[pid 1444170] 15:59:09 pread64(17, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000059>
[pid 1444170] 15:59:09 pread64(17, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000051>
[pid 1444170] 15:59:09 pread64(17, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000048>
[pid 1444170] 15:59:09 pread64(17, "\0\v\3foo\2M\0\0\0\0\0\0\0\270 \0\v\4foo\2I\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000051>
[pid 1444169] 15:59:09 pread64(17, "\0\v\200\10foo\2P\0\0\0\0\0\0)U?MSg_)j(roFn($e"..., 4157, 0) = 4157 <0.000084>
[pid 1444169] 15:59:09 pread64(17, "\0\v\200\10foo\2L\0\0\0\0\0\0GSVZDFD&|!YtEo~YO"..., 4157, 4157) = 4157 <0.000079>
[pid 1444169] 15:59:09 pread64(17, "\0\v\200\10foo\2H\0\0\0\0\0\0t.M<x#,jM}@?e~Lr`"..., 2087, 8314) = 2087 <0.000091>
[pid 1444169] 15:59:09 pread64(16, "\0\v\200\10foo\2F\0\0\0\0\0\0k[h3%.OPH_^:\\S7T&"..., 4157, 0) = 4157 <0.000080>
[pid 1444169] 15:59:09 pread64(16, "\0\v\200\10foo\2B\0\0\0\0\0\0w+xmRH72CGk',bS~S"..., 4157, 4157) = 4157 <0.000111>
[pid 1444169] 15:59:09 pread64(16, "\0\v\200\10foo\2>\0\0\0\0\0\0TG{8?`X{Nyx\\R:XhD"..., 2087, 8314) = 2087 <0.000075>
[pid 1444169] 15:59:09 pread64(15, "\0\v\200\10foo\2<\0\0\0\0\0\0+qToi_c{*@S+4:N(:"..., 4157, 0) = 4157 <0.000094>
[pid 1444169] 15:59:09 pread64(15, "\0\v\200\10foo\0028\0\0\0\0\0\0%A;s[f<U5ai0I+*CH"..., 4157, 4157) = 4157 <0.000077>
[pid 1444169] 15:59:09 pread64(15, "\0\v\200\10foo\0024\0\0\0\0\0\0Wb/emt0Ud$_e~'GXg"..., 2087, 8314) = 2087 <0.000074>
[pid 1444169] 15:59:09 pread64(14, "\0\v\200\10foo\0022\0\0\0\0\0\0%hw%OMa\"}9I609Q!B"..., 4157, 0) = 4157 <0.000063>
[pid 1444169] 15:59:09 pread64(14, "\0\v\200\10foo\2.\0\0\0\0\0\0004c,0\\}2viJ$9M=-:G"..., 4157, 4157) = 4157 <0.000062>
[pid 1444169] 15:59:09 pread64(14, "\0\v\200\10foo\2*\0\0\0\0\0\0PM=N_;)k7%eW c%oO"..., 2087, 8314) = 2087 <0.000058>
[pid 1444169] 15:59:09 pread64(13, "\0\v\200\10foo\2(\0\0\0\0\0\0p}Y&mu^DcaSGb2&nP"..., 4157, 0) = 4157 <0.000061>
[pid 1444169] 15:59:09 pread64(13, "\0\v\200\10foo\2$\0\0\0\0\0\0+vv-$+kdphJ8bFJ/{"..., 4157, 4157) = 4157 <0.000059>
[pid 1444169] 15:59:09 pread64(13, "\0\v\200\10foo\2 \0\0\0\0\0\0ui1WS:iR*8hT3}xYD"..., 2087, 8314) = 2087 <0.000060>
[pid 1444169] 15:59:09 pread64(12, "\0\v\200\10foo\2\36\0\0\0\0\0\0YIyW#]oSs^6VHfB<`"..., 4157, 0) = 4157 <0.000064>
[pid 1444169] 15:59:09 pread64(12, "\0\v\200\10foo\2\32\0\0\0\0\0\0wk0<rmcI$L'6e&m?7"..., 4157, 4157) = 4157 <0.000064>
[pid 1444169] 15:59:09 pread64(12, "\0\v\200\10foo\2\26\0\0\0\0\0\0:i9egIwNR+lHlT^mn"..., 2087, 8314) = 2087 <0.000058>
[pid 1444169] 15:59:09 pread64(11, "\0\v\200\10foo\2\24\0\0\0\0\0\0mfF8Jel/*Zf :-#s("..., 4157, 0) = 4157 <0.000074>
[pid 1444169] 15:59:09 pread64(11, "\0\v\200\10foo\2\20\0\0\0\0\0\0yny:8n/=F+eT8u$Iw"..., 4157, 4157) = 4157 <0.000072>
[pid 1444169] 15:59:09 pread64(11, "\0\v\200\10foo\2\f\0\0\0\0\0\0de!]RDhvbD'~tM_^<"..., 2087, 8314) = 2087 <0.000071>
[pid 1444169] 15:59:09 pread64(9, "\0\v\200\10foo\2\n\0\0\0\0\0\0\\X'cjiHX)D,RSj1X!"..., 4157, 0) = 4157 <0.000086>
[pid 1444169] 15:59:09 pread64(9, "\0\v\200\10foo\2\6\0\0\0\0\0\0Veqm7~'1]W0q-q4*#"..., 4157, 4157) = 4157 <0.000088>
[pid 1444169] 15:59:09 pread64(9, "\0\v\200\10foo\2\2\0\0\0\0\0\0007}`ivbAy:0_VWtW+c"..., 2087, 8314) = 2087 <0.000084>
[pid 1444169] 15:59:09 pread64(8, "\1\315\5 \36\30\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 754) = 53 <0.000063>
[pid 1444169] 15:59:09 pread64(8, "\0\22\3rocksdb.properties;\215\5\0\0\0\0\1\0\0\0"..., 37, 717) = 37 <0.000070>
[pid 1444169] 15:59:09 pread64(8, "\0$\4rocksdb.block.based.table.ind"..., 658, 59) = 658 <0.000059>
[pid 1444169] 15:59:09 pread64(8, "\0\v\2foo\1\0\0\0\0\0\0\0\0\31\0\0\0\0\1\0\0\0\0\212\216\222P", 29, 30) = 29 <0.000065>

The strace below is from the PR:

[pid 1501089] 16:03:07 pread64(9, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000056>
[pid 1501089] 16:03:07 pread64(9, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000062>
[pid 1501089] 16:03:07 pread64(9, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000072>
[pid 1501089] 16:03:07 pread64(9, "\0\v\3foo\2\7\0\0\0\0\0\0\0\270 \0\v\4foo\2\3\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000095>
[pid 1501089] 16:03:07 pread64(11, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000090>
[pid 1501089] 16:03:07 pread64(11, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000078>
[pid 1501089] 16:03:07 pread64(11, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000069>
[pid 1501089] 16:03:07 pread64(11, "\0\v\3foo\2\21\0\0\0\0\0\0\0\270 \0\v\4foo\2\r\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000088>
[pid 1501089] 16:03:07 pread64(12, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000120>
[pid 1501089] 16:03:07 pread64(12, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000085>
[pid 1501089] 16:03:07 pread64(12, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000065>
[pid 1501089] 16:03:07 pread64(12, "\0\v\3foo\2\33\0\0\0\0\0\0\0\270 \0\v\4foo\2\27\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000067>
[pid 1501089] 16:03:07 pread64(13, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000090>
[pid 1501089] 16:03:07 pread64(13, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000070>
[pid 1501089] 16:03:07 pread64(13, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000084>
[pid 1501089] 16:03:07 pread64(13, "\0\v\3foo\2%\0\0\0\0\0\0\0\270 \0\v\4foo\2!\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000090>
[pid 1501089] 16:03:07 pread64(14, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000094>
[pid 1501089] 16:03:07 pread64(14, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000055>
[pid 1501089] 16:03:07 pread64(14, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000088>
[pid 1501089] 16:03:07 pread64(14, "\0\v\3foo\2/\0\0\0\0\0\0\0\270 \0\v\4foo\2+\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000084>
[pid 1501089] 16:03:07 pread64(15, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000093>
[pid 1501089] 16:03:07 pread64(15, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000079>
[pid 1501089] 16:03:07 pread64(15, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000065>
[pid 1501089] 16:03:07 pread64(15, "\0\v\3foo\0029\0\0\0\0\0\0\0\270 \0\v\4foo\0025\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000084>
[pid 1501089] 16:03:07 pread64(16, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000083>
[pid 1501089] 16:03:07 pread64(16, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000043>
[pid 1501089] 16:03:07 pread64(16, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000063>
[pid 1501089] 16:03:07 pread64(16, "\0\v\3foo\2C\0\0\0\0\0\0\0\270 \0\v\4foo\2?\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000055>
[pid 1501089] 16:03:07 pread64(17, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000075>
[pid 1501089] 16:03:07 pread64(17, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000070>
[pid 1501089] 16:03:07 pread64(17, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000078>
[pid 1501089] 16:03:07 pread64(17, "\0\v\3foo\2M\0\0\0\0\0\0\0\270 \0\v\4foo\2I\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000062>
[pid 1501088] 16:03:07 pread64(17, "\0\v\200\10foo\2P\0\0\0\0\0\0)U?MSg_)j(roFn($e"..., 4157, 0) = 4157 <0.000082>
[pid 1501088] 16:03:07 pread64(17, "\0\v\200\10foo\2L\0\0\0\0\0\0GSVZDFD&|!YtEo~YO"..., 4157, 4157) = 4157 <0.000070>
[pid 1501088] 16:03:07 pread64(17, "\0\v\200\10foo\2H\0\0\0\0\0\0t.M<x#,jM}@?e~Lr`"..., 2087, 8314) = 2087 <0.000128>
[pid 1501088] 16:03:07 pread64(16, "\0\v\200\10foo\2F\0\0\0\0\0\0k[h3%.OPH_^:\\S7T&"..., 4157, 0) = 4157 <0.000089>
[pid 1501088] 16:03:07 pread64(16, "\0\v\200\10foo\2B\0\0\0\0\0\0w+xmRH72CGk',bS~S"..., 4157, 4157) = 4157 <0.000089>
[pid 1501088] 16:03:07 pread64(16, "\0\v\200\10foo\2>\0\0\0\0\0\0TG{8?`X{Nyx\\R:XhD"..., 2087, 8314) = 2087 <0.000061>
[pid 1501088] 16:03:07 pread64(15, "\0\v\200\10foo\2<\0\0\0\0\0\0+qToi_c{*@S+4:N(:"..., 4157, 0) = 4157 <0.000260>
[pid 1501088] 16:03:07 pread64(15, "\0\v\200\10foo\0028\0\0\0\0\0\0%A;s[f<U5ai0I+*CH"..., 4157, 4157) = 4157 <0.000092>
[pid 1501088] 16:03:07 pread64(15, "\0\v\200\10foo\0024\0\0\0\0\0\0Wb/emt0Ud$_e~'GXg"..., 2087, 8314) = 2087 <0.000096>
[pid 1501088] 16:03:07 pread64(14, "\0\v\200\10foo\0022\0\0\0\0\0\0%hw%OMa\"}9I609Q!B"..., 4157, 0) = 4157 <0.000086>
[pid 1501088] 16:03:07 pread64(14, "\0\v\200\10foo\2.\0\0\0\0\0\0004c,0\\}2viJ$9M=-:G"..., 4157, 4157) = 4157 <0.000094>
[pid 1501088] 16:03:07 pread64(14, "\0\v\200\10foo\2*\0\0\0\0\0\0PM=N_;)k7%eW c%oO"..., 2087, 8314) = 2087 <0.000088>
[pid 1501088] 16:03:07 pread64(13, "\0\v\200\10foo\2(\0\0\0\0\0\0p}Y&mu^DcaSGb2&nP"..., 4157, 0) = 4157 <0.000094>
[pid 1501088] 16:03:07 pread64(13, "\0\v\200\10foo\2$\0\0\0\0\0\0+vv-$+kdphJ8bFJ/{"..., 4157, 4157) = 4157 <0.000082>
[pid 1501088] 16:03:07 pread64(13, "\0\v\200\10foo\2 \0\0\0\0\0\0ui1WS:iR*8hT3}xYD"..., 2087, 8314) = 2087 <0.000083>
[pid 1501088] 16:03:07 pread64(12, "\0\v\200\10foo\2\36\0\0\0\0\0\0YIyW#]oSs^6VHfB<`"..., 4157, 0) = 4157 <0.000075>
[pid 1501088] 16:03:07 pread64(12, "\0\v\200\10foo\2\32\0\0\0\0\0\0wk0<rmcI$L'6e&m?7"..., 4157, 4157) = 4157 <0.000181>
[pid 1501088] 16:03:07 pread64(12, "\0\v\200\10foo\2\26\0\0\0\0\0\0:i9egIwNR+lHlT^mn"..., 2087, 8314) = 2087 <0.000069>
[pid 1501088] 16:03:07 pread64(11, "\0\v\200\10foo\2\24\0\0\0\0\0\0mfF8Jel/*Zf :-#s("..., 4157, 0) = 4157 <0.000069>
[pid 1501088] 16:03:07 pread64(11, "\0\v\200\10foo\2\20\0\0\0\0\0\0yny:8n/=F+eT8u$Iw"..., 4157, 4157) = 4157 <0.000081>
[pid 1501088] 16:03:07 pread64(11, "\0\v\200\10foo\2\f\0\0\0\0\0\0de!]RDhvbD'~tM_^<"..., 2087, 8314) = 2087 <0.000072>
[pid 1501088] 16:03:07 pread64(9, "\0\v\200\10foo\2\n\0\0\0\0\0\0\\X'cjiHX)D,RSj1X!"..., 4157, 0) = 4157 <0.000074>
[pid 1501088] 16:03:07 pread64(9, "\0\v\200\10foo\2\6\0\0\0\0\0\0Veqm7~'1]W0q-q4*#"..., 4157, 4157) = 4157 <0.000069>
[pid 1501088] 16:03:07 pread64(9, "\0\v\200\10foo\2\2\0\0\0\0\0\0007}`ivbAy:0_VWtW+c"..., 2087, 8314) = 2087 <0.000074>
[pid 1501088] 16:03:07 pread64(8, "\1\315\5 \36\30\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 754) = 53 <0.000093>
[pid 1501088] 16:03:07 pread64(8, "\0\22\3rocksdb.properties;\215\5\0\0\0\0\1\0\0\0"..., 37, 717) = 37 <0.000068>
[pid 1501088] 16:03:07 pread64(8, "\0$\4rocksdb.block.based.table.ind"..., 658, 59) = 658 <0.000131>
[pid 1501088] 16:03:07 pread64(8, "\0\v\2foo\1\0\0\0\0\0\0\0\0\31\0\0\0\0\1\0\0\0\0\212\216\222P", 29, 30) = 29 <0.000082>

The P99.99 latencies vary wildly and it's attributed mostly to other background tasks/io that might be going on in the devserver. Overall ops/sec and micros/op show no regression.

Used strace -e pread64 -f -T -t ./db_compaction_test --gtest_filter=DBCompactionTest.PartialManualCompaction to verify the pread64 calls haven't changed before and after the code change.