RocksDB Iterator explicit-user-initiated-readahead improvement experiments

gistfile1.txt

Experiments done for: https://github.com/facebook/rocksdb/pull/5246

Each experiment run thrice. 3 runs.

Master at commit: 1c8cbf315

Database size: 1B keys
Fill database:
TEST_TMPDIR=/data/users/$USER/benchmarks/iter ./db_bench -benchmarks=fillrandom -num=1000000000 -compression_type="none" -level_compaction_dynamic_level_bytes


------------
Buffered IO:
------------
ReadOptions.readahead_size: 1MB

Page cache cleared before each run using: sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"

db_bench seekrandom command:
TEST_TMPDIR=/data/users/$USER/benchmarks/iter ./db_bench -benchmarks=seekrandom -duration=60 -num=1000000000 -use_existing_db -seek_nexts=10000 -readahead_size=1048576 -statistics -histogram

Master (1c8cbf315)

| Micros/op | ops/sec | MB/s  | File Opens | File Closes |
| :-------: | :-----: | :--:  | :--------: | :---------: |
| 10462.200 | 95      | 105.7 | 31198      | 29708       |
| 10217.777 | 97      | 108.3 | 31198      | 29708       |
| 10624.633 | 94      | 104.1 | 31198      | 29708       |

Avg MB/s: 106.03

By depending on Page Cache Prefetch (ie. using readahead posix syscall)

| Micros/op | ops/sec | MB/s  | File Opens | File Closes |
| :-------: | :-----: | :--:  | :--------: | :---------: |
| 6675.437  | 149     | 165.7 | 1490       | 0           |
| 6732.621  | 148     | 164.3 | 1490       | 0           |
| 6544.479  | 152     | 169   | 1490       | 0           |

Avg MB/s: 166.33
Improvement over master: 56.8%

With the changes in PR 5246 (By using our own buffer (FilePrefetchBuffer) instead of dependin on page cache):

| Micros/op | ops/sec | MB/s  | File Opens | File Closes |
| :-------: | :-----: | :--:  | :--------: | :---------: |
| 8067.641  | 123     | 137.1 | 1490       | 0           |
| 8211.664  | 121     | 134.7 | 1490       | 0           |
| 8080.985  | 123     | 136.9 | 1490       | 0           |

Avg MB/s: 136.23
Improvement over master: 28.48%

----
ReadOptions.readahead_size: 512KB

db_bench seekrandom command:
TEST_TMPDIR=/data/users/$USER/benchmarks/iter ./db_bench -benchmarks=seekrandom -duration=60 -num=1000000000 -use_existing_db -seek_nexts=10000 -readahead_size=524288 -statistics -histogram

Master (1c8cbf315)

| Micros/op | ops/sec | MB/s  | 
| :-------: | :-----: | :--:  | 
| 8999.127  | 111     | 122.9 | 
| 8870.461  | 111     | 124.7 | 
| 8971.241  | 111     | 123.3 | 

Avg MB/s: 123.63

With PR 5246:

| Micros/op | ops/sec | MB/s  |
| :-------: | :-----: | :--:  |
| 6894.400  | 145     | 160.5 |
| 6979.636  | 143     | 158.5 |
| 6836.007  | 146     | 161.8 |

Avg MB/s: 160.26
Improvement over master: 29.63%%

------------
Direct IO:
------------
ReadOptions.readahead_size: 1MB

db_bench seekrandom command:
TEST_TMPDIR=/data/users/$USER/benchmarks/iter ./db_bench -benchmarks=seekrandom -use_direct_reads -duration=60 -num=1000000000 -use_existing_db -seek_nexts=10000 -readahead_size=1048576 -statistics -histogram

Master (1c8cbf315)

| Micros/op | ops/sec | MB/s  | File Opens | File Closes |
| :-------: | :-----: | :--:  | :--------: | :---------: |
| 18129.750 | 55      | 61.0  | 21851      | 20353       |
| 17164.220 | 58      | 64.5  | 21316      | 19826       |
| 17560.412 | 56      | 63.0  | 21316      | 19826       |

Avg MB/s: 62.83

With PR 5246:

| Micros/op | ops/sec | MB/s  | File Opens | File Closes |
| :-------: | :-----: | :--:  | :--------: | :---------: |
| 11884.032 | 84      | 93.1  | 1490       | 0           |
| 11666.172 | 85      | 94.8  | 1490       | 0           |
| 11717.853 | 85      | 94.4  | 1490       | 0           |

Avg MB/s: 94.1
Improvement over master: 49.77%

----
ReadOptions.readahead_size: 512KB

db_bench seekrandom command:
TEST_TMPDIR=/data/users/$USER/benchmarks/iter ./db_bench -benchmarks=seekrandom -use_direct_reads -duration=60 -num=1000000000 -use_existing_db -seek_nexts=10000 -readahead_size=524288 -statistics -histogram

Master (1c8cbf315)

| Micros/op | ops/sec | MB/s  | 
| :-------: | :-----: | :--:  | 
| 14590.795 | 68      | 75.8  | 
| 14595.452 | 68      | 75.8  | 
| 14675.948 | 68      | 75.4  | 

Avg MB/s: 75.66

With PR 5246:

| Micros/op | ops/sec | MB/s  |
| :-------: | :-----: | :--:  |
| 8727.319  | 114     | 126.8 |
| 8785.247  | 113     | 125.9 |
| 8703.879  | 114     | 127.1 |

Avg MB/s: 126.6
Improvement over master: 67.33%