gistfile1.txt

1M keys, 4kb values 50% compressible, merge operator was "put"

The "Test" column has the test name, number of threads (t1, t2, t4, t8, t16, t32 for 1,2,4,8,16,32 threads) and s0 means "sync=0".
I don't list results

Notes:
* overwrite and mergerandom have similar ops/sec rates. overwrite is slightly faster and I assume that is because compaction is cheaper
  from not handling merges.

* overwrite and mergerandom reach peak throughput at 1 thread when sync=0.

* for 4kb values updaterandom is slower than overwrite at low concurrency but then slightly faster at high concurrency. This is odd because update random
  does a read with each write. The reads are cheap, from cache, but that is still a lot more work. This looks like RocksDB suffers too much
  from mutex contention during writes and there is more contention in the overwrite test because.

** overwrite has more Stall-time than updaterandom at high concurrency

** overwrite has larger commit groups than updaterandom at high concurrency, check out "writes per batch" below
Cumulative writes: 63M writes, 63M keys, 2948K batches, 21.5 writes per batch, ingest: 242.95 GB, 141.48 MB/s --> updaterandom
Cumulative writes: 57M writes, 57M keys, 2579K batches, 22.1 writes per batch, ingest: 219.15 GB, 126.51 MB/s --> overwrite

** While Stall-time was larger for overwrite than updaterandom at high concurrency, stall counts were larger for updaterandom
Stalls(count): 2658 level0_slowdown, 114 level0_numfiles, 0 memtable_compaction, 0 leveln_slowdown_soft, 132 leveln_slowdown_hard --> updaterandom
Stalls(count): 2401 level0_slowdown, 120 level0_numfiles, 0 memtable_compaction, 0 leveln_slowdown_soft, 119 leveln_slowdown_hard --> overwrite

* for 400b values updaterandom was always slower than overwrite, but for that test the database was much larger than RAM so reads were slower,
  maybe several hundred usecs/read.


ops/sec mb/sec  L0_GB   Sum_GB  W-Amp   W-MB/s  usec/op p50     p75     p99     p99.9   p99.99  Uptime  Stall-time      Stall%  Test
27408   107.6   181.8   882.2   4.8     517.3   36.5    11.8    13.7    184     245     1176    1746    00:12:12.367    42.0    overwrite.t1.s0
28230   110.8   186.7   894.5   4.7     525.5   35.4    38.8    48.5    200     246     1158    1743    00:10:31.412    36.2    overwrite.t2.s0
29169   114.5   192.6   895.5   4.6     524.8   34.3    76.4    205.9   382     546     1189    1747    00:10:31.574    36.2    overwrite.t4.s0
30358   119.2   200.9   902.7   4.4     529.1   32.9    149.2   283.2   715     1271    54456   1747    00:10:36.802    36.4    overwrite.t8.s0
31721   124.5   211.0   917.4   4.3     535.5   31.5    325.4   448.0   1169    1366    485893  1754    00:10:58.957    37.6    overwrite.t16.s0
32206   126.4   216.4   936.3   4.3     540.4   31.1    553.0   693.9   1368    1542    979362  1774    00:10:31.967    35.6    overwrite.t32.s0

ops/sec mb/sec  L0_GB   Sum_GB  W-Amp   W-MB/s  usec/op p50     p75     p99     p99.9   p99.99  Uptime  Stall-time      Stall%  Test
26884   105.5   177.4   843.4   4.7     495.2   37.2    12.7    13.9    190     245     1081    1744    00:11:48.811    40.6    mergerandom.t1.s0
26472   103.9   174.8   847.9   4.8     498.1   37.8    42.7    48.7    200     244     1104    1743    00:10:14.410    35.2    mergerandom.t2.s0
28228   110.8   186.3   862.4   4.6     506.3   35.4    71.6    209.2   388     534     1190    1744    00:10:45.767    37.0    mergerandom.t4.s0
29522   115.9   195.4   873.0   4.4     511.7   33.9    144.4   286.4   757     1252    40750   1747    00:11:6.831     38.2    mergerandom.t8.s0
30689   120.5   203.9   884.5   4.3     516.6   32.6    323.1   457.0   1168    1312    556126  1753    00:11:27.429    39.2    mergerandom.t16.s0
30924   121.4   207.8   897.8   4.3     518.2   32.3    547.9   692.8   1369    1549    1032416 1774    00:11:7.363     37.6    mergerandom.t32.s0

ops/sec mb/sec  L0_GB   Sum_GB  W-Amp   W-MB/s  usec/op p50     p75     p99     p99.9   p99.99  Uptime  Stall-time      Stall%  Test
20745   162.9   136.8   894.5   6.5     526.1   48.2    50.4    57.6    87      115     163     1741    00:00:7.865     0.5     updaterandom.t1.s0
27629   216.9   182.5   952.2   5.2     559.7   36.2    61.5    74.0    240     272     300     1742    00:02:56.202    10.1    updaterandom.t2.s0
30317   238.0   200.1   986.4   4.9     579.1   33.0    89.6    119.8   284     299     1273    1744    00:06:56.287    23.9    updaterandom.t4.s0
32806   257.5   217.0   1009.1  4.6     591.4   30.5    158.3   277.6   544     1276    66708   1747    00:08:11.389    28.1    updaterandom.t8.s0
34489   270.8   228.3   1015.5  4.4     594.5   29.0    297.1   430.5   1125    1388    337439  1749    00:08:53.080    30.5    updaterandom.t16.s0
36032   282.9   239.9   1048.8  4.3     610.9   27.8    561.9   741.7   1378    1596    781947  1758    00:08:48.396    30.1    updaterandom.t32.s0

---

2B keys, 400b values 50% compressible. This shows better throughput with concurrency when sync=1

with sync=0

ops/sec mb/sec  L0_GB   Sum_GB  W-Amp   W-MB/s  usec/op p50     p75     p99     p99.9   p99.99  Uptime  Stall-time      Stall%  Test
91417   36.6    130.4   2132.9  16.3    616.4   10.9    5.7     7.0     172     183     232     3543    00:19:56.100    33.8    overwrite.t1.s0
88425   35.4    126.5   2099.0  16.5    606.3   11.3    42.3    50.6    189     234     447     3545    00:03:19.459    5.6     overwrite.t4.s0
91574   36.7    130.9   2146.3  16.3    619.8   10.9    148.1   187.7   340     419     1738    3546    00:07:20.348    12.4    overwrite.t16.s0
95147   38.1    136.4   2149.5  15.7    619.3   10.5    307.7   379.3   589     929     4282    3554    00:08:14.198    13.9    overwrite.t32.s0

ops/sec mb/sec  L0_GB   Sum_GB  W-Amp   W-MB/s  usec/op p50     p75     p99     p99.9   p99.99  Uptime  Stall-time      Stall%  Test
86694   34.7    123.6   2038.8  16.4    589.2   11.5    6.3     7.8     172     185     234     3543    00:18:23.863    31.2    mergerandom.t1.s0
84998   34.0    121.4   2028.8  16.7    586.3   11.8    43.4    51.3    194     240     430     3543    00:04:23.355    7.4     mergerandom.t4.s0
88910   35.6    127.1   2073.9  16.3    598.3   11.2    150.0   199.0   345     428     1484    3549    00:09:32.971    16.1    mergerandom.t16.s0
91949   36.8    131.6   2069.3  15.7    596.0   10.9    322.8   393.4   608     917     4063    3555    00:10:41.759    18.1    mergerandom.t32.s0

ops/sec mb/sec  L0_GB   Sum_GB  W-Amp   W-MB/s  usec/op p50     p75     p99     p99.9   p99.99  Uptime  Stall-time      Stall%  Test
15419   12.4    22.3    364.9   16.3    105.4   64.9    31.2    41.7    221     455     4842    3545    00:00:0.000     0.0     updaterandom.t1.s033839   27.1    48.5    797.9   16.4    230.3   29.6    58.9    124.5   354     1466    125493  3547    00:00:0.000     0.0     updaterandom.t4.s052567   42.1    75.5    1238.8  16.4    356.8   19.0    151.9   212.6   626     8114    261150  3555    00:00:0.000     0.0     updaterandom.t16.s057822   46.3    83.1    1352.1  16.2    388.4   17.3    287.2   364.7   998     109806  340590  3564    00:00:0.000     0.0     updaterandom.t32.s0

with sync=1

ops/sec mb/sec  L0_GB   Sum_GB  W-Amp   W-MB/s  usec/op p50     p75     p99     p99.9   p99.99  Uptime  Stall-time      Stall%  Test10488   4.2     15.1    331.0   21.9    95.6    95.3    79.9    88.2    133     299     5621    3545    00:00:0.000     0.0     overwrite.t1.s118647   7.5     26.7    507.2   18.9    146.3   53.6    195.0   221.8   291     1310    126837  3548    00:00:0.000     0.0     overwrite.t4.s134005   13.6    48.8    905.4   18.5    260.5   29.4    340.3   373.5   496     18241   222528  3559    00:00:0.000     0.0     overwrite.t16.s143757   17.5    63.0    1135.6  18.0    325.7   22.9    482.5   544.2   844     110554  297490  3570    00:00:5.707     0.2     overwrite.t32.s1
ops/sec mb/sec  L0_GB   Sum_GB  W-Amp   W-MB/s  usec/op p50     p75     p99     p99.9   p99.99  Uptime  Stall-time      Stall%  Test
7274    5.8     10.4    170.2   16.3    49.1    137.5   115.2   129.6   293     1379    4720    3546    00:00:0.000     0.0     updaterandom.t1.s114914   11.9    21.3    345.6   16.2    99.6    67.0    226.1   248.5   446     1992    125839  3550    00:00:0.000     0.0     updaterandom.t4.s132666   26.2    46.8    764.2   16.3    219.9   30.6    350.2   383.2   1144    11662   177110  3558    00:00:0.000     0.0     updaterandom.t16.s142505   34.1    61.2    1009.3  16.4    289.6   23.5    494.4   558.5   1572    113731  281662  3568    00:00:0.000     0.0     updaterandom.t32.s1ops/sec mb/sec  L0_GB   Sum_GB  W-Amp   W-MB/s  usec/op p50     p75     p99     p99.9   p99.99  Uptime  Stall-time      Stall%  Test
10445   4.2     15.0    309.0   20.6    89.2    95.7    83.2    89.0    128     231     3633    3544    00:00:0.000     0.0     mergerandom.t1.s1
17024   6.8     24.3    466.5   19.1    134.5   58.7    198.0   223.9   294     1265    128109  3550    00:00:5.079     0.1     mergerandom.t4.s1
33714   13.5    48.3    900.8   18.6    259.1   29.7    341.0   372.8   490     19214   216997  3560    00:00:3.726     0.1     mergerandom.t16.s1
43521   17.4    62.6    1123.7  17.9    322.3   23.0    485.7   548.1   791     111989  298190  3570    00:00:2.574     0.1     mergerandom.t32.s1

---

command lines for 1M keys, 4kb values, 1 thread

./db_bench --benchmarks=overwrite --use_existing_db=1 --sync=0 --db=foobar --wal_dir=/txlogs --disable_data_sync=0 --num=1048576 --num_levels=6 --key_size=20 --value_size=4096 --block_size=32768 --cache_size=34359738368 --cache_numshardbits=6 --compression_type=snappy --min_level_to_compress=3 --compression_ratio=0.5 --level_compaction_dynamic_level_bytes=true --hard_rate_limit=3 --rate_limit_delay_max_milliseconds=1000000 --write_buffer_size=134217728 --max_write_buffer_number=8 --target_file_size_base=134217728 --max_bytes_for_level_base=1073741824 --verify_checksum=1 --delete_obsolete_files_period_micros=62914560 --max_grandparent_overlap_factor=8 --max_bytes_for_level_multiplier=8 --statistics=1 --stats_per_interval=1 --stats_interval_seconds=60 --histogram=1 --memtablerep=skip_list --bloom_bits=10 --open_files=20480 --level0_file_num_compaction_trigger=4 --level0_slowdown_writes_trigger=12 --level0_stop_writes_trigger=20 --max_background_compactions=16 --max_background_flushes=7 --threads=1 --merge_operator="put

./db_bench --benchmarks=mergerandom --use_existing_db=1 --sync=0 --db=foobar --wal_dir=/txlogs --disable_data_sync=0 --num=1048576 --num_levels=6 --key_size=20 --value_size=4096 --block_size=32768 --cache_size=34359738368 --cache_numshardbits=6 --compression_type=snappy --min_level_to_compress=3 --compression_ratio=0.5 --level_compaction_dynamic_level_bytes=true --hard_rate_limit=3 --rate_limit_delay_max_milliseconds=1000000 --write_buffer_size=134217728 --max_write_buffer_number=8 --target_file_size_base=134217728 --max_bytes_for_level_base=1073741824 --verify_checksum=1 --delete_obsolete_files_period_micros=62914560 --max_grandparent_overlap_factor=8 --max_bytes_for_level_multiplier=8 --statistics=1 --stats_per_interval=1 --stats_interval_seconds=60 --histogram=1 --memtablerep=skip_list --bloom_bits=10 --open_files=20480 --duration=1800 --level0_file_num_compaction_trigger=4 --level0_slowdown_writes_trigger=12 --level0_stop_writes_trigger=20 --max_background_compactions=16 --max_background_flushes=7 --threads=1 --merge_operator="put"

./db_bench --benchmarks=updaterandom --use_existing_db=1 --sync=0 --db=foobar --wal_dir=/txlogs --disable_data_sync=0 --num=1048576 --num_levels=6 --key_size=20 --value_size=4096 --block_size=32768 --cache_size=34359738368 --cache_numshardbits=6 --compression_type=snappy --min_level_to_compress=3 --compression_ratio=0.5 --level_compaction_dynamic_level_bytes=true --hard_rate_limit=3 --rate_limit_delay_max_milliseconds=1000000 --write_buffer_size=134217728 --max_write_buffer_number=8 --target_file_size_base=134217728 --max_bytes_for_level_base=1073741824 --verify_checksum=1 --delete_obsolete_files_period_micros=62914560 --max_grandparent_overlap_factor=8 --max_bytes_for_level_multiplier=8 --statistics=1 --stats_per_interval=1 --stats_interval_seconds=60 --histogram=1 --memtablerep=skip_list --bloom_bits=10 --open_files=20480 --duration=1800 --level0_file_num_compaction_trigger=4 --level0_slowdown_writes_trigger=12 --level0_stop_writes_trigger=20 --max_background_compactions=16 --max_background_flushes=7 --threads=1 --merge_operator="put"