new_db_scheme

How to benchmark my proposed changes for block data db storage:

- export your chain data (preferably to a physical drive different from the one you put the chain db on)
- download my branch:
https://github.com/zsfelfoldi/go-ethereum/tree/chaindb
- set useNewDb in core/database_util.go to false
- compile
- import chain data (specify a new datadir)
- run "geth headerbenchmark" with the same datadir
- set useNewDb in core/database_util.go to true and repeat the last three steps
- compare and publish results (complete console output is not needed, just the 2*3 numbers, see below)

My results were:

- original db:
imported 965000 blocks in 7h7m22s
fetched 10000 headers in 2m14s
fetched 965000 headers in 4m31s

see complete console output here: https://gist.github.com/zsfelfoldi/c5bb0d925b92f3b88151782e9235ac58

- new db:
imported 965000 blocks in 4h18m58s (65% speed gain)
fetched 10000 headers in 1.8s (7340% speed gain)
fetched 965000 headers in 57s (375% speed gain)

see complete console output here: https://gist.github.com/zsfelfoldi/b60e66da87262fa822ccc266c948e5b3



Original storage scheme:

block-(hash)-header -> HEADER
block-(hash)-body -> BODY
block-(hash)-td -> TD
receipts-block-(hash) -> BLOCKRECEIPTS
block-num-(num) -> HASH   // num is encoded as BigInt.Bytes()

New storage scheme:

header-(num)(hash) -> HEADER  // num is always encoded as big endian uint64
blk-(num)(hash)-body -> BODY
blk-(num)(hash)-td -> TD
blk-(num)(hash)-receipts -> BLOCKRECEIPTS
header-(num)-hash -> HASH
blk-hash-(hash) -> NUM