InvalidStateRoot after BlockReorg in OE

README.MD

It seems that there is an heisenbug in block reorganization in Parity 2.5.13 that also happens in OpenEthereum 3.1, that corrupts the database. The node seems that is not syncing but if you generate traces with -l sync=trace (you can dump into a file with also adding --log-file=<logfile>) you see the following pattern:

Block import failed for #<block_number> (block_hash)
Error: Error(Trie(InvalidStateRoot(<stateroot>)), State { next_error: None, backtrace: InternalBacktrace { backtrace: None } })

With a lot of Transactions:[Tx XXX] UnverifiedTransaction

Warning

This is an experimental fix, backup your database before using it!

1) Compile the following code

This is a branch with some experimental db tools https://github.com/openethereum/backport-3.1/tree/adria0/dbtools

2) Go back 10 blocks

Run openethereum db reset 10

3) Remove all future blocks in the db

This is done by executing openethereum db experimental-drop-future-blocks, a list of removed blocks should be shown, like:

2020-10-08 19:39:32 UTC Removed future block 10964073 0xc4ec…e471
2020-10-08 19:39:32 UTC Removed future block 10964074 0x63f8…0936
2020-10-08 19:39:32 UTC Removed future block 10964075 0xa0b6…1957
2020-10-08 19:39:32 UTC Removed future block 10964076 0x22bf…768e
2020-10-08 19:39:32 UTC Removed future block 10964077 0x317c…b49b
2020-10-08 19:39:32 UTC Removed future block 10964078 0x4c9a…3acf
2020-10-08 19:39:32 UTC Removed future block 10964079 0xd11b…004d

4) Execute again openethereum in sync mode

If continues without working:

• take a look of the created logs in the last run (after executing the experimental-drop-future-blocks) with -l sync=trace, it should be this pattern in the logs:

TRACE sync  NewBlocks: Block already in chain <BlockHash>

or

TRACE sync  NewBlocks: Head already in chain <BlockHash>

• for each BlockHash that is NOT listed in openethereum db info (usually only one) run the command openethreum experimental-drop-block <BlockHash> (NOTE: remove the 0x prefix)

Most excellent, thank you!

Couple notes: I see a <Best block hash> for "already in chain", not a number. And the command for dropping the ones after that, but not the best block, would be openethereum db experimental-drop-block <BLOCK> .

It looks like this saved me quite a bit of time. Hopefully the elusive async chain reorg bug can be found and squashed in due time.

following up in case others end up here.

I was unable to use this solution with ubuntu xenial + oe 3.1

db tools did not successfully remove unwanted blocks :(

details in this issue. openethereum/openethereum#121