Around 17.06.2019 our DASH node started showing some strange behaviour. When requesting blocks from the node, their reported difficulty was around ~50x lower than the actual network.
Average Daily Dash Difficulty
| Date | Difficulty |
|---|---|
| 06/23/2019 | 2,401,351.60049360 |
| 06/22/2019 | 2,401,351.60049360 |
| 06/21/2019 | 1,951,253.21641451 |
| 06/20/2019 | 1,973,985.24137748 |
| 06/19/2019 | 1,981,790.80543594 |
| 06/18/2019 | 7,805,494.43518114 |
| 06/17/2019 | 42,292,710.61580382 |
| 06/16/2019 | 136,319,608.52123660 |
| 06/15/2019 | 137,118,602.56724745 |
| 06/14/2019 | 140,281,925.68376438 |
| source: taken from coinwarz |
The values reported by our node was in line with those from coinwarz above.
This caused issues such as our mining profitability estimations being extremely high. However, the node seemed to be syncing and blocks could be queried.
Another problem was that transactions that should have existed on chain (and could be found in the official dash explorer), weren't found by our applications.
The node version running at this point was dashpay/dash v0.13.3.0.
Our node was not the only node with this issue however. It was found that:
- coinwarz indicated the same drop in difficulty
- blockcypher showed their node to be stuck on block
1089632(2019-06-19T07:29:19Z)
The node was reporting warnings with version bit's not being understood, but this was discarded as due to miners rolling version bits as part of the block nonce (bitcoin/bitcoin#15471).
Finally, this behaviour is believed to be have been due to the activation of DIP4/DIP8 trigger (see v0.14 adoption tab)
The activation of DIP8 roughly coincides to the time when these issues were first seen and around the same block that blockcypher's node is stuck. https://twitter.com/Dashpay/status/1140900039138717698
Initially upgrading the node to v0.14.0.1 also did not seem to resolve the issue. Instead the node got stuck during upgrade.
- Existing issue on github #2995 for a similar problem was found.
Using the same
reconsiderblock 00000000000000112e41e4b3afda8b233b8cc07c532d2eac5de097b68358c43edash-cli command on the DIP8 Activation Block [1088640] method seemed to work.
It is noteworthy that my node showed no indication of being stuck as in issues #2995.
After running the command, the node did it's rollback resynced onwards.
It is possible our node was part of a network segment of nodes the did not upgrade to the new v0.14.0.X version and didn't understand some of the new header data fields being sent. Potentially causing a network segment. See changes here:
@Auronmatrix yes, pre-0.14 nodes forked off because they were not able to recognize coinbase extra payload v2 (https://github.com/dashpay/dips/blob/master/dip-0004.md#coinbase-special-transaction) which was activated via bit 4. Few miners and explorers did not upgrade and kept extending/displaying the wrong chain which caused a confusion.
