Skip to content

Instantly share code, notes, and snippets.

@andy108369

andy108369/akash-recover-from-apphash-error.md

Last active November 12, 2023 01:11
Show Gist options
  • Save andy108369/44f5a676935286e0115431015ef66e1c to your computer and use it in GitHub Desktop.
Save andy108369/44f5a676935286e0115431015ef66e1c to your computer and use it in GitHub Desktop.
Download ZIP
Recovering from the AppHash error without having to use snapshot / state-sync
Raw
akash-recover-from-apphash-error.md

Recovering from the AppHash error without having to use snapshot / state-sync

akash 0.26.2 (cosmos-sdk v0.45.16, tendermint v0.34.27)

Follow this doc https://gist.github.com/andy108369/da1279257c018be6370310c990c25738

akash 0.16.4 (cosmos-sdk v0.45.4, tendermint v0.34.19)

With the 0.14.x client's TX create certificate transaction sent over 0.16.4 RPC causing the majority of 0.16.3 to reject that TX, it in turn caused the minority of 0.16.4 validators halt at 6955214 block. This is due to this commit in 0.16.4 which allowed old type of TX be present in the block whilst 0.16.3 - did not.

As well as many of the 0.16.4 RPC nodes halted at 6955214 block, with the 6955215 block in store which cannot be replayed because the majority (0.16.3) produced a different block hash than it was proposed and absorbed by the minority of 0.16.4 due to the above reason.

The RPC nodes would just hang at 6955214 height:

  "SyncInfo": {
    "latest_block_hash": "9920E6ACDCB30532EE625289ECD9A301BB66C8CB3709E99DB60F6A3C5C556ACD",
    "latest_app_hash": "D7BF7C12B7212B3E9FEBB538A839330711C81189CB72EF460216F6FF12985424",
    "latest_block_height": "6955214",
    "latest_block_time": "2022-07-29T16:18:14.626299012Z",
    "earliest_block_hash": "50AC0F1435391BFD95F607BD94C5B5BD7770BE935CED1A60A588335DC42C906C",
    "earliest_app_hash": "E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855",
    "earliest_block_height": "1",
    "earliest_block_time": "2021-03-08T15:00:00Z",
    "catching_up": false
  },

And fail on the attempt to re-start due to the following AppHash error:

9:29AM INF Starting IndexerService service impl=IndexerService module=txindex
9:29AM INF ABCI Handshake App Info hash="\x05�F�)ͧ\x10G\x1f4���Њ�Q�B�\x16������?l��" height=6955214 module=consensus protocol-version=0 software-version=v0.16.4
9:29AM INF ABCI Replay Blocks appHeight=6955214 module=consensus stateHeight=6955214 storeHeight=6955215
9:29AM INF Replay last block using real app module=consensus
Error: error during handshake: error on replay: wrong Block.Header.AppHash.  Expected 05EA469C29CDA710471F34F4E2E4D08ADD51B2429E169FBCA0A0E2C63F6CE687, got 1C09ABF4C088901A7ACC8B731371FCBEDF42D03073A359068532A0FD72C19C9E

Below is the way you can recover from the AppHash error without having to use snapshot / state-sync.

Worth noting that the following recovery procedure can also be used after a failed chain upgrade, so there is no need to hunt for a working snapshot, i.e. a snapshot until the block X causing the AppHash error.

Apply the rollback fix

1. tendermint

  1. Fork Akash's tendermint

https://github.com/akash-network/tendermint

  1. Clone your fork
git clone git@github.com:andy108369/tendermint.git
cd tendermint
  1. Add the upstream
git remote add upstream git@github.com:tendermint/tendermint.git
  1. Apply the patch
git checkout tags/v0.34.19-patches -b v0.34.19-patches-w-upstream-pr-8574
git fetch upstream pull/8574/head:pr8574
git cherry-pick -S 81344ac9464db02421fa115041f162ab9ace9372
  1. Tag and push the updates back to your fork
git tag v0.34.19-patches-w-upstream-pr-8574
git push --tags origin +HEAD

2. cosmos-sdk

  1. Fork Akash's cosmos-sdk

https://github.com/akash-network/cosmos-sdk

  1. Clone your fork
git clone git@github.com:andy108369/cosmos-sdk.git
cd cosmos-sdk
  1. Add the upstream
git remote add upstream git@github.com:cosmos/cosmos-sdk.git
  1. Apply the patch
git checkout tags/v0.45.4-akash.1 -b v0.45.4-akash.1-w-upstream-pr-11982
git fetch upstream pull/11982/head:pr11982
git cherry-pick -S 614906d69177db9555ea16ebb3e895b2f4617ff3
  1. Tag and push the updates back to your fork
git tag v0.45.4-akash.1-w-upstream-pr-11982
git push --tags origin +HEAD

3. akash

  1. Clone Akash
git clone git@github.com:ovrclk/akash.git
cd akash
  1. Checkout the mainnet version
git checkout -b tag-v0.16.4 v0.16.4
  1. Point to your patched tendermint & cosmos-sdk repos
go mod edit -replace github.com/tendermint/tendermint=github.com/andy108369/tendermint@v0.34.19-patches-w-upstream-pr-8574
go mod edit -replace github.com/cosmos/cosmos-sdk=github.com/andy108369/cosmos-sdk@v0.45.4-akash.1-w-upstream-pr-11982
  1. Build akash

using internal Go link mode without CGO enabled - to make sure the binary will start on the host with different GLIBC version.

GO_LINKMODE="internal" CGO_ENABLED=0 make
  1. Verify the binary got built with the patches
$ .cache/bin/akash version
0.16.4
$ .cache/bin/akash version --long | grep -E 'cosmos-sdk@|tendermint@'
- github.com/cosmos/cosmos-sdk@v0.45.1 => github.com/andy108369/cosmos-sdk@v0.45.4-akash.1-w-upstream-pr-11982
- github.com/tendermint/tendermint@v0.34.20-0.20220517115723-e6f071164839 => github.com/andy108369/tendermint@v0.34.19-patches-w-upstream-pr-8574

Now you can upload that binary to your Akash RPC node and use it to rollback the chain state.

(Optional?) Update AKASH_P2P_PERSISTENT_PEERS

Not sure if this is a must, but I thought it would be safer to start syncing the RPC off of a good-known node, in this example I've used Polkachu's state-sync live peer from here

export AKASH_P2P_PERSISTENT_PEERS="d1e47b071859497089c944dc082e920403484c1a@65.108.128.201:28656"

Might be also worthy turning off peer exchange service export AKASH_P2P_PEX=off, for at least until it will get through those 1-2 bad blocks produced by the 0.16.4 minority validator nodes still hanging around in the network.

You can put your peers back once you get past the AppHash error.

Rollback the chain

Note: I hit the AppHash error again when rolled back only once. But it worked when I've rolled back twice down to 6955212 height.

IMPORTANT Make sure you have set the same pruning setting in your ~/.akash/config/app.toml file if you are running the akash tool in different context (i.e. outside the container). This will corrupt the IAVL DB since changing the pruning strategy is not supported.

A single rollback takes about 30-40 minutes for the full chain (~670 GiB). No extra disk space is required.

# ./akash rollback --delete-pending-block
rollback pending block 6955215
Rolled back state to height 6955213 and hash D7BF7C12B7212B3E9FEBB538A839330711C81189CB72EF460216F6FF12985424

# ./akash rollback --delete-pending-block
rollback pending block 6955214
Rolled back state to height 6955212 and hash 6A815F05C7521E2ABB7DB6F85F11D185EF5DD1311C01D31D19B04596D40C089E

Start your RPC node

Start your RPC node with your original (official) akash binary.

I've used 0.16.3 since the majority of votes is being held by 0.16.3 as of today (31 July 2022).

RPC is syncing past through 6955214 block

Yay!

4:53PM INF starting node with ABCI Tendermint in-process
4:53PM INF Starting multiAppConn service impl=multiAppConn module=proxy
4:53PM INF Starting localClient service connection=query impl=localClient module=abci-client
4:53PM INF Starting localClient service connection=snapshot impl=localClient module=abci-client
4:53PM INF Starting localClient service connection=mempool impl=localClient module=abci-client
4:53PM INF Starting localClient service connection=consensus impl=localClient module=abci-client
4:53PM INF Starting EventBus service impl=EventBus module=events
4:53PM INF Starting PubSub service impl=PubSub module=pubsub
4:53PM INF Starting IndexerService service impl=IndexerService module=txindex
4:53PM INF ABCI Handshake App Info hash="j�_\x05�R\x1e*�}��_\x11х�]�1\x1c\x01�\x1d\x19�E��\f\b�" height=6955212 module=consensus protocol-version=0 software-version=v0.16.3
4:53PM INF ABCI Replay Blocks appHeight=6955212 module=consensus stateHeight=6955212 storeHeight=6955213
4:53PM INF Replay last block using real app module=consensus
4:53PM INF minted coins from module account amount=5238522uakt from=mint module=x/bank
4:53PM INF executed block height=6955213 module=consensus num_invalid_txs=0 num_valid_txs=0
4:53PM INF commit synced commit=436F6D6D697449447B5B323135203139312031323420313820313833203333203433203632203135392032333520313831203536203136382035372035312037203137203230302031372031333720323033203131342032333920373020322032322032343620323535203138203135322038342033365D3A3641323043447D
4:53PM INF committed state app_hash=D7BF7C12B7212B3E9FEBB538A839330711C81189CB72EF460216F6FF12985424 height=6955213 module=consensus num_txs=0
4:53PM INF Completed ABCI Handshake - Tendermint and App are synced appHash="j�_\x05�R\x1e*�}��_\x11х�]�1\x1c\x01�\x1d\x19�E��\f\b�" appHeight=6955212 module=consensus
4:53PM INF Version info block=11 p2p=8 tendermint_version=0.34.19
4:53PM INF This node is not a validator addr=A9E03498A2E43E0B5C9FC97032E51F17C68BDD7C module=consensus pubKey=+E1LeOuCY9Mew4u5XVCyqFJ17sniFvFhkGBNGbog7CQ=
4:53PM INF indexed block height=6955213 module=txindex
4:53PM INF P2P Node ID ID=b165de9bcfef068ec2ec292b2232583b058184d4 file=/akash/config/node_key.json module=p2p
4:53PM INF Add our address to book addr={"id":"b165de9bcfef068ec2ec292b2232583b058184d4","ip":"0.0.0.0","port":26656} book=/akash/config/addrbook.json module=p2p
4:53PM INF Starting Node service impl=Node
4:53PM INF Starting pprof server laddr=localhost:6060
4:53PM INF Starting RPC HTTP server on [::]:26657 module=rpc-server
4:53PM INF Starting P2P Switch service impl="P2P Switch" module=p2p
4:53PM INF Starting Evidence service impl=Evidence module=evidence
4:53PM INF Starting StateSync service impl=StateSync module=statesync
4:53PM INF Starting PEX service impl=PEX module=pex
4:53PM INF Starting AddrBook service book=/akash/config/addrbook.json impl=AddrBook module=p2p
4:53PM INF Starting Mempool service impl=Mempool module=mempool
4:53PM INF Starting BlockchainReactor service impl=BlockchainReactor module=blockchain
4:53PM INF Starting BlockPool service impl=BlockPool module=blockchain
4:53PM INF No addresses to dial. Falling back to seeds module=pex
4:53PM INF Starting Consensus service impl=ConsensusReactor module=consensus
4:53PM INF Reactor  module=consensus waitSync=true
4:53PM INF Saving AddrBook to file book=/akash/config/addrbook.json module=p2p size=1
4:53PM INF minted coins from module account amount=5238521uakt from=mint module=x/bank
4:53PM INF executed block height=6955214 module=state num_invalid_txs=1 num_valid_txs=1
4:53PM INF commit synced commit=436F6D6D697449447B5B32382039203137312032343420313932203133362031343420323620313232203230342031333920313135203139203131332032353220313930203232332036362032303820343820313135203136332038392036203133332035302031363020323533203131342031393320313536203135385D3A3641323043457D
4:53PM INF committed state app_hash=1C09ABF4C088901A7ACC8B731371FCBEDF42D03073A359068532A0FD72C19C9E height=6955214 module=state num_txs=2
4:53PM INF indexed block height=6955214 module=txindex
4:53PM INF minted coins from module account amount=5238521uakt from=mint module=x/bank
4:53PM INF executed block height=6955215 module=state num_invalid_txs=0 num_valid_txs=0
4:53PM INF commit synced commit=436F6D6D697449447B5B34332031313720323033203230382032303120313237203130322032323920323532203339203132332031363320392032303020323435203634203335203133332032313720313339203238203230372031303720323233203820383820393320343520313733203437203138342031355D3A3641323043467D
4:53PM INF committed state app_hash=2B75CBD0C97F66E5FC277BA309C8F5402385D98B1CCF6BDF08585D2DAD2FB80F height=6955215 module=state num_txs=0
4:53PM INF indexed block height=6955215 module=txindex
4:53PM INF minted coins from module account amount=5238520uakt from=mint module=x/bank
4:53PM INF executed block height=6955216 module=state num_invalid_txs=0 num_valid_txs=1
4:53PM INF commit synced commit=436F6D6D697449447B5B352032343120313835203232352037392032303020313820353220363920323432203232302036382038203231392039392039382032353420383320313036203136342032343420313334203137342033332031313120313934203536203133342039302031353620323032203232345D3A3641323044307D
4:53PM INF committed state app_hash=05F1B9E14FC8123445F2DC4408DB6362FE536AA4F486AE216FC238865A9CCAE0 height=6955216 module=state num_txs=1
4:53PM INF indexed block height=6955216 module=txindex
4:53PM INF minted coins from module account amount=5238520uakt from=mint module=x/bank
4:53PM INF executed block height=6955217 module=state num_invalid_txs=0 num_valid_txs=0
4:53PM INF commit synced commit=436F6D6D697449447B5B32372032313720333620313520393820323031203630203135332034332031393820323430203138372031373020313335203233312031343520323530203136302031383620313037203137392031363720313330203131342036352032353020323137203131332032343320373820313937203137395D3A3641323044317D
4:53PM INF committed state app_hash=1BD9240F62C93C992BC6F0BBAA87E791FAA0BA6BB3A7827241FAD971F34EC5B3 height=6955217 module=state num_txs=0
4:53PM INF indexed block height=6955217 module=txindex
4:53PM INF minted coins from module account amount=5238519uakt from=mint module=x/bank
4:53PM INF executed block height=6955218 module=state num_invalid_txs=0 num_valid_txs=1
4:53PM INF commit synced commit=436F6D6D697449447B5B323439203333203130392031393820313739203233382031373820313238203133322038203237203934203133382031353120373120313935203234342031363820313335203131362031393120323438203020313630203735203131342031373420323531203731203137392035392031365D3A3641323044327D
4:53PM INF committed state app_hash=F9216DC6B3EEB28084081B5E8A9747C3F4A88774BFF800A04B72AEFB47B33B10 height=6955218 module=state num_txs=1
4:53PM INF indexed block height=6955218 module=txindex
4:53PM INF minted coins from module account amount=5238518uakt from=mint module=x/bank
4:53PM INF executed block height=6955219 module=state num_invalid_txs=0 num_valid_txs=0
4:53PM INF commit synced commit=436F6D6D697449447B5B3130332038302031392031342031343520373120323320313235203230382034332032313620323020313230203137362038332034342031362033302039362034203632203234392038322034312032313920373420323520313836203820313338203235342038305D3A3641323044337D
4:53PM INF committed state app_hash=6750130E9147177DD02BD81478B0532C101E60043EF95229DB4A19BA088AFE50 height=6955219 module=state num_txs=0
4:53PM INF indexed block height=6955219 module=txindex
4:53PM INF minted coins from module account amount=5238518uakt from=mint module=x/bank
4:53PM INF executed block height=6955220 module=state num_invalid_txs=0 num_valid_txs=0
4:53PM INF commit synced commit=436F6D6D697449447B5B313130203137362031343320323120352031393520313339203132312031322030203137203739203234362032343920313134203232362031353820373920313038203136342032353220313034203137312031353520323139203132392032323120342034302031323320313936203130395D3A3641323044347D
4:53PM INF committed state app_hash=6EB08F1505C38B790C00114FF6F972E29E4F6CA4FC68AB9BDB81DD04287BC46D height=6955220 module=state num_txs=0
4:53PM INF indexed block height=6955220 module=txindex
4:53PM INF minted coins from module account amount=5238517uakt from=mint module=x/bank
4:53PM INF executed block height=6955221 module=state num_invalid_txs=0 num_valid_txs=1
4:53PM INF commit synced commit=436F6D6D697449447B5B3138372033392033362033352032343020313036203131203730203935203134332032353120313937203220383220363120313030203636203137302031312031353920323436203138312032333620353120363220313239203235332032343120313436203633203133322033355D3A3641323044357D
4:53PM INF committed state app_hash=BB272423F06A0B465F8FFBC502523D6442AA0B9FF6B5EC333E81FDF1923F8423 height=6955221 module=state num_txs=1
4:53PM INF indexed block height=6955221 module=txindex
4:53PM INF minted coins from module account amount=5238517uakt from=mint module=x/bank
4:53PM INF executed block height=6955222 module=state num_invalid_txs=0 num_valid_txs=0
4:53PM INF commit synced commit=436F6D6D697449447B5B313931203136203230342037322031313520313332203431203137332037372031303420313132203220313431203232312031382032303620323531203234332039342032303320323134203135203134312032343520343220323037203533203732203135203837203536203139315D3A3641323044367D
4:53PM INF committed state app_hash=BF10CC48738429AD4D6870028DDD12CEFBF35ECBD60F8DF52ACF35480F5738BF height=6955222 module=state num_txs=0
4:53PM INF indexed block height=6955222 module=txindex
4:53PM INF minted coins from module account amount=5238516uakt from=mint module=x/bank
4:53PM INF executed block height=6955223 module=state num_invalid_txs=0 num_valid_txs=0
4:53PM INF commit synced commit=436F6D6D697449447B5B323420333620393420323430203132342032303120323431203131312032372035392031373720313031203320313630203136342034332032372031383520313733203832203136312032333420312032332031333220313837203130342032303520353420323439203937203132315D3A3641323044377D
4:53PM INF committed state app_hash=18245EF07CC9F16F1B3BB16503A0A42B1BB9AD52A1EA011784BB68CD36F96179 height=6955223 module=state num_txs=0
4:53PM INF indexed block height=6955223 module=txindex
4:53PM INF minted coins from module account amount=5238515uakt from=mint module=x/bank
4:53PM INF executed block height=6955224 module=state num_invalid_txs=0 num_valid_txs=0
4:53PM INF commit synced commit=436F6D6D697449447B5B323435203130372039332033342032333520343320323036203139392039322032343720313633203332203235352031313220323235203231322031343220323135203231362031313320383620373720323136203734203730203139352032323720313132203233322031313120323130203135345D3A3641323044387D
4:53PM INF committed state app_hash=F56B5D22EB2BCEC75CF7A320FF70E1D48ED7D871564DD84A46C3E370E86FD29A height=6955224 module=state num_txs=0
4:53PM INF indexed block height=6955224 module=txindex
4:53PM INF minted coins from module account amount=5238515uakt from=mint module=x/bank
4:53PM INF executed block height=6955225 module=state num_invalid_txs=0 num_valid_txs=1
4:53PM INF commit synced commit=436F6D6D697449447B5B35372032313520313836203532203233372031323220313630203239203231342032333620323437203139372032303320313839203232362039362031373820353720323320313237203234362033392031353520323434203139312032343020313437203131342037332038372031373020335D3A3641323044397D
4:53PM INF committed state app_hash=39D7BA34ED7AA01DD6ECF7C5CBBDE260B239177FF6279BF4BFF093724957AA03 height=6955225 module=state num_txs=1
4:53PM INF indexed block height=6955225 module=txindex
4:53PM INF minted coins from module account amount=5238514uakt from=mint module=x/bank
4:53PM INF executed block height=6955226 module=state num_invalid_txs=0 num_valid_txs=0
4:53PM INF commit synced commit=436F6D6D697449447B5B363220313136203132352039332032303720313835203134382031373920313738203234203138302032313720313933203139352037203139342031313420392038362031303620363620313033203230322031313920363220323620313020313433203133392032343120313733203233305D3A3641323044417D
4:53PM INF committed state app_hash=3E747D5DCFB994B3B218B4D9C1C307C27209566A4267CA773E1A0A8F8BF1ADE6 height=6955226 module=state num_txs=0
4:53PM INF indexed block height=6955226 module=txindex
4:53PM INF minted coins from module account amount=5238514uakt from=mint module=x/bank
4:53PM INF executed block height=6955227 module=state num_invalid_txs=0 num_valid_txs=0
4:53PM INF commit synced commit=436F6D6D697449447B5B31313520313038203933203133352031332031313320333020313437203231203131322032343820313630203234302031323020312031323020353420353620323133203131362032343720313933203134203133322037362032333120373120323230203139203533203139203138375D3A3641323044427D
4:53PM INF committed state app_hash=736C5D870D711E931570F8A0F07801783638D574F7C10E844CE747DC133513BB height=6955227 module=state num_txs=0
4:53PM INF indexed block height=6955227 module=txindex
4:53PM INF minted coins from module account amount=5238513uakt from=mint module=x/bank
4:53PM INF executed block height=6955228 module=state num_invalid_txs=0 num_valid_txs=0
4:53PM INF commit synced commit=436F6D6D697449447B5B353720313420313136203230372031343920313437203235302039312031313220323031203636203636203231352031373620393520313338203230322032323920313139203230203835203137302032303120313533203133302032343320313634203230312032343320313436203832203137305D3A3641323044437D
4:53PM INF committed state app_hash=390E74CF9593FA5B70C94242D7B05F8ACAE5771455AAC99982F3A4C9F39252AA height=6955228 module=state num_txs=0
4:53PM INF indexed block height=6955228 module=txindex
4:53PM INF minted coins from module account amount=5238512uakt from=mint module=x/bank
4:53PM INF executed block height=6955229 module=state num_invalid_txs=0 num_valid_txs=1
4:53PM INF commit synced commit=436F6D6D697449447B5B3130362031342031323820313520313937203135392031363720313339203135362032313820313633203333203138392032343920353920313738203134352034203333203231332031313820313620353020383620363620323039203139352032332032343220313835203136312031395D3A3641323044447D
4:53PM INF committed state app_hash=6A0E800FC59FA78B9CDAA321BDF93BB2910421D57610325642D1C317F2B9A113 height=6955229 module=state num_txs=1
4:53PM INF indexed block height=6955229 module=txindex
4:53PM INF minted coins from module account amount=5238512uakt from=mint module=x/bank
4:53PM INF executed block height=6955230 module=state num_invalid_txs=0 num_valid_txs=0
4:53PM INF commit synced commit=436F6D6D697449447B5B32343520313536203234392032343520333120353620313937203439203737203139302032343420323820343620323135203136322034332031343720333620363420323531203535203920343020323032203235332031393420373120343920313036203520313430203134335D3A3641323044457D
4:53PM INF committed state app_hash=F59CF9F51F38C5314DBEF41C2ED7A22B932440FB370928CAFDC247316A058C8F height=6955230 module=state num_txs=0
4:53PM INF indexed block height=6955230 module=txindex
4:53PM INF minted coins from module account amount=5238511uakt from=mint module=x/bank
4:53PM INF executed block height=6955231 module=state num_invalid_txs=0 num_valid_txs=0
4:53PM INF commit synced commit=436F6D6D697449447B5B31353020323137203135382033302032302034342032313020383920372031373120323136203539203136312031313220323438203530203137382031333120313834203230362032303220313535203137322031393820323139203232392031393720323233203238203136203130203130325D3A3641323044467D
4:53PM INF committed state app_hash=96D99E1E142CD25907ABD83BA170F832B283B8CECA9BACC6DBE5C5DF1C100A66 height=6955231 module=state num_txs=0
4:53PM INF indexed block height=6955231 module=txindex
4:53PM INF minted coins from module account amount=5238511uakt from=mint module=x/bank
4:53PM INF executed block height=6955232 module=state num_invalid_txs=0 num_valid_txs=1
4:53PM INF commit synced commit=436F6D6D697449447B5B3639203234203235203135312031373820323136203130342036322031303120383320313530203230352031322031393520323134203133322031392031323220313320333720353020313333203834203138342032313120313734203534203131392031353020393220323231203233375D3A3641323045307D
4:53PM INF committed state app_hash=45181997B2D8683E655396CD0CC3D684137A0D25328554B8D3AE3677965CDDED height=6955232 module=state num_txs=1
4:53PM INF indexed block height=6955232 module=txindex
4:53PM INF minted coins from module account amount=5238510uakt from=mint module=x/bank
4:53PM INF executed block height=6955233 module=state num_invalid_txs=0 num_valid_txs=1
4:53PM INF commit synced commit=436F6D6D697449447B5B32323920313136203136372032313920353520323334203733203738203130342033302031302037372034302031383120322031353520373020313334203135342037342038322031353420313739203930203131312031303720333320373120323330203235203230312034305D3A3641323045317D
4:53PM INF committed state app_hash=E574A7DB37EA494E681E0A4D28B5029B46869A4A529AB35A6F6B2147E619C928 height=6955233 module=state num_txs=1
4:53PM INF indexed block height=6955233 module=txindex
4:53PM INF minted coins from module account amount=5238509uakt from=mint module=x/bank
4:53PM INF executed block height=6955234 module=state num_invalid_txs=0 num_valid_txs=0
4:53PM INF commit synced commit=436F6D6D697449447B5B313138203234352032313820362032333820313133203234342032303620343520313136203230332031313720313137203135312033342031203232342031323320313639203233382032343920362031343220323135203138372032352031343120313339203232342039203832203137345D3A3641323045327D
4:53PM INF committed state app_hash=76F5DA06EE71F4CE2D74CB7575972201E07BA9EEF9068ED7BB198D8BE00952AE height=6955234 module=state num_txs=0
4:53PM INF indexed block height=6955234 module=txindex
4:53PM INF minted coins from module account amount=5238509uakt from=mint module=x/bank
4:53PM INF packet sent dst_channel=channel-1 dst_port=transfer module=x/ibc/channel sequence=88026 src_channel=channel-9 src_port=transfer
4:53PM INF IBC fungible token transfer amount=87300000 module=x/ibc-transfer receiver=osmo1aszpxp86gq9qnw8yyyxfkcfcau0870u3zm4w3a sender=akash1aszpxp86gq9qnw8yyyxfkcfcau0870u38mte74 token=uakt
4:53PM INF executed block height=6955235 module=state num_invalid_txs=0 num_valid_txs=1
4:53PM INF commit synced commit=436F6D6D697449447B5B313237203233322039302031323020323131203230302031333520313532203935203131392031383220313739203238203830203433203232372034312032343520323535203138203133342035312031393620313535203130382031352038382032343820323031203135332039322035365D3A3641323045337D
4:53PM INF committed state app_hash=7FE85A78D3C887985F77B6B31C502BE329F5FF128633C49B6C0F58F8C9995C38 height=6955235 module=state num_txs=1
4:53PM INF indexed block height=6955235 module=txindex
4:53PM INF minted coins from module account amount=5238508uakt from=mint module=x/bank
4:53PM INF executed block height=6955236 module=state num_invalid_txs=0 num_valid_txs=0
4:53PM INF commit synced commit=436F6D6D697449447B5B3832203232322031303120323232203230392035342032323520313120313737203230362032303820323530203131382031343920323438203636203435203830203133302031312031343420313237203635203134392036312032343520323530203830203938203234342037203134385D3A3641323045347D
4:53PM INF committed state app_hash=52DE65DED136E10BB1CED0FA7695F8422D50820B907F41953DF5FA5062F40794 height=6955236 module=state num_txs=0
4:53PM INF indexed block height=6955236 module=txindex
4:53PM INF minted coins from module account amount=5238508uakt from=mint module=x/bank
4:53PM INF executed block height=6955237 module=state num_invalid_txs=0 num_valid_txs=1
4:53PM INF commit synced commit=436F6D6D697449447B5B3230312032302037382031333120313037203838203137372031203134352031393720313520313831203133382032392032343020313330203234382032343620332032333020393120323420313538203235352032343320323130203231203231352031343120383920363620365D3A3641323045357D
4:53PM INF committed state app_hash=C9144E836B58B10191C50FB58A1DF082F8F603E65B189EFFF3D215D78D594206 height=6955237 module=state num_txs=1
4:53PM INF indexed block height=6955237 module=txindex
4:53PM INF minted coins from module account amount=5238507uakt from=mint module=x/bank
4:53PM INF executed block height=6955238 module=state num_invalid_txs=0 num_valid_txs=0
4:53PM INF commit synced commit=436F6D6D697449447B5B3137392032343220383420323336203938203132302031313520333120343020313832203832203820313033203232332031343620313730203720392031363220323520323120313635203232312031383920353220323432203233392039372031353320323334203737203138395D3A3641323045367D
4:53PM INF committed state app_hash=B3F254EC6278731F28B6520867DF92AA0709A21915A5DDBD34F2EF6199EA4DBD height=6955238 module=state num_txs=0
4:53PM INF indexed block height=6955238 module=txindex
...

RPC is catching up

root@primary:~# akash status | jq 
...
  "SyncInfo": {
    "latest_block_hash": "0BFAB4EC28A1B891C0FDB8E305D45F2C102C650112E6B21DD0FB68831997EDCF",
    "latest_app_hash": "992EDDC3C63E5660CDEA5E06F4A1F42565FD70A1F3A661A6CC38BFC7F89F0410",
    "latest_block_height": "6964776",
    "latest_block_time": "2022-07-30T09:21:13.928486335Z",
    "earliest_block_hash": "50AC0F1435391BFD95F607BD94C5B5BD7770BE935CED1A60A588335DC42C906C",
    "earliest_app_hash": "E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855",
    "earliest_block_height": "1",
    "earliest_block_time": "2021-03-08T15:00:00Z",
    "catching_up": true
  },
...

Blocks and their hashes around the 6955215 height:

root@primary:~# for i in $(seq 6955212 6955217); do printf "%d\t%s\n" $i $(akash query block $i | jq -r '.block_id.hash'); done
6955212	EB65621F5C03D91B369D1344D5778949C1FA3ACD4D8E54A13B07A07AC1217548
6955213	8BB9DD1A1C108179517ACB748908018D679E6F388D616E80E773FD3B3A3DAF83
6955214	9920E6ACDCB30532EE625289ECD9A301BB66C8CB3709E99DB60F6A3C5C556ACD
6955215	B964D620DBAB650CAA0EABC18F1DAE9FAF01B1F02502BBED80F9AD6E4AB4BC42
6955216	4ED974CDB2FBA47789E62EED018C165FC9D383969773FD369E73CF4ACF913B8E
6955217	9F6D394FFE1F783614F09E7E6FBBD6A60936D9C029657807DA42CACFDC4CC507

Update: RPC catched up!

  "SyncInfo": {
    "latest_block_hash": "7FAB99CD9FA06AB20062E00C7C62FFEF49BCDEA5120C2D422AE98FAF6569EB80",
    "latest_app_hash": "1B5ACF0B66C93C90F210D97BEDF4CB7F2CDEBF0F731A75415C8B8963BD7876CD",
    "latest_block_height": "6984178",
    "latest_block_time": "2022-07-31T18:12:10.250798116Z",
    "earliest_block_hash": "50AC0F1435391BFD95F607BD94C5B5BD7770BE935CED1A60A588335DC42C906C",
    "earliest_app_hash": "E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855",
    "earliest_block_height": "1",
    "earliest_block_time": "2021-03-08T15:00:00Z",
    "catching_up": false
  },

Kudos & References

Huge kudos going to:

  • Artur Troian for quick action identifying the issue;
  • yihuang for pointing me to the rollback fix code (1, 2) for tendermint 0.34.x during the weekend;
  • polkachu | polkachu.com#1249 (on Discord, their website) for their State-Sync node and awesome procedure on how to use state-sync;
  • Everyone else contributed I forgot to mention!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment