Consensus failure - Gaia main-75f0714294d2a43462cc2a9e53a1762882d468b6

issue.md

Summary

The Liquid Staking Module user flows include a set of accounting tests. One sequence of transactions result in a consensus failure.

Steps to reproduce

• Place both lsm_failure.sh and start_lsm_gaia.sh in the same folder.
• Follow the Gaia log with journalctl -fu lsm1.
• Run lsm_failure.sh.

The script does the following:

1. Start a single-validator chain using a build from commit 75f0714294d2a43462cc2a9e53a1762882d468b6 in the Gaia repo.
2. Delegate 100ATOM to the validator
3. Validator bond the delegation
4. Delegate 10ATOM more
5. Unbond 10ATOM
6. The chain will halt with a CONSENSUS FAILURE error.

Bug fix

See cosmos/cosmos-sdk#17436 for the bug fix in the Cosmos SDK repo.

lsm1.log

Aug 17 03:39:13 dante-lsm-test gaiad-lsm[3523541]: 3:39AM INF minted coins from module account amount=411uatom from=mint module=x/bank
Aug 17 03:39:13 dante-lsm-test gaiad-lsm[3523541]: 3:39AM INF executed block height=7 module=state num_invalid_txs=0 num_valid_txs=1
Aug 17 03:39:13 dante-lsm-test gaiad-lsm[3523541]: 3:39AM INF commit synced commit=436F6D6D697449447B5B3136322031313920323135203436203735203138342031333620323320363520372031363220352031363120333320373020373720323032203730203134382031303820323231203130352031323820363620393420373320313730203132392031383720323034203234362034305D3A377D
Aug 17 03:39:13 dante-lsm-test gaiad-lsm[3523541]: 3:39AM INF committed state app_hash=A277D72E4BB888174107A205A121464DCA46946CDD6980425E49AA81BBCCF628 height=7 module=state num_txs=1
Aug 17 03:39:13 dante-lsm-test gaiad-lsm[3523541]: 3:39AM INF indexed block exents height=7 module=txindex
Aug 17 03:39:14 dante-lsm-test gaiad-lsm[3523541]: 3:39AM INF Timed out dur=982.426026 height=8 module=consensus round=0 step=1
Aug 17 03:39:14 dante-lsm-test gaiad-lsm[3523541]: 3:39AM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"EA7A37E28D49A7D18E664DE0131BA5FAE6562AA16A66D4435553F5BC6E0376CE","parts":{"hash":"C612B20F916FAB0833F5781AF48C8BBAB44EB40D31BEA632933EFE76DC28B087","total":1}},"height":8,"pol_round":-1,"round":0,"signature":"AMrGKKrWduQY6E1bDx3plIwHgUMwadSw/L9Rt2fMv80yy92wly58yzbHzsD8I17Kd/XVg+uCnHO7prHUcMkAAA==","timestamp":"2023-08-17T03:39:14.594009593Z"}
Aug 17 03:39:14 dante-lsm-test gaiad-lsm[3523541]: 3:39AM INF received complete proposal block hash=EA7A37E28D49A7D18E664DE0131BA5FAE6562AA16A66D4435553F5BC6E0376CE height=8 module=consensus
Aug 17 03:39:14 dante-lsm-test gaiad-lsm[3523541]: 3:39AM INF finalizing commit of block hash={} height=8 module=consensus num_txs=1 root=A277D72E4BB888174107A205A121464DCA46946CDD6980425E49AA81BBCCF628
Aug 17 03:39:14 dante-lsm-test gaiad-lsm[3523541]: 3:39AM INF minted coins from module account amount=411uatom from=mint module=x/bank
Aug 17 03:39:14 dante-lsm-test gaiad-lsm[3523541]: 3:39AM INF executed block height=8 module=state num_invalid_txs=0 num_valid_txs=1
Aug 17 03:39:14 dante-lsm-test gaiad-lsm[3523541]: 3:39AM ERR CONSENSUS FAILURE!!! err="failed to apply block; error commit failed for application: error changing validator set: duplicate entry Validator{9C9224B36391989479B384D0761D2006A5DABDCA PubKeyEd25519{19C5969E2CC505AE52FD3F532ECE74E95EFDBBF4157A77050F71C0F6DC9E1959} VP:890 A:0} in [Validator{9C9224B36391989479B384D0761D2006A5DABDCA PubKeyEd25519{19C5969E2CC505AE52FD3F532ECE74E95EFDBBF4157A77050F71C0F6DC9E1959} VP:890 A:0} Validator{9C9224B36391989479B384D0761D2006A5DABDCA PubKeyEd25519{19C5969E2CC505AE52FD3F532ECE74E95EFDBBF4157A77050F71C0F6DC9E1959} VP:890 A:0}]" module=consensus stack="goroutine 91 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x65\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:732 +0x4c\npanic({0x1b28fe0, 0xc003a8e430})\n\truntime/panic.go:884 +0x213\ngithub.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc000b30000, 0x8)\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:1670 +0xfab\ngithub.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc000b30000, 0x8)\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:1570 +0x2ff\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit.func1()\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:1505 +0xaa\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit(0xc000b30000, 0x8, 0x0)\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:1543 +0xccf\ngithub.com/tendermint/tendermint/consensus.(*State).addVote(0xc000b30000, 0xc0037e1f40, {0x0, 0x0})\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:2164 +0x18dc\ngithub.com/tendermint/tendermint/consensus.(*State).tryAddVote(0xc000b30000, 0xc0037e1f40, {0x0?, 0x47cf34?})\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:1962 +0x2c\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc000b30000, {{0x26e2a80, 0xc003362800}, {0x0, 0x0}})\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:861 +0x3ff\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc000b30000, 0x0)\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:788 +0x4dc\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:379 +0x12d\n"
Aug 17 03:39:14 dante-lsm-test gaiad-lsm[3523541]: 3:39AM INF service stop impl={"Logger":{}} module=consensus msg={} wal=/root/.lsm1/data/cs.wal/wal
Aug 17 03:39:14 dante-lsm-test gaiad-lsm[3523541]: 3:39AM INF service stop impl={"Dir":"/root/.lsm1/data/cs.wal","Head":{"ID":"HHVBuscqELBO:/root/.lsm1/data/cs.wal/wal","Path":"/root/.lsm1/data/cs.wal/wal"},"ID":"group:HHVBuscqELBO:/root/.lsm1/data/cs.wal/wal","Logger":{}} module=consensus msg={} wal=/root/.lsm1/data/cs.wal/wal
Aug 17 03:39:17 dante-lsm-test gaiad-lsm[3523541]: 3:39AM INF Timed out dur=3000 height=8 module=consensus round=0 step=3
Aug 17 03:39:23 dante-lsm-test gaiad-lsm[3523541]: 3:39AM ERR Error on broadcastTxCommit err="timed out waiting for tx to be included in a block" module=rpc

lsm_failure.sh

#!/bin/bash
# Trigger consensus failure

mkdir -p $HOME/go/bin

export LSM_HOME_1=~/.lsm1
export LSM_SERVICE_1=lsm1.service
export LSM1_API_PORT="25001"
export LSM1_P2P_PORT="26001"
export LSM1_RPC_PORT="27001"
export LSM1_GRPC_PORT="28001"
export LSM1_PPROF_PORT="29001"
export VALOPER_1=cosmosvaloper1r5v5srda7xfth3hn2s26txvrcrntldju7lnwmv
export MNEMONIC_1="abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon art"
export MNEMONIC_2="abandon cabbage abandon cabbage abandon cabbage abandon cabbage abandon cabbage abandon cabbage abandon cabbage abandon cabbage abandon cabbage abandon cabbage abandon cabbage abandon garage"
export MONIKER_1=val1
export MONIKER_2=val2
export WALLET_1=cosmos1r5v5srda7xfth3hn2s26txvrcrntldjumt8mhl
export WALLET_2=cosmos1ay4dpm0kjmvtpug28vgw5w32yyjxa5sp97pjqq

export DENOM=uatom
export VAL_FUNDS="10000000000"
export VAL1_STAKE="800000000"

export LSM_BINARY=gaiad-lsm
export LSM_CHAIN_ID=lsm

# Install required packages
sudo apt install curl jq wget python-is-python3 python3-pip -y
python -m pip install toml-cli

rm -rf temp
mkdir temp

# Start LSM gaia
./start_lsm_gaia.sh

sleep 5

echo "Delegate and bond with $MONIKER_2..."
$LSM_BINARY tx staking delegate $VALOPER_1 100000000uatom --from $MONIKER_2 --home $LSM_HOME_1 --gas auto --fees 500uatom --gas-adjustment 1.2 -b block -y
$LSM_BINARY tx staking validator-bond $VALOPER_1 --from $MONIKER_2 --home $LSM_HOME_1 --gas auto --fees 1000uatom -b block -y --gas-adjustment 1.2

echo "delegating 10000000uatom from $MONIKER_2..."
$LSM_BINARY tx staking delegate $VALOPER_1 10000000uatom --from $MONIKER_2 --home $LSM_HOME_1 --gas auto --fees 1000uatom --gas-adjustment 1.2 -b block -y

echo "unbonding 10000000uatom from $MONIKER_2..."
$LSM_BINARY tx staking unbond $VALOPER_1 10000000uatom --from $MONIKER_2 --home $LSM_HOME_1 --gas auto --fees 1000uatom --gas-adjustment 1.2 -b block -y

start_lsm_gaia.sh

#!/bin/bash
# 1. Set up a one-validator chain

echo "Stopping services..."
systemctl disable $LSM_SERVICE_1 --now

# Install Gaia binary
echo "Installing Gaia..."
git clone https://github.com/cosmos/gaia.git
cd gaia
git checkout main
make build
cp build/gaiad ~/go/bin/gaiad-lsm
cd ..

# Initialize home directories
echo "Initializing node homes..."
rm -rf $LSM_HOME_1
$LSM_BINARY config chain-id $LSM_CHAIN_ID --home $LSM_HOME_1
$LSM_BINARY config keyring-backend test --home $LSM_HOME_1
$LSM_BINARY config broadcast-mode block --home $LSM_HOME_1
$LSM_BINARY config node tcp://localhost:$LSM1_RPC_PORT --home $LSM_HOME_1
$LSM_BINARY init $MONIKER_1 --chain-id $LSM_CHAIN_ID --home $LSM_HOME_1

# Create self-delegation accounts
echo $MNEMONIC_1 | $LSM_BINARY keys add $MONIKER_1 --keyring-backend test --home $LSM_HOME_1 --recover
echo $MNEMONIC_2 | $LSM_BINARY keys add $MONIKER_2 --keyring-backend test --home $LSM_HOME_1 --recover

# Update genesis file with right denom
echo "Setting denom to $DENOM..."
jq -r --arg denom "$DENOM" '.app_state.crisis.constant_fee.denom |= $denom' $LSM_HOME_1/config/genesis.json > temp/crisis.json
jq -r --arg denom "$DENOM" '.app_state.gov.deposit_params.min_deposit[0].denom |= $denom' temp/crisis.json > temp/min_deposit.json
jq -r --arg denom "$DENOM" '.app_state.mint.params.mint_denom |= $denom' temp/min_deposit.json > temp/mint.json
jq -r --arg denom "$DENOM" '.app_state.staking.params.bond_denom |= $denom' temp/mint.json > temp/bond_denom.json
cp temp/bond_denom.json $LSM_HOME_1/config/genesis.json
# cat $HOME_1/config/genesis.json

# Add funds to accounts
$LSM_BINARY add-genesis-account $MONIKER_1 $VAL_FUNDS$DENOM --home $LSM_HOME_1
$LSM_BINARY add-genesis-account $MONIKER_2 $VAL_FUNDS$DENOM --home $LSM_HOME_1

echo "Creating and collecting gentxs..."
mkdir -p $LSM_HOME_1/config/gentx
LSM1_NODE_ID=$($LSM_BINARY tendermint show-node-id --home $LSM_HOME_1)
$LSM_BINARY gentx $MONIKER_1 $VAL1_STAKE$DENOM --pubkey "$($LSM_BINARY tendermint show-validator --home $LSM_HOME_1)" --node-id $LSM1_NODE_ID --moniker $MONIKER_1 --chain-id $LSM_CHAIN_ID --home $LSM_HOME_1 --output-document $LSM_HOME_1/config/gentx/$MONIKER_1-gentx.json
$LSM_BINARY collect-gentxs --home $LSM_HOME_1

echo "Patching genesis file for fast governance..."
jq -r '.app_state.gov.voting_params.voting_period = "10s"' $LSM_HOME_1/config/genesis.json  > temp/voting.json
cp temp/voting.json $LSM_HOME_1/config/genesis.json

echo "Patching config files..."
# app.toml
# minimum_gas_prices
sed -i -e "/minimum-gas-prices =/ s^= .*^= \"0.0025$DENOM\"^" $LSM_HOME_1/config/app.toml

# Enable API
toml set --toml-path $LSM_HOME_1/config/app.toml api.enable true

# Set different ports for api
toml set --toml-path $LSM_HOME_1/config/app.toml api.address "tcp://0.0.0.0:$LSM1_API_PORT"

# Set different ports for grpc
toml set --toml-path $LSM_HOME_1/config/app.toml grpc.address "0.0.0.0:$LSM1_GRPC_PORT"

# Turn off grpc web
toml set --toml-path $LSM_HOME_1/config/app.toml grpc-web.enable false

# config.toml
echo "Setting ports for rpc..."
toml set --toml-path $LSM_HOME_1/config/config.toml rpc.laddr "tcp://0.0.0.0:$LSM1_RPC_PORT"

echo "Setting ports for rpc pprof..."
toml set --toml-path $LSM_HOME_1/config/config.toml rpc.pprof_laddr "localhost:$LSM1_PPROF_PORT"

echo "Setting ports for p2p..."
toml set --toml-path $LSM_HOME_1/config/config.toml p2p.laddr "tcp://0.0.0.0:$LSM1_P2P_PORT"

echo "Setting a short timeout commit..."
toml set --toml-path $LSM_HOME_1/config/config.toml consensus.timeout_commit "1s"

echo "Allow duplicate IP connections..."
toml set --toml-path $LSM_HOME_1/config/config.toml p2p.allow_duplicate_ip true

# Set fast_sync to false
toml set --toml-path $LSM_HOME_1/config/config.toml fast_sync false

echo "Setting up services..."

rm /etc/systemd/system/$LSM_SERVICE_1
touch /etc/systemd/system/$LSM_SERVICE_1
echo "[Unit]"                               | sudo tee /etc/systemd/system/$LSM_SERVICE_1
echo "Description=Gaia service"             | sudo tee /etc/systemd/system/$LSM_SERVICE_1 -a
echo "After=network-online.target"          | sudo tee /etc/systemd/system/$LSM_SERVICE_1 -a
echo ""                                     | sudo tee /etc/systemd/system/$LSM_SERVICE_1 -a
echo "[Service]"                            | sudo tee /etc/systemd/system/$LSM_SERVICE_1 -a
echo "User=$USER"                           | sudo tee /etc/systemd/system/$LSM_SERVICE_1 -a
echo "ExecStart=$HOME/go/bin/$LSM_BINARY start --x-crisis-skip-assert-invariants --home $LSM_HOME_1" | sudo tee /etc/systemd/system/$LSM_SERVICE_1 -a
echo "Restart=no"                           | sudo tee /etc/systemd/system/$LSM_SERVICE_1 -a
echo "LimitNOFILE=4096"                     | sudo tee /etc/systemd/system/$LSM_SERVICE_1 -a
echo ""                                     | sudo tee /etc/systemd/system/$LSM_SERVICE_1 -a
echo "[Install]"                            | sudo tee /etc/systemd/system/$LSM_SERVICE_1 -a
echo "WantedBy=multi-user.target"           | sudo tee /etc/systemd/system/$LSM_SERVICE_1 -a

sudo systemctl daemon-reload
sudo systemctl enable $LSM_SERVICE_1 --now

echo "***********************"
echo "To see the Gaia log enter:"
echo "journalctl -fu $LSM_SERVICE_1"
echo "***********************"