Skip to content

Instantly share code, notes, and snippets.

@clemensgg

clemensgg/nomic_ibc_issues.md

Last active January 17, 2024 07:00
Show Gist options
  • Save clemensgg/84e57a8ddb98c72f55ea7663465027c8 to your computer and use it in GitHub Desktop.
Save clemensgg/84e57a8ddb98c72f55ea7663465027c8 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
nomic_ibc_issues.md

We've detected issues when relaying ICS-20 transfer packets between nomic-stakenet-3 and osmosis-1

hermes logs:

hermes --config ~/.hermes/config-neutron.toml clear packets --chain nomic-stakenet-3 --port transfer --channel channel-1
2023-12-26T18:59:06.521409Z  INFO ThreadId(01) running Hermes v1.7.4+d01ed3f3
2023-12-26T18:59:06.635166Z  INFO ThreadId(01) Querying Omosis EIP-1559 base fee from http://162.55.92.114:2001/
2023-12-26T18:59:06.660181Z  INFO ThreadId(01) Omosis EIP-1559 base fee is 0.002536788435807151
2023-12-26T18:59:06.660325Z  INFO ThreadId(01) Querying Omosis EIP-1559 base fee from http://162.55.92.114:2001/
2023-12-26T18:59:06.689762Z  INFO ThreadId(01) Omosis EIP-1559 base fee is 0.002536788435807151
2023-12-26T18:59:07.333822Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}: 1 unreceived packets found: [ 1453 ] 
2023-12-26T18:59:07.617412Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}: pulled packet data for 1 events out of 1 sequences: 1453..=1453; events.total=1 events.left=0
2023-12-26T18:59:07.727119Z DEBUG ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:schedule{odata=packet-recv ->Destination @1-12951593; len=1}: connection delay need not be taken into account: client update message will be prepended later
2023-12-26T18:59:07.727140Z DEBUG ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: retrying retry.current=1 retry.max=5
2023-12-26T18:59:07.727145Z DEBUG ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: prepending Destination client update at height 1-12951594
2023-12-26T18:59:10.378496Z DEBUG ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:foreign_client.wait_and_build_update_client_with_trusted{client=osmosis-1->nomic-stakenet-3:07-tendermint-1 target_height=1-12951594}:foreign_client.build_update_client_with_trusted{client=osmosis-1->nomic-stakenet-3:07-tendermint-1 target_height=1-12951594}: building a MsgUpdateAnyClient from trusted height 1-12938815 to target height 1-12951594
2023-12-26T18:59:10.378687Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: assembled batch of 2 message(s)
2023-12-26T18:59:10.382248Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}: sending 2 messages as 1 batches to chain nomic-stakenet-3 in parallel
2023-12-26T18:59:10.382270Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: max fee, for use in tx simulation: Fee { amount: "75001unbtc", gas_limit: 3000000 }
2023-12-26T18:59:10.384360Z  WARN ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}:estimate_gas: tx simulation successful but no gas amount used was returned, falling back on default gas: 1000000
2023-12-26T18:59:10.384392Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: send_tx: using 1000000 gas, fee Fee { amount: "32501unbtc", gas_limit: 1300000 } id=nomic-stakenet-3
2023-12-26T18:59:10.410151Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: gas estimation succeeded
2023-12-26T18:59:10.410181Z ERROR ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: failed to broadcast tx with unrecoverable error response=Response { code: Err(1), data: b"", log: "Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``", hash: Hash::Sha256(6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A) } diagnostic=unknown TX sync response error: 1
2023-12-26T18:59:10.410281Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: [Sync->nomic-stakenet-3] result events:
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``") at height 3-1

2023-12-26T18:59:10.410310Z ERROR ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: error ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``")
2023-12-26T18:59:10.410322Z  WARN ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: failed. Regenerate operational data from 1 events
2023-12-26T18:59:10.435635Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: will retry odata=packet-recv ->Destination @1-12951593; len=1
2023-12-26T18:59:10.435664Z DEBUG ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: retrying retry.current=2 retry.max=5
2023-12-26T18:59:10.435669Z DEBUG ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: prepending Destination client update at height 1-12951594
2023-12-26T18:59:10.499374Z DEBUG ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:foreign_client.wait_and_build_update_client_with_trusted{client=osmosis-1->nomic-stakenet-3:07-tendermint-1 target_height=1-12951594}:foreign_client.build_update_client_with_trusted{client=osmosis-1->nomic-stakenet-3:07-tendermint-1 target_height=1-12951594}: building a MsgUpdateAnyClient from trusted height 1-12938815 to target height 1-12951594
2023-12-26T18:59:10.499507Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: assembled batch of 2 message(s)
2023-12-26T18:59:10.499704Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}: sending 2 messages as 1 batches to chain nomic-stakenet-3 in parallel
2023-12-26T18:59:10.499720Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: max fee, for use in tx simulation: Fee { amount: "75001unbtc", gas_limit: 3000000 }
2023-12-26T18:59:10.501978Z  WARN ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}:estimate_gas: tx simulation successful but no gas amount used was returned, falling back on default gas: 1000000
2023-12-26T18:59:10.502000Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: send_tx: using 1000000 gas, fee Fee { amount: "32501unbtc", gas_limit: 1300000 } id=nomic-stakenet-3
2023-12-26T18:59:10.524323Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: gas estimation succeeded
2023-12-26T18:59:10.524344Z ERROR ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: failed to broadcast tx with unrecoverable error response=Response { code: Err(1), data: b"", log: "Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``", hash: Hash::Sha256(6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A) } diagnostic=unknown TX sync response error: 1
2023-12-26T18:59:10.524398Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: [Sync->nomic-stakenet-3] result events:
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``") at height 3-1

2023-12-26T18:59:10.524418Z ERROR ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: error ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``")
2023-12-26T18:59:10.524425Z  WARN ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: failed. Regenerate operational data from 1 events
2023-12-26T18:59:10.595466Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: will retry odata=packet-recv ->Destination @1-12951593; len=1
2023-12-26T18:59:10.595492Z DEBUG ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: retrying retry.current=3 retry.max=5
2023-12-26T18:59:10.595500Z DEBUG ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: prepending Destination client update at height 1-12951594
2023-12-26T18:59:10.738713Z DEBUG ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:foreign_client.wait_and_build_update_client_with_trusted{client=osmosis-1->nomic-stakenet-3:07-tendermint-1 target_height=1-12951594}:foreign_client.build_update_client_with_trusted{client=osmosis-1->nomic-stakenet-3:07-tendermint-1 target_height=1-12951594}: building a MsgUpdateAnyClient from trusted height 1-12938815 to target height 1-12951594
2023-12-26T18:59:10.738831Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: assembled batch of 2 message(s)
2023-12-26T18:59:10.739010Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}: sending 2 messages as 1 batches to chain nomic-stakenet-3 in parallel
2023-12-26T18:59:10.739024Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: max fee, for use in tx simulation: Fee { amount: "75001unbtc", gas_limit: 3000000 }
2023-12-26T18:59:10.740937Z  WARN ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}:estimate_gas: tx simulation successful but no gas amount used was returned, falling back on default gas: 1000000
2023-12-26T18:59:10.740952Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: send_tx: using 1000000 gas, fee Fee { amount: "32501unbtc", gas_limit: 1300000 } id=nomic-stakenet-3
2023-12-26T18:59:10.766588Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: gas estimation succeeded
2023-12-26T18:59:10.766602Z ERROR ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: failed to broadcast tx with unrecoverable error response=Response { code: Err(1), data: b"", log: "Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``", hash: Hash::Sha256(6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A) } diagnostic=unknown TX sync response error: 1
2023-12-26T18:59:10.766645Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: [Sync->nomic-stakenet-3] result events:
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``") at height 3-1

2023-12-26T18:59:10.766661Z ERROR ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: error ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``")
2023-12-26T18:59:10.766668Z  WARN ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: failed. Regenerate operational data from 1 events
2023-12-26T18:59:10.794214Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: will retry odata=packet-recv ->Destination @1-12951593; len=1
2023-12-26T18:59:10.794230Z DEBUG ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: retrying retry.current=4 retry.max=5
2023-12-26T18:59:10.794239Z DEBUG ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: prepending Destination client update at height 1-12951594
2023-12-26T18:59:10.890486Z DEBUG ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:foreign_client.wait_and_build_update_client_with_trusted{client=osmosis-1->nomic-stakenet-3:07-tendermint-1 target_height=1-12951594}:foreign_client.build_update_client_with_trusted{client=osmosis-1->nomic-stakenet-3:07-tendermint-1 target_height=1-12951594}: building a MsgUpdateAnyClient from trusted height 1-12938815 to target height 1-12951594
2023-12-26T18:59:10.890585Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: assembled batch of 2 message(s)
2023-12-26T18:59:10.890829Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}: sending 2 messages as 1 batches to chain nomic-stakenet-3 in parallel
2023-12-26T18:59:10.890846Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: max fee, for use in tx simulation: Fee { amount: "75001unbtc", gas_limit: 3000000 }
2023-12-26T18:59:10.892802Z  WARN ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}:estimate_gas: tx simulation successful but no gas amount used was returned, falling back on default gas: 1000000
2023-12-26T18:59:10.892822Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: send_tx: using 1000000 gas, fee Fee { amount: "32501unbtc", gas_limit: 1300000 } id=nomic-stakenet-3
2023-12-26T18:59:10.913656Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: gas estimation succeeded
2023-12-26T18:59:10.913670Z ERROR ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: failed to broadcast tx with unrecoverable error response=Response { code: Err(1), data: b"", log: "Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``", hash: Hash::Sha256(6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A) } diagnostic=unknown TX sync response error: 1
2023-12-26T18:59:10.913709Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: [Sync->nomic-stakenet-3] result events:
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``") at height 3-1

2023-12-26T18:59:10.913726Z ERROR ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: error ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``")
2023-12-26T18:59:10.913732Z  WARN ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: failed. Regenerate operational data from 1 events
2023-12-26T18:59:10.936717Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: will retry odata=packet-recv ->Destination @1-12951593; len=1
2023-12-26T18:59:10.936727Z DEBUG ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: retrying retry.current=5 retry.max=5
2023-12-26T18:59:10.936733Z DEBUG ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: prepending Destination client update at height 1-12951594
2023-12-26T18:59:10.982809Z DEBUG ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:foreign_client.wait_and_build_update_client_with_trusted{client=osmosis-1->nomic-stakenet-3:07-tendermint-1 target_height=1-12951594}:foreign_client.build_update_client_with_trusted{client=osmosis-1->nomic-stakenet-3:07-tendermint-1 target_height=1-12951594}: building a MsgUpdateAnyClient from trusted height 1-12938815 to target height 1-12951594
2023-12-26T18:59:10.982907Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: assembled batch of 2 message(s)
2023-12-26T18:59:10.983068Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}: sending 2 messages as 1 batches to chain nomic-stakenet-3 in parallel
2023-12-26T18:59:10.983081Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: max fee, for use in tx simulation: Fee { amount: "75001unbtc", gas_limit: 3000000 }
2023-12-26T18:59:10.985131Z  WARN ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}:estimate_gas: tx simulation successful but no gas amount used was returned, falling back on default gas: 1000000
2023-12-26T18:59:10.985151Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: send_tx: using 1000000 gas, fee Fee { amount: "32501unbtc", gas_limit: 1300000 } id=nomic-stakenet-3
2023-12-26T18:59:11.008366Z DEBUG ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: gas estimation succeeded
2023-12-26T18:59:11.008379Z ERROR ThreadId(34) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: failed to broadcast tx with unrecoverable error response=Response { code: Err(1), data: b"", log: "Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``", hash: Hash::Sha256(6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A) } diagnostic=unknown TX sync response error: 1
2023-12-26T18:59:11.008416Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: [Sync->nomic-stakenet-3] result events:
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``") at height 3-1

2023-12-26T18:59:11.008431Z ERROR ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: error ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 6287434FC016A2DE7688201DEC1038BBB754BE904A79337C826934946BCB6F4A reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: header verification failed with reason: `header trusted next validator set hash does not match hash stored on chain``")
2023-12-26T18:59:11.008436Z ERROR ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-recv ->Destination @1-12951593; len=1}: 5/5 retries exhausted. giving up
2023-12-26T18:59:12.315891Z  INFO ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}: 35 unreceived acknowledgements found: 2117..=2151 
2023-12-26T18:59:25.095127Z  INFO ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}: pulled packet data for 35 events out of 35 sequences: 2117..=2151; events.total=35 events.left=0
2023-12-26T18:59:27.104075Z DEBUG ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:schedule{odata=packet-ack ->Destination @1-12951594; len=35}: connection delay need not be taken into account: client update message will be prepended later
2023-12-26T18:59:27.104100Z DEBUG ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: retrying retry.current=1 retry.max=5
2023-12-26T18:59:27.104108Z DEBUG ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: prepending Destination client update at height 1-12951595
2023-12-26T18:59:27.160069Z DEBUG ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:foreign_client.wait_and_build_update_client_with_trusted{client=osmosis-1->nomic-stakenet-3:07-tendermint-1 target_height=1-12951595}:foreign_client.build_update_client_with_trusted{client=osmosis-1->nomic-stakenet-3:07-tendermint-1 target_height=1-12951595}: building a MsgUpdateAnyClient from trusted height 1-12938815 to target height 1-12951595
2023-12-26T18:59:27.160234Z  INFO ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: assembled batch of 36 message(s)
2023-12-26T18:59:27.160410Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}: sending 36 messages as 2 batches to chain nomic-stakenet-3 in parallel
2023-12-26T18:59:27.160424Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: max fee, for use in tx simulation: Fee { amount: "75001unbtc", gas_limit: 3000000 }
2023-12-26T18:59:27.162772Z  WARN ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}:estimate_gas: tx simulation successful but no gas amount used was returned, falling back on default gas: 1000000
2023-12-26T18:59:27.162787Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: send_tx: using 1000000 gas, fee Fee { amount: "32501unbtc", gas_limit: 1300000 } id=nomic-stakenet-3
2023-12-26T18:59:27.174821Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: gas estimation succeeded
2023-12-26T18:59:27.174832Z ERROR ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: failed to broadcast tx with unrecoverable error response=Response { code: Err(1), data: b"", log: "Unexpected byte: 0", hash: Hash::Sha256(58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE) } diagnostic=unknown TX sync response error: 1
2023-12-26T18:59:27.174864Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: max fee, for use in tx simulation: Fee { amount: "75001unbtc", gas_limit: 3000000 }
2023-12-26T18:59:27.177141Z  WARN ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}:estimate_gas: tx simulation successful but no gas amount used was returned, falling back on default gas: 1000000
2023-12-26T18:59:27.177155Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: send_tx: using 1000000 gas, fee Fee { amount: "32501unbtc", gas_limit: 1300000 } id=nomic-stakenet-3
2023-12-26T18:59:27.236369Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: gas estimation succeeded
2023-12-26T18:59:27.236391Z ERROR ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: failed to broadcast tx with unrecoverable error response=Response { code: Err(1), data: b"", log: "Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``", hash: Hash::Sha256(92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427) } diagnostic=unknown TX sync response error: 1
2023-12-26T18:59:27.236457Z  INFO ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: [Sync->nomic-stakenet-3] result events:
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1

2023-12-26T18:59:27.236548Z ERROR ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: error ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0")
2023-12-26T18:59:27.236605Z  WARN ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: failed. Regenerate operational data from 35 events
2023-12-26T18:59:29.379523Z  INFO ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: will retry odata=packet-ack ->Destination @1-12951594; len=35
2023-12-26T18:59:29.379570Z DEBUG ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: retrying retry.current=2 retry.max=5
2023-12-26T18:59:29.379575Z DEBUG ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: prepending Destination client update at height 1-12951595
2023-12-26T18:59:29.468856Z DEBUG ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:foreign_client.wait_and_build_update_client_with_trusted{client=osmosis-1->nomic-stakenet-3:07-tendermint-1 target_height=1-12951595}:foreign_client.build_update_client_with_trusted{client=osmosis-1->nomic-stakenet-3:07-tendermint-1 target_height=1-12951595}: building a MsgUpdateAnyClient from trusted height 1-12938815 to target height 1-12951595
2023-12-26T18:59:29.468975Z  INFO ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: assembled batch of 36 message(s)
2023-12-26T18:59:29.469142Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}: sending 36 messages as 2 batches to chain nomic-stakenet-3 in parallel
2023-12-26T18:59:29.469153Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: max fee, for use in tx simulation: Fee { amount: "75001unbtc", gas_limit: 3000000 }
2023-12-26T18:59:29.471833Z  WARN ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}:estimate_gas: tx simulation successful but no gas amount used was returned, falling back on default gas: 1000000
2023-12-26T18:59:29.471851Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: send_tx: using 1000000 gas, fee Fee { amount: "32501unbtc", gas_limit: 1300000 } id=nomic-stakenet-3
2023-12-26T18:59:29.482267Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: gas estimation succeeded
2023-12-26T18:59:29.482276Z ERROR ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: failed to broadcast tx with unrecoverable error response=Response { code: Err(1), data: b"", log: "Unexpected byte: 0", hash: Hash::Sha256(58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE) } diagnostic=unknown TX sync response error: 1
2023-12-26T18:59:29.482301Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: max fee, for use in tx simulation: Fee { amount: "75001unbtc", gas_limit: 3000000 }
2023-12-26T18:59:29.484232Z  WARN ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}:estimate_gas: tx simulation successful but no gas amount used was returned, falling back on default gas: 1000000
2023-12-26T18:59:29.484244Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: send_tx: using 1000000 gas, fee Fee { amount: "32501unbtc", gas_limit: 1300000 } id=nomic-stakenet-3
2023-12-26T18:59:29.500981Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: gas estimation succeeded
2023-12-26T18:59:29.500990Z ERROR ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: failed to broadcast tx with unrecoverable error response=Response { code: Err(1), data: b"", log: "Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``", hash: Hash::Sha256(92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427) } diagnostic=unknown TX sync response error: 1
2023-12-26T18:59:29.501018Z  INFO ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: [Sync->nomic-stakenet-3] result events:
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1

2023-12-26T18:59:29.501078Z ERROR ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: error ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0")
2023-12-26T18:59:29.501109Z  WARN ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: failed. Regenerate operational data from 35 events
2023-12-26T18:59:31.574239Z  INFO ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: will retry odata=packet-ack ->Destination @1-12951594; len=35
2023-12-26T18:59:31.574283Z DEBUG ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: retrying retry.current=3 retry.max=5
2023-12-26T18:59:31.574288Z DEBUG ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: prepending Destination client update at height 1-12951595
2023-12-26T18:59:31.710476Z DEBUG ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:foreign_client.wait_and_build_update_client_with_trusted{client=osmosis-1->nomic-stakenet-3:07-tendermint-1 target_height=1-12951595}:foreign_client.build_update_client_with_trusted{client=osmosis-1->nomic-stakenet-3:07-tendermint-1 target_height=1-12951595}: building a MsgUpdateAnyClient from trusted height 1-12938815 to target height 1-12951595
2023-12-26T18:59:31.710606Z  INFO ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: assembled batch of 36 message(s)
2023-12-26T18:59:31.710783Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}: sending 36 messages as 2 batches to chain nomic-stakenet-3 in parallel
2023-12-26T18:59:31.710797Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: max fee, for use in tx simulation: Fee { amount: "75001unbtc", gas_limit: 3000000 }
2023-12-26T18:59:31.713677Z  WARN ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}:estimate_gas: tx simulation successful but no gas amount used was returned, falling back on default gas: 1000000
2023-12-26T18:59:31.713697Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: send_tx: using 1000000 gas, fee Fee { amount: "32501unbtc", gas_limit: 1300000 } id=nomic-stakenet-3
2023-12-26T18:59:31.724464Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: gas estimation succeeded
2023-12-26T18:59:31.724474Z ERROR ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: failed to broadcast tx with unrecoverable error response=Response { code: Err(1), data: b"", log: "Unexpected byte: 0", hash: Hash::Sha256(58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE) } diagnostic=unknown TX sync response error: 1
2023-12-26T18:59:31.724506Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: max fee, for use in tx simulation: Fee { amount: "75001unbtc", gas_limit: 3000000 }
2023-12-26T18:59:31.726475Z  WARN ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}:estimate_gas: tx simulation successful but no gas amount used was returned, falling back on default gas: 1000000
2023-12-26T18:59:31.726491Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: send_tx: using 1000000 gas, fee Fee { amount: "32501unbtc", gas_limit: 1300000 } id=nomic-stakenet-3
2023-12-26T18:59:31.743070Z DEBUG ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: gas estimation succeeded
2023-12-26T18:59:31.743083Z ERROR ThreadId(34) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}:send_messages_and_wait_commit{chain=nomic-stakenet-3 tracking_id=packet-ack}:send_tx_with_account_sequence_retry{chain=nomic-stakenet-3 account.sequence=0}: failed to broadcast tx with unrecoverable error response=Response { code: Err(1), data: b"", log: "Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``", hash: Hash::Sha256(92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427) } diagnostic=unknown TX sync response error: 1
2023-12-26T18:59:31.743131Z  INFO ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: [Sync->nomic-stakenet-3] result events:
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1
	ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 92E0ACDA4D4B207C733A1F0D78EF62F94828DC98EEDF4AC7FE869B76896FE427 reports error: code=Err(1), log="Ibc Error: context error: `ICS02 Client error: the proof height is insufficient: latest_height=`1-12938815` proof_height=`1-12951595``") at height 3-1

2023-12-26T18:59:31.743208Z ERROR ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: error ChainError(check_tx (broadcast_tx_sync) on chain nomic-stakenet-3 for Tx hash 58F974D19D9791BF4246A28693CD163F9CC9CEFE421A396CBB79099C3BCFD4FE reports error: code=Err(1), log="Unexpected byte: 0")
2023-12-26T18:59:31.743261Z  WARN ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: failed. Regenerate operational data from 35 events
2023-12-26T18:59:50.277718Z ERROR ThreadId(01) relay_ack_packet_messages{src_chain=osmosis-1 src_port=transfer src_channel=channel-6897 dst_chain=nomic-stakenet-3}:relay{odata=packet-ack ->Destination @1-12951594; len=35}: failed to regenerate operational data from initial data: packet-ack ->Destination @1-12951594; len=35 with error link failed with underlying error: gRPC call `query_unreceived_acknowledgements` failed with status: status: Aborted, message: "HTTP error\n\nCaused by:\n    connection closed before message completed\n\nLocation:\n    /home/nomic/.cargo/registry/src/index.crates.io-6f17d22bba15001f/flex-error-0.4.4/src/tracer_impl/eyre.rs:10:9", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 26 Dec 2023 18:59:50 GMT", "content-length": "0"} }, discarding this op. data
SUCCESS []

hermes config: nomic proof_specs

proof_specs = '''
[
  {
    "inner_spec": {
      "child_order": [
        0,
        1,
        2
      ],
      "child_size": 32,
      "empty_child": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
      "min_prefix_length": 1,
      "max_prefix_length": 1,
      "hash": 6
    },
    "leaf_spec": {
      "hash": 6,
      "prehash_key": 0,
      "prehash_value": 0,
      "length": 4,
      "prefix": "AA"
    },
    "max_depth": 0,
    "min_depth": 0
  },
  {
    "inner_spec": {
      "child_order": [
        0
      ],
      "child_size": 32,
      "empty_child": "",
      "min_prefix_length": 0,
      "max_prefix_length": 0,
      "hash": 6
    },
    "leaf_spec": {
      "hash": 6,
      "prehash_key": 0,
      "prehash_value": 0,
      "length": 0,
      "prefix": ""
    },
    "max_depth": 0,
    "min_depth": 0
  }
]'''
@perfogic
Copy link

Hello, this error came from the convention rule of old version ibc-rs. Your chain has to be osmosis-local-0 for example!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment