I ran into an issue when trying to create the channel due to a mismatch of keys. The output below shows the error that I received. The issue was - the keys generated via gm keys were not in sync with the keys that hermes had known about.
hermes create channel --a-chain ibc-0 --b-chain agoriclocal --a-port transfer --b-port pegasus --new-client-connection
2023-05-31T22:18:46.926628Z INFO ThreadId(01) using default configuration from '/Users/tgreco/.hermes/config.toml'
2023-05-31T22:18:46.926810Z INFO ThreadId(01) running Hermes v1.4.0
WARN: Are you sure you want a new connection & clients to be created? Hermes will use default security parameters.
Hint: Consider using the default invocation
hermes create channel --a-port <PORT-ID> --b-port <PORT-ID> --a-chain <CHAIN-A-ID> --a-connection <CONNECTION-A-ID>
WARN: Are you sure you want a new connection & clients to be created? Hermes will use default security parameters.
Hint: Consider using the default invocation
hermes create channel --a-port <PORT-ID> --b-port <PORT-ID> --a-chain <CHAIN-A-ID> --a-connection <CONNECTION-A-ID>
to re-use a pre-existing connection. yes
2023-05-31T22:18:48.298113Z INFO ThreadId(01) Creating new clients, new connection, and a new channel with order ORDER_UNORDERED
2023-05-31T22:18:48.317640Z ERROR ThreadId(01) foreign_client.create{client=agoriclocal->ibc-0:07-tendermint-0}: failed to create client: error raised while creating client for chain ibc-0: failed sending message to dst chain : gRPC call failed with status: status: NotFound, message: "account cosmos1jjrue7k7c90lpw5l9agge6d650hx99aarjssz4 not found", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "601"} }
ERROR error raised while creating client for chain ibc-0: failed sending message to dst chain : gRPC call failed with status: status: NotFound, message: "account cosmos1jjrue7k7c90lpw5l9agge6d650hx99aarjssz4 not found", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "601"} }In order to get past this I had to run the following command
hermes keys add --chain ibc-0 --mnemonic-file ./.gm/ibc-0/wallet_seed.json --overwriteAfter registering the key with hermes I was fine to set up the new client connection.
I noticed that this added a key for the account "wallet", whereas the key name in the hermes config is "wallet1". If the student is using the gm tool for the first time, then I think the "key" should be "wallet".
[[chains]]
id = 'ibc-0'
rpc_addr = 'http://localhost:27030'
grpc_addr = 'http://localhost:27032'
websocket_addr = 'ws://localhost:27030/websocket'
rpc_timeout = '15s'
account_prefix = 'cosmos'
key_name = 'wallet'The student is instructed to get the "wallet" address by running the following command:
gaiad --home ~/.gm/ibc-0 keys --keyring-backend="test" show wallet -aThis works fine, as the key that I have associated with hermes as well as my gaia instance has the key "wallet". This key issue comes up again checking the balance. I'm trying to query information about the "wallet1" key as opposed to the "wallet" key which maps to the address that actually receives the funds.
Instead of using this script:
gaiad --node tcp://localhost:27030 query bank balances $(gaiad --home ~/.gm/ibc-0 keys --keyring-backend="test" show wallet1 -a)I needed to use this script:
gaiad --node tcp://localhost:27030 query bank balances $(gaiad --home ~/.gm/ibc-0 keys --keyring-backend="test" show wallet -a)In the initial phases of doing the IBC transaction from agoric, the demo instructs students to add the cosmos address to the codeblock below. I interpreted this to mean that any of the additional properties within the object being exported could be removed.
Copy and past it's output to pegasus/scripts/deploy-config.js file's cosmos.address location:
export default {
agoric: {
channel: 'channel-0',
localPegId: 'peg-channel-0-IST',
},
cosmos: {
address: [DESTINATION_ADDRESS] // Paste your output here!!!
},
};This was fine at first, but when I went to run hermes ft-transfer I ran into an issue as to the values I had removed were needed in order to run the deploy-peg-remote script. Without the rest of the properties, the ag-solo wallet is unable to handle the setup for the local samolean issue.
This feedback isn't specific to your demo but I thought it was important to make note of it.
The errors below all come from the log following the execution of hermes start. I noticed various issues arising when trying to send IBC transactions, and they all seemed to have stemmed from the agoriclocal chain not having minimum gas price config set.
If you search for "error" in the output below you'll see several logs similiar to this one:
2023-05-31T22:37:06.212096Z ERROR ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=d6cdcb6b ->Destination @0-185; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=d6cdcb6b}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=48}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 61, got 48: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '48801' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "399"} }I'm not sure if this is because of a minimum gas price, or if it's something else, but I thought it was worth making note of in case any students run into similiar errors.
2023-05-31T22:27:26.714373Z INFO ThreadId(01) using default configuration from '/Users/tgreco/.hermes/config.toml'
2023-05-31T22:27:26.716108Z INFO ThreadId(01) running Hermes v1.4.0
2023-05-31T22:27:26.720324Z INFO ThreadId(01) telemetry: telemetry service running, exposing metrics at http://127.0.0.1:3001/metrics
2023-05-31T22:27:26.720449Z INFO ThreadId(01) rest: REST server disabled
2023-05-31T22:27:26.743160Z WARN ThreadId(25) health_check{chain=ibc-0}: Chain 'ibc-0' has no minimum gas price value configured for denomination 'stake'. This is usually a sign of misconfiguration, please check your chain and relayer configurations
2023-05-31T22:27:26.750030Z INFO ThreadId(01) health_check{chain=ibc-0}: chain is healthy
2023-05-31T22:27:26.753151Z WARN ThreadId(26) health_check{chain=agoriclocal}: Chain 'agoriclocal' has no minimum gas price value configured for denomination 'ubld'. This is usually a sign of misconfiguration, please check your chain and relayer configurations
2023-05-31T22:27:26.757636Z INFO ThreadId(01) health_check{chain=agoriclocal}: chain is healthy
2023-05-31T22:27:26.760047Z INFO ThreadId(01) scan.chain{chain=ibc-0}: scanning chain...
2023-05-31T22:27:26.760097Z INFO ThreadId(01) scan.chain{chain=ibc-0}: scanning chain for all clients, connections and channels
2023-05-31T22:27:26.760104Z INFO ThreadId(01) scan.chain{chain=ibc-0}: scanning all clients...
2023-05-31T22:27:26.762007Z INFO ThreadId(01) scan.chain{chain=ibc-0}:scan.client{client=07-tendermint-0}: scanning client...
2023-05-31T22:27:26.764459Z INFO ThreadId(01) scan.chain{chain=ibc-0}:scan.client{client=07-tendermint-0}:scan.connection{connection=connection-0}: scanning connection...
2023-05-31T22:27:26.768201Z INFO ThreadId(01) scan.chain{chain=agoriclocal}: scanning chain...
2023-05-31T22:27:26.768234Z INFO ThreadId(01) scan.chain{chain=agoriclocal}: scanning chain for all clients, connections and channels
2023-05-31T22:27:26.768237Z INFO ThreadId(01) scan.chain{chain=agoriclocal}: scanning all clients...
2023-05-31T22:27:26.769278Z INFO ThreadId(01) scan.chain{chain=agoriclocal}:scan.client{client=07-tendermint-0}: scanning client...
2023-05-31T22:27:26.770162Z INFO ThreadId(01) scan.chain{chain=agoriclocal}:scan.client{client=07-tendermint-0}:scan.connection{connection=connection-0}: scanning connection...
2023-05-31T22:27:26.771718Z INFO ThreadId(01) scanned chains:
2023-05-31T22:27:26.771724Z INFO ThreadId(01) # Chain: ibc-0
- Client: 07-tendermint-0
* Connection: connection-0
| State: OPEN
| Counterparty state: OPEN
+ Channel: channel-0
| Port: transfer
| State: OPEN
| Counterparty: channel-0
# Chain: agoriclocal
- Client: 07-tendermint-0
* Connection: connection-0
| State: OPEN
| Counterparty state: OPEN
+ Channel: channel-0
| Port: pegasus
| State: OPEN
| Counterparty: channel-0
2023-05-31T22:27:26.771758Z INFO ThreadId(01) spawn:chain{chain=ibc-0}:client{client=07-tendermint-0}:connection{connection=connection-0}: connection is OPEN, state on destination chain is OPEN chain=ibc-0 connection=connection-0 counterparty_chain=agoriclocal
2023-05-31T22:27:26.771765Z INFO ThreadId(01) spawn:chain{chain=ibc-0}:client{client=07-tendermint-0}:connection{connection=connection-0}: connection is already open, not spawning Connection worker chain=ibc-0 connection=connection-0
2023-05-31T22:27:26.771829Z INFO ThreadId(01) spawn:chain{chain=ibc-0}:client{client=07-tendermint-0}:connection{connection=connection-0}: no connection workers were spawn chain=ibc-0 connection=connection-0
2023-05-31T22:27:26.772958Z INFO ThreadId(01) spawn:chain{chain=ibc-0}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}: channel is OPEN, state on destination chain is OPEN chain=ibc-0 counterparty_chain=agoriclocal channel=channel-0
2023-05-31T22:27:26.774887Z INFO ThreadId(01) spawn:chain{chain=ibc-0}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}: spawned client worker: client::agoriclocal->ibc-0:07-tendermint-0
2023-05-31T22:27:26.779385Z INFO ThreadId(01) spawn:chain{chain=ibc-0}:client{client=07-tendermint-0}:connection{connection=connection-0}: done spawning channel workers chain=ibc-0 channel=channel-0
2023-05-31T22:27:26.779433Z INFO ThreadId(01) spawn:chain{chain=ibc-0}: spawning Wallet worker: wallet::ibc-0
2023-05-31T22:27:26.779450Z INFO ThreadId(01) spawn:chain{chain=agoriclocal}:client{client=07-tendermint-0}:connection{connection=connection-0}: connection is OPEN, state on destination chain is OPEN chain=agoriclocal connection=connection-0 counterparty_chain=ibc-0
2023-05-31T22:27:26.779459Z INFO ThreadId(01) spawn:chain{chain=agoriclocal}:client{client=07-tendermint-0}:connection{connection=connection-0}: connection is already open, not spawning Connection worker chain=agoriclocal connection=connection-0
2023-05-31T22:27:26.779500Z INFO ThreadId(01) spawn:chain{chain=agoriclocal}:client{client=07-tendermint-0}:connection{connection=connection-0}: no connection workers were spawn chain=agoriclocal connection=connection-0
2023-05-31T22:27:26.779515Z INFO ThreadId(01) spawn:chain{chain=agoriclocal}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}: channel is OPEN, state on destination chain is OPEN chain=agoriclocal counterparty_chain=ibc-0 channel=channel-0
2023-05-31T22:27:26.782108Z INFO ThreadId(01) spawn:chain{chain=agoriclocal}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}: spawned client worker: client::ibc-0->agoriclocal:07-tendermint-0
2023-05-31T22:27:26.785823Z INFO ThreadId(01) spawn:chain{chain=agoriclocal}:client{client=07-tendermint-0}:connection{connection=connection-0}: done spawning channel workers chain=agoriclocal channel=channel-0
2023-05-31T22:27:26.785876Z INFO ThreadId(01) spawn:chain{chain=agoriclocal}: spawning Wallet worker: wallet::agoriclocal
2023-05-31T22:27:26.792890Z INFO ThreadId(01) Hermes has started
2023-05-31T22:27:28.119178Z INFO ThreadId(28) spawn:chain{chain=ibc-0}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=agoriclocal dst_chain=ibc-0}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=None client=agoriclocal->ibc-0:07-tendermint-0}: evidence submission result: [ ]
2023-05-31T22:27:28.119205Z INFO ThreadId(28) spawn:chain{chain=ibc-0}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=agoriclocal dst_chain=ibc-0}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=None client=agoriclocal->ibc-0:07-tendermint-0}: client is valid
2023-05-31T22:27:28.745783Z INFO ThreadId(31) spawn:chain{chain=agoriclocal}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=ibc-0 dst_chain=agoriclocal}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=None client=ibc-0->agoriclocal:07-tendermint-0}: evidence submission result: [ ]
2023-05-31T22:27:28.745810Z INFO ThreadId(31) spawn:chain{chain=agoriclocal}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=ibc-0 dst_chain=agoriclocal}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=None client=ibc-0->agoriclocal:07-tendermint-0}: client is valid
2023-05-31T22:33:18.418005Z INFO ThreadId(184) worker.batch{chain=agoriclocal}:supervisor.handle_batch{chain=agoriclocal}:supervisor.process_batch{chain=agoriclocal}:worker.packet.cmd{src_chain=agoriclocal src_port=pegasus src_channel=channel-0 dst_chain=ibc-0}:schedule_packet_clearing{height=Some(Height { revision: 0, height: 353 })}:relay_pending_packets{height=Some(Height { revision: 0, height: 352 })}:schedule_recv_packet_and_timeout_msgs{query_height=0-352}: pulled packet data for 0 events out of 1 sequences: 1..=1; events.total=1 events.left=0
2023-05-31T22:33:18.440513Z INFO ThreadId(184) worker.batch{chain=agoriclocal}:supervisor.handle_batch{chain=agoriclocal}:supervisor.process_batch{chain=agoriclocal}:worker.packet.cmd{src_chain=agoriclocal src_port=pegasus src_channel=channel-0 dst_chain=ibc-0}:relay{odata=303e0f73 ->Destination @0-353; len=1}: assembled batch of 2 message(s)
2023-05-31T22:33:18.448174Z INFO ThreadId(184) worker.batch{chain=agoriclocal}:supervisor.handle_batch{chain=agoriclocal}:supervisor.process_batch{chain=agoriclocal}:worker.packet.cmd{src_chain=agoriclocal src_port=pegasus src_channel=channel-0 dst_chain=ibc-0}:relay{odata=303e0f73 ->Destination @0-353; len=1}: response(s): 1; Ok:3D5A58CA062BF54162190BD09331CC556B5610DB5D9ED663DA5278F36AAB7DDB target_chain=ibc-0
2023-05-31T22:33:18.448200Z INFO ThreadId(184) worker.batch{chain=agoriclocal}:supervisor.handle_batch{chain=agoriclocal}:supervisor.process_batch{chain=agoriclocal}:worker.packet.cmd{src_chain=agoriclocal src_port=pegasus src_channel=channel-0 dst_chain=ibc-0}:relay{odata=303e0f73 ->Destination @0-353; len=1}: submitted
2023-05-31T22:33:24.848648Z INFO ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:schedule_packet_clearing{height=Some(Height { revision: 0, height: 141 })}:relay_pending_packets{height=Some(Height { revision: 0, height: 140 })}:build_packet_ack_msgs{query_height=0-140}: pulled packet data for 0 events out of 1 sequences: 1..=1; events.total=1 events.left=0
2023-05-31T22:33:24.865886Z INFO ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=66b21a4c ->Destination @0-141; len=1}: assembled batch of 2 message(s)
2023-05-31T22:33:24.869198Z ERROR ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=66b21a4c ->Destination @0-141; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=66b21a4c}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=48}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 49, got 48: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '48803' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "355"} }
2023-05-31T22:33:24.869251Z WARN ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=66b21a4c ->Destination @0-141; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=66b21a4c}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=48}: failed to estimate gas because of a mismatched account sequence number, refreshing account sequence number and retrying once error=gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 49, got 48: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '48803' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "355"} }
2023-05-31T22:33:24.874495Z INFO ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=66b21a4c ->Destination @0-141; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=66b21a4c}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=48}: refresh: retrieved account sequence=48 number=11
2023-05-31T22:33:25.177172Z ERROR ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=66b21a4c ->Destination @0-141; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=66b21a4c}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=48}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 49, got 48: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '48803' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "355"} }
2023-05-31T22:33:25.177244Z ERROR ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}: will retry: schedule execution encountered error: link failed with underlying error: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 49, got 48: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '48803' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "355"} }
2023-05-31T22:33:25.177334Z WARN ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}: task encountered ignorable error: link error: link failed with underlying error: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 49, got 48: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '48803' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "355"} }
2023-05-31T22:33:25.284171Z INFO ThreadId(28) spawn:chain{chain=ibc-0}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=agoriclocal dst_chain=ibc-0}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=Some(UpdateClient { common: Attributes { client_id: ClientId("07-tendermint-0"), client_type: Tendermint, consensus_height: Height { revision: 0, height: 354 } }, header: Some( Header {...}) }) client=agoriclocal->ibc-0:07-tendermint-0}: evidence submission result: [ ]
2023-05-31T22:33:25.284193Z INFO ThreadId(28) spawn:chain{chain=ibc-0}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=agoriclocal dst_chain=ibc-0}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=Some(UpdateClient { common: Attributes { client_id: ClientId("07-tendermint-0"), client_type: Tendermint, consensus_height: Height { revision: 0, height: 354 } }, header: Some( Header {...}) }) client=agoriclocal->ibc-0:07-tendermint-0}: client is valid
2023-05-31T22:37:00.024791Z INFO ThreadId(184) worker.batch{chain=agoriclocal}:supervisor.handle_batch{chain=agoriclocal}:supervisor.process_batch{chain=agoriclocal}:worker.packet.cmd{src_chain=agoriclocal src_port=pegasus src_channel=channel-0 dst_chain=ibc-0}:relay{odata=fbad75ef ->Destination @0-397; len=1}: assembled batch of 2 message(s)
2023-05-31T22:37:00.030039Z INFO ThreadId(184) worker.batch{chain=agoriclocal}:supervisor.handle_batch{chain=agoriclocal}:supervisor.process_batch{chain=agoriclocal}:worker.packet.cmd{src_chain=agoriclocal src_port=pegasus src_channel=channel-0 dst_chain=ibc-0}:relay{odata=fbad75ef ->Destination @0-397; len=1}: response(s): 1; Ok:705B413B0076795A50F17F187B3858DD05A00428232FB2498CD1D6FD83122FF2 target_chain=ibc-0
2023-05-31T22:37:00.030059Z INFO ThreadId(184) worker.batch{chain=agoriclocal}:supervisor.handle_batch{chain=agoriclocal}:supervisor.process_batch{chain=agoriclocal}:worker.packet.cmd{src_chain=agoriclocal src_port=pegasus src_channel=channel-0 dst_chain=ibc-0}:relay{odata=fbad75ef ->Destination @0-397; len=1}: submitted
2023-05-31T22:37:06.210311Z INFO ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=d6cdcb6b ->Destination @0-185; len=1}: assembled batch of 2 message(s)
2023-05-31T22:37:06.212096Z ERROR ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=d6cdcb6b ->Destination @0-185; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=d6cdcb6b}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=48}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 61, got 48: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '48801' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "399"} }
2023-05-31T22:37:06.212139Z WARN ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=d6cdcb6b ->Destination @0-185; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=d6cdcb6b}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=48}: failed to estimate gas because of a mismatched account sequence number, refreshing account sequence number and retrying once error=gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 61, got 48: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '48801' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "399"} }
2023-05-31T22:37:06.212964Z INFO ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=d6cdcb6b ->Destination @0-185; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=d6cdcb6b}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=48}: refresh: retrieved account sequence=60 number=11
2023-05-31T22:37:06.515242Z ERROR ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=d6cdcb6b ->Destination @0-185; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=d6cdcb6b}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=48}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 61, got 60: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '48801' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "399"} }
2023-05-31T22:37:06.515309Z ERROR ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}: will retry: schedule execution encountered error: link failed with underlying error: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 61, got 60: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '48801' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "399"} }
2023-05-31T22:37:06.515397Z WARN ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}: task encountered ignorable error: link error: link failed with underlying error: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 61, got 60: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '48801' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "399"} }
2023-05-31T22:37:06.625510Z INFO ThreadId(28) spawn:chain{chain=ibc-0}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=agoriclocal dst_chain=ibc-0}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=Some(UpdateClient { common: Attributes { client_id: ClientId("07-tendermint-0"), client_type: Tendermint, consensus_height: Height { revision: 0, height: 398 } }, header: Some( Header {...}) }) client=agoriclocal->ibc-0:07-tendermint-0}: evidence submission result: [ ]
2023-05-31T22:37:06.625541Z INFO ThreadId(28) spawn:chain{chain=ibc-0}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=agoriclocal dst_chain=ibc-0}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=Some(UpdateClient { common: Attributes { client_id: ClientId("07-tendermint-0"), client_type: Tendermint, consensus_height: Height { revision: 0, height: 398 } }, header: Some( Header {...}) }) client=agoriclocal->ibc-0:07-tendermint-0}: client is valid
2023-05-31T22:38:21.269963Z INFO ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:schedule_packet_clearing{height=Some(Height { revision: 0, height: 200 })}:relay_pending_packets{height=Some(Height { revision: 0, height: 199 })}:build_packet_ack_msgs{query_height=0-199}: pulled packet data for 2 events out of 2 sequences: 1..=2; events.total=2 events.left=0
2023-05-31T22:38:21.284527Z INFO ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=cleared/e5768f0a ->Destination @0-199; len=2}: assembled batch of 3 message(s)
2023-05-31T22:38:21.286218Z ERROR ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=cleared/e5768f0a ->Destination @0-199; len=2}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=cleared/e5768f0a}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=60}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 61, got 60: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '49740' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "414"} }
2023-05-31T22:38:21.286249Z WARN ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=cleared/e5768f0a ->Destination @0-199; len=2}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=cleared/e5768f0a}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=60}: failed to estimate gas because of a mismatched account sequence number, refreshing account sequence number and retrying once error=gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 61, got 60: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '49740' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "414"} }
2023-05-31T22:38:21.287097Z INFO ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=cleared/e5768f0a ->Destination @0-199; len=2}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=cleared/e5768f0a}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=60}: refresh: retrieved account sequence=61 number=11
2023-05-31T22:38:21.599887Z INFO ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=cleared/e5768f0a ->Destination @0-199; len=2}: response(s): 1; Ok:735D5AF7E25BAEBB463CAFD3694E28E6569BD0F0E87215664BD3C2D31518F349 target_chain=agoriclocal
2023-05-31T22:38:21.599911Z INFO ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=cleared/e5768f0a ->Destination @0-199; len=2}: submitted
2023-05-31T22:38:30.582970Z INFO ThreadId(31) spawn:chain{chain=agoriclocal}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=ibc-0 dst_chain=agoriclocal}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=Some(UpdateClient { common: Attributes { client_id: ClientId("07-tendermint-0"), client_type: Tendermint, consensus_height: Height { revision: 0, height: 200 } }, header: Some( Header {...}) }) client=ibc-0->agoriclocal:07-tendermint-0}: evidence submission result: [ ]
2023-05-31T22:38:30.582996Z INFO ThreadId(31) spawn:chain{chain=agoriclocal}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=ibc-0 dst_chain=agoriclocal}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=Some(UpdateClient { common: Attributes { client_id: ClientId("07-tendermint-0"), client_type: Tendermint, consensus_height: Height { revision: 0, height: 200 } }, header: Some( Header {...}) }) client=ibc-0->agoriclocal:07-tendermint-0}: client is valid
2023-05-31T23:38:07.154086Z INFO ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=808b8d5e ->Destination @0-913; len=1}: assembled batch of 2 message(s)
2023-05-31T23:38:07.157196Z ERROR ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=808b8d5e ->Destination @0-913; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=808b8d5e}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=61}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 67, got 61: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '48776' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "1127"} }
2023-05-31T23:38:07.157282Z WARN ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=808b8d5e ->Destination @0-913; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=808b8d5e}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=61}: failed to estimate gas because of a mismatched account sequence number, refreshing account sequence number and retrying once error=gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 67, got 61: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '48776' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "1127"} }
2023-05-31T23:38:07.158515Z INFO ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=808b8d5e ->Destination @0-913; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=808b8d5e}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=61}: refresh: retrieved account sequence=67 number=11
2023-05-31T23:38:07.463378Z INFO ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=808b8d5e ->Destination @0-913; len=1}: response(s): 1; Ok:01E16DBBD9835DE548BCF7A22DCB7B9A9775DD26A303A29F1C174A9271116BD4 target_chain=agoriclocal
2023-05-31T23:38:07.463402Z INFO ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=808b8d5e ->Destination @0-913; len=1}: submitted
2023-05-31T23:38:14.665220Z INFO ThreadId(31) spawn:chain{chain=agoriclocal}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=ibc-0 dst_chain=agoriclocal}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=Some(UpdateClient { common: Attributes { client_id: ClientId("07-tendermint-0"), client_type: Tendermint, consensus_height: Height { revision: 0, height: 914 } }, header: Some( Header {...}) }) client=ibc-0->agoriclocal:07-tendermint-0}: evidence submission result: [ ]
2023-05-31T23:38:14.665250Z INFO ThreadId(31) spawn:chain{chain=agoriclocal}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=ibc-0 dst_chain=agoriclocal}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=Some(UpdateClient { common: Attributes { client_id: ClientId("07-tendermint-0"), client_type: Tendermint, consensus_height: Height { revision: 0, height: 914 } }, header: Some( Header {...}) }) client=ibc-0->agoriclocal:07-tendermint-0}: client is valid
^[2023-05-31T23:40:42.815132Z INFO ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=34e8a6de ->Destination @0-944; len=1}: assembled batch of 2 message(s)
2023-05-31T23:40:42.817372Z ERROR ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=34e8a6de ->Destination @0-944; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=34e8a6de}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=67}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 69, got 67: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '48820' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "1158"} }
2023-05-31T23:40:42.817421Z WARN ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=34e8a6de ->Destination @0-944; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=34e8a6de}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=67}: failed to estimate gas because of a mismatched account sequence number, refreshing account sequence number and retrying once error=gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 69, got 67: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '48820' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "1158"} }
2023-05-31T23:40:42.818568Z INFO ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=34e8a6de ->Destination @0-944; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=34e8a6de}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=67}: refresh: retrieved account sequence=69 number=11
2023-05-31T23:40:43.123201Z INFO ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=34e8a6de ->Destination @0-944; len=1}: response(s): 1; Ok:AF98DE97D565B22EA4F806AEF4A8792C2BF4643DFB77E86B23095E40D1620E25 target_chain=agoriclocal
2023-05-31T23:40:43.123226Z INFO ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=34e8a6de ->Destination @0-944; len=1}: submitted
2023-05-31T23:40:50.767221Z INFO ThreadId(31) spawn:chain{chain=agoriclocal}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=ibc-0 dst_chain=agoriclocal}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=Some(UpdateClient { common: Attributes { client_id: ClientId("07-tendermint-0"), client_type: Tendermint, consensus_height: Height { revision: 0, height: 945 } }, header: Some( Header {...}) }) client=ibc-0->agoriclocal:07-tendermint-0}: evidence submission result: [ ]
2023-05-31T23:40:50.767248Z INFO ThreadId(31) spawn:chain{chain=agoriclocal}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=ibc-0 dst_chain=agoriclocal}:foreign_client.detect_misbehaviour_and_submit_evidence{update_event=Some(UpdateClient { common: Attributes { client_id: ClientId("07-tendermint-0"), client_type: Tendermint, consensus_height: Height { revision: 0, height: 945 } }, header: Some( Header {...}) }) client=ibc-0->agoriclocal:07-tendermint-0}: client is valid
2023-06-01T00:39:18.770147Z INFO ThreadId(204) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=0909bca1 ->Destination @0-1643; len=1}: assembled batch of 2 message(s)
2023-06-01T00:39:18.771992Z ERROR ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=0909bca1 ->Destination @0-1643; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=0909bca1}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=69}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 85, got 69: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '48818' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "1857"} }
2023-06-01T00:39:18.772032Z WARN ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=0909bca1 ->Destination @0-1643; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=0909bca1}:send_tx_with_account_sequence_retry{chain=agoriclocal account.sequence=69}: failed to estimate gas because of a mismatched account sequence number, refreshing account sequence number and retrying once error=gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 85, got 69: incorrect account sequence [agoric-labs/cosmos-sdk@v0.45.11-alpha.agoric.1.0.20230323035240-8551678e04db/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '48818' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "1857"} }
2023-06-01T00:39:18.772942Z INFO ThreadId(26) worker.batch{chain=ibc-0}:supervisor.handle_batch{chain=ibc-0}:supervisor.process_batch{chain=ibc-0}:worker.packet.cmd{src_chain=ibc-0 src_port=transfer src_channel=channel-0 dst_chain=agoriclocal}:relay{odata=0909bca1 ->Destination @0-1643; len=1}:send_messages_and_wait_check_tx{chain=agoriclocal tracking_id=0909bca1}:send_tx_with_accountAfter hitting "accept" on the IBC transnfers, I received the following errors. On the UI, it showed that my IST purse had subtracted the amount I was sending to the ibc-0 chain. I inspected the logs and found the following:
2023-05-31T22:37:09.626Z SwingSet: vat: v10: RemoteError(error:liveSlots:v15#70004)#4: (an object) was not a live payment for brand "[Alleged: IST brand]". It could be a used-up payment, a payment for another brand, or it might not be a payment at all.
2023-05-31T22:37:09.626Z SwingSet: vat: v10: Error: (an object) was not a live payment for brand "[Alleged: IST brand]". It could be a used-up payment, a payment for another brand, or it might not be a payment at all.
at construct ()
at Error (/bundled-source/.../node_modules/ses/src/error/tame-error-constructor.js:56)
at makeError (/bundled-source/.../node_modules/ses/src/error/assert.js:243)
at decodeErrorCommon (/bundled-source/.../node_modules/@endo/marshal/src/marshal.js:281)
at decodeFromSmallcaps (/bundled-source/.../node_modules/@endo/marshal/src/encodeToSmallcaps.js:434)
at fromCapData (/bundled-source/.../node_modules/@endo/marshal/src/marshal.js:356)
at notifyOnePromise (/bundled-source/.../packages/swingset-liveslots/src/liveslots.js:1216)
at notify (/bundled-source/.../packages/swingset-liveslots/src/liveslots.js:1240)
at dispatchToUserspace (/bundled-source/.../packages/swingset-liveslots/src/liveslots.js:1505)
at runWithoutMetering (/bundled-source/.../packages/swingset-xsnap-supervisor/lib/supervisor-subprocess-xsnap.js:61)
at ()
2023-05-31T22:37:09.640Z outbound: invoking deliverator; 0 new messages for e178199072a59a6a07dff48154cadb41a0dd043b811587354a378493ab3319d0
2023-05-31T22:38:25.168Z SwingSet: xsnap: v10: UnhandledPromiseRejectionWarning: (RemoteError(error:liveSlots:v7#70005)#5)
2023-05-31T22:38:25.168Z SwingSet: xsnap: v10: RemoteError(error:liveSlots:v7#70005)#5: Cannot read past end of iteration.
2023-05-31T22:38:25.168Z SwingSet: xsnap: v10: Error: Cannot read past end of iteration.
at construct ()
at Error (/bundled-source/.../node_modules/ses/src/error/tame-error-constructor.js:56)
at makeError (/bundled-source/.../node_modules/ses/src/error/assert.js:243)
at decodeErrorCommon (/bundled-source/.../node_modules/@endo/marshal/src/marshal.js:281)
at decodeFromSmallcaps (/bundled-source/.../node_modules/@endo/marshal/src/encodeToSmallcaps.js:434)
at fromCapData (/bundled-source/.../node_modules/@endo/marshal/src/marshal.js:356)
at notifyOnePromise (/bundled-source/.../packages/swingset-liveslots/src/liveslots.js:1216)
at notify (/bundled-source/.../packages/swingset-liveslots/src/liveslots.js:1240)
at dispatchToUserspace (/bundled-source/.../packages/swingset-liveslots/src/liveslots.js:1505)
at runWithoutMetering (/bundled-source/.../packages/swingset-xsnap-supervisor/lib/supervisor-subprocess-xsnap.js:61)
at ()
2023-05-31T22:38:25.176Z SwingSet: xsnap: v10: UnhandledPromiseRejectionWarning: (RemoteError(error:liveSlots:v7#70006)#6)
2023-05-31T22:38:25.176Z SwingSet: xsnap: v10: RemoteError(error:liveSlots:v7#70006)#6: Cannot read past end of iteration.
2023-05-31T22:38:25.176Z SwingSet: xsnap: v10: Error: Cannot read past end of iteration.
at construct ()
at Error (/bundled-source/.../node_modules/ses/src/error/tame-error-constructor.js:56)
at makeError (/bundled-source/.../node_modules/ses/src/error/assert.js:243)
at decodeErrorCommon (/bundled-source/.../node_modules/@endo/marshal/src/marshal.js:281)
at decodeFromSmallcaps (/bundled-source/.../node_modules/@endo/marshal/src/encodeToSmallcaps.js:434)
at fromCapData (/bundled-source/.../node_modules/@endo/marshal/src/marshal.js:356)
at notifyOnePromise (/bundled-source/.../packages/swingset-liveslots/src/liveslots.js:1216)
at notify (/bundled-source/.../packages/swingset-liveslots/src/liveslots.js:1240)
at dispatchToUserspace (/bundled-source/.../packages/swingset-liveslots/src/liveslots.js:1505)
at runWithoutMetering (/bundled-source/.../packages/swingset-xsnap-supervisor/lib/supervisor-subprocess-xsnap.js:61)
at ()After these errors, my purse showed my IST balance to be back at the initial amount i started with. I thought that the transaction failed so it made sense that my IST balance was back where it was when it had started. I made a second attempt at executing agoric deploy scripts/deploy-peg-local.js scripts/deploy-ibc-send.js, and I followed up by accepting the offer to send.
I stepped away and had some dinner, and when I had come back to my computer, I noticed that my IST balance had been subtracted by the amounts sent in both my first and second attempt. This is where things got quite weird because as I just mentioned, the initial IST i sent out had been put back into my purse.
To confirm, I checked my balances and got the output below. This shows that both IBC transactions did in fact go through, however they were received as entirely different denoms despite both being IST.
gaiad --node tcp://localhost:27030 query bank balances $(gaiad --home ~/.gm/ibc-0 keys --keyring-backend="test" show wallet -a)
balances:
- amount: "333333333"
denom: ibc/49354C7C6EB9AA687C4F924B3B08CBB138D38D941E9D00EBA78398F3C46E9A15
- amount: "333333333"
denom: ibc/792B2F2AD2AB87AFB998D1C0FE8F993AA7EEDA9FCE81E602EB2FC596FA393722
- amount: "88000000"
denom: samoleans
- amount: "99998302"
denom: stake
pagination:
next_key: null
total: "0"
context: