Problem with private node

problem_with_private_node.md

Hardware

I have upgraded the private node to a AWS t3a.large with 8GB Ram. It crashes after the same minutes, I haven't noticed any improvement.

Swap

• Swap space: 8GB.

I have monitored it while the private node is syncing, and just before it crashes, and it's barely used (a few KB).

Config used

<user>@<private-node> ~/tezos/tezos-node config show
{ "data-dir": "/ebsvolume/.tezos-node",
  "rpc": { "listen-addrs": [ "[::]:8732" ] },
  "p2p":
    { "bootstrap-peers": [ "<public-IP>:9732" ],
      "listen-addr": "[::]:9732", "private-mode": true,
      "limits":
        { "connection-timeout": 10, "min-connections": 0,
          "expected-connections": 1, "max-connections": 1,
          "max_known_points": [ 8, 6 ], "max_known_peer_ids": [ 8, 6 ] } },
  "log": { "output": "/ebsvolume/log/tezos.log" },
  "shell": { "chain_validator": { "bootstrap_threshold": 1 } } }

Connection problem

First I ran this:

<user>@<private-node> ~/tezos/tezos-admin-client trust address <public-node>:9732
Address <public-node>:9732 is now trusted.
<user>@<private-node> ~/tezos/tezos-admin-client connect address <public-node>:9732
Connection to <public-node>:9732 established.

And after the node crashes (the error trace that you can read below), the public node is removed from the trusted peers' list, and I have to trust it again:

<user>@<private-node> ~/tezos/tezos-admin-client connect address <public-node>:9732
Error:
  Node is in private mode.
 <user>@<private-node> ~/tezos/tezos-admin-client trust address <public-node>:9732
Address <public-node>:9732 is now trusted.
<user>@<private-node> ~/tezos/tezos-admin-client connect address <public-node>:9732
Connection to <public-node>:9732 established.```

Error trace

Oct 24 10:32:03 - main: validator-chain_1 : Update current head to BKnC9iXAiYexU7tHuvnkx3E2suLkVXxkoAVncwj7pz8MwDxTkJr (fitness 01::000000000000059e), same branch
Oct 24 10:32:03 - main:                     Pushed: 2019-10-24T10:32:03-00:00, Treated: 2019-10-24T10:32:03-00:00, Completed: 2019-10-24T10:32:03-00:00(2019-10-24T10:32:03-00:00)
Oct 24 10:32:04 - main: validator-block : Block BML74jYWtc8c8FLgxatiBM8Xek6NDzLM1ThNZYDKnbiopjt8616 successfully validated
Oct 24 10:32:04 - main:                   Pushed: 2019-10-24T10:32:03-00:00, Treated: 2019-10-24T10:32:03-00:00, Completed: 2019-10-24T10:32:04-00:00(2019-10-24T10:32:04-00:00)
Oct 24 10:32:04 - node.validator.bootstrap_pipeline: Unexpected error (validator):
Oct 24 10:32:04 - node.validator.bootstrap_pipeline: Error, dumping error stack:
Oct 24 10:32:04 - node.validator.bootstrap_pipeline:   Fetch of operations BLn1Vu7qNEfAz6F53Uu7vz1KbWR537Q9GwYNkGywCB7JaZCytad:3 timed out
Oct 24 10:32:04 - node.validator.bootstrap_pipeline:   Fetch of operations BLn1Vu7qNEfAz6F53Uu7vz1KbWR537Q9GwYNkGywCB7JaZCytad:2 timed out
Oct 24 10:32:04 - node.validator.bootstrap_pipeline:   Fetch of operations BLn1Vu7qNEfAz6F53Uu7vz1KbWR537Q9GwYNkGywCB7JaZCytad:1 timed out
Oct 24 10:32:04 - node.validator.bootstrap_pipeline:   Fetch of operations BLn1Vu7qNEfAz6F53Uu7vz1KbWR537Q9GwYNkGywCB7JaZCytad:0 timed out
Oct 24 10:32:04 - node.validator.bootstrap_pipeline:
Oct 24 10:32:04 - main: prevalidator-NetXdQprcVkpa-PsBabyM1eUXZ_1 : switching to new head BML74jYWtc8c8FLgxatiBM8Xek6NDzLM1ThNZYDKnbiopjt8616
Oct 24 10:32:04 - main:                                             Pushed: 2019-10-24T10:32:04-00:00, Treated: 2019-10-24T10:32:04-00:00, Completed: 2019-10-24T10:32:04-00:00(2019-10-24T10:32:04-00:00)
Oct 24 10:32:04 - main: validator-chain_1 : Update current head to BML74jYWtc8c8FLgxatiBM8Xek6NDzLM1ThNZYDKnbiopjt8616 (fitness 01::000000000000059f), same branch
Oct 24 10:32:04 - main:                     Pushed: 2019-10-24T10:32:04-00:00, Treated: 2019-10-24T10:32:04-00:00, Completed: 2019-10-24T10:32:04-00:00(2019-10-24T10:32:04-00:00)
Oct 24 10:32:04 - main: validator-peer_1 : Worker crashed [validator-peer_1]:
Oct 24 10:32:04 - main:                    Fetch of operations BLn1Vu7qNEfAz6F53Uu7vz1KbWR537Q9GwYNkGywCB7JaZCytad:0 timed out
Oct 24 10:32:04 - main:                    Fetch of operations BLn1Vu7qNEfAz6F53Uu7vz1KbWR537Q9GwYNkGywCB7JaZCytad:1 timed out
Oct 24 10:32:04 - main:                    Fetch of operations BLn1Vu7qNEfAz6F53Uu7vz1KbWR537Q9GwYNkGywCB7JaZCytad:2 timed out
Oct 24 10:32:04 - main:                    Fetch of operations BLn1Vu7qNEfAz6F53Uu7vz1KbWR537Q9GwYNkGywCB7JaZCytad:3 timed out(2019-10-24T10:32:04-00:00)