consul-server-0

==> Starting Consul agent...
           Node ID: 'c8bb9a79-dcf1-edd3-66b9-851265043986'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
           Version: '1.9.0'
         Node name: 'consul-server-0'
==> Log data will now stream in as it occurs:
       Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
      Cluster Addr: 10.233.67.184 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
    2020-12-01T14:44:32.328Z [WARN]  agent: bootstrap_expect > 0: expecting 3 servers
    2020-12-01T14:44:32.340Z [WARN]  agent.auto_config: bootstrap_expect > 0: expecting 3 servers
    2020-12-01T14:44:32.368Z [INFO]  agent.server.raft: restored from snapshot: id=220-1458518-1606817868101
    2020-12-01T14:44:32.492Z [INFO]  agent.server.raft: entering follower state: follower="Node at 10.233.67.184:8300 [Follower]" leader=
    2020-12-01T14:44:32.491Z [INFO]  agent.server.raft: initial configuration: index=1462090 servers="[{Suffrage:Voter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300} {Suffrage:Nonvoter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.201:8300}]"
    2020-12-01T14:44:32.493Z [INFO]  agent.server.serf.wan: serf: EventMemberJoin: consul-server-0.dc1 10.233.67.184
    2020-12-01T14:44:32.495Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
    2020-12-01T14:44:32.495Z [INFO]  agent.router: Initializing LAN area manager
    2020-12-01T14:44:32.496Z [INFO]  agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
    2020-12-01T14:44:32.496Z [INFO]  agent.server: Handled event for server in area: event=member-join server=consul-server-0.dc1 area=wan
    2020-12-01T14:44:32.496Z [INFO]  agent.server: Raft data found, disabling bootstrap mode
    2020-12-01T14:44:32.500Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=udp
    2020-12-01T14:44:32.500Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=tcp
    2020-12-01T14:44:32.501Z [INFO]  agent: started state syncer
    2020-12-01T14:44:32.501Z [INFO]  agent: Joining cluster...: cluster=LAN
    2020-12-01T14:44:32.501Z [INFO]  agent: Starting server: address=[::]:8500 network=tcp protocol=http
    2020-12-01T14:44:32.501Z [WARN]  agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them.
==> Consul agent running!
    2020-12-01T14:44:32.501Z [INFO]  agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
    2020-12-01T14:44:32.501Z [INFO]  agent: (LAN) joining: lan_addresses=[consul-server-0.consul-server.consul.svc, consul-server-1.consul-server.consul.svc, consul-server-2.consul-server.consul.svc]
    2020-12-01T14:44:32.511Z [WARN]  agent.server.memberlist.lan: memberlist: Failed to resolve consul-server-0.consul-server.consul.svc: lookup consul-server-0.consul-server.consul.svc on 10.233.0.3:53: no such host
    2020-12-01T14:44:32.514Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20
    2020-12-01T14:44:32.515Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
    2020-12-01T14:44:32.515Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.201
    2020-12-01T14:44:32.515Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: consul-server-2 10.233.68.21
    2020-12-01T14:44:32.515Z [INFO]  agent.server: Adding LAN server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
    2020-12-01T14:44:32.515Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: worker1.k8s 10.233.69.200
    2020-12-01T14:44:32.515Z [INFO]  agent.server: Adding LAN server: server="consul-server-2 (Addr: tcp/10.233.68.21:8300) (DC: dc1)"
    2020-12-01T14:44:32.514Z [WARN]  agent.server.memberlist.lan: memberlist: Refuting a dead message (from: consul-server-0)
    2020-12-01T14:44:32.516Z [INFO]  agent.server.serf.wan: serf: EventMemberJoin: consul-server-2.dc1 10.233.68.21
    2020-12-01T14:44:32.516Z [INFO]  agent.server.serf.wan: serf: EventMemberJoin: consul-server-1.dc1 10.233.69.201
    2020-12-01T14:44:32.516Z [WARN]  agent.server.memberlist.wan: memberlist: Refuting a dead message (from: consul-server-0.dc1)
    2020-12-01T14:44:32.517Z [INFO]  agent.server: Handled event for server in area: event=member-join server=consul-server-2.dc1 area=wan
    2020-12-01T14:44:32.517Z [INFO]  agent.server: Handled event for server in area: event=member-join server=consul-server-1.dc1 area=wan
    2020-12-01T14:44:32.519Z [INFO]  agent: (LAN) joined: number_of_nodes=2
    2020-12-01T14:44:32.519Z [INFO]  agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=2
    2020-12-01T14:44:32.525Z [WARN]  agent.server.raft: failed to get previous log: previous-index=1462103 last-index=1462090 error="log not found"
    2020-12-01T14:44:32.611Z [INFO]  agent: Synced node info
    2020-12-01T14:44:32.623Z [INFO]  agent: Synced service: service=sandbox-8082
    2020-12-01T14:44:34.922Z [INFO]  agent: Synced check: check=service:sandbox-8082
    2020-12-01T15:01:44.496Z [INFO]  agent.server.memberlist.lan: memberlist: Suspect worker1.k8s has failed, no acks received
    2020-12-01T15:01:45.080Z [WARN]  agent.server.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-2)
    2020-12-01T15:01:48.011Z [INFO]  agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
    2020-12-01T15:01:49.496Z [INFO]  agent.server.memberlist.lan: memberlist: Suspect worker2.k8s has failed, no acks received
    2020-12-01T15:01:52.280Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
    2020-12-01T15:02:12.496Z [INFO]  agent.server.memberlist.lan: memberlist: Suspect consul-server-2 has failed, no acks received
    2020-12-01T15:02:13.881Z [WARN]  agent.server.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-2)
    2020-12-01T15:02:16.496Z [INFO]  agent.server.memberlist.lan: memberlist: Suspect worker2.k8s has failed, no acks received
    2020-12-01T15:02:17.245Z [WARN]  agent.server.memberlist.wan: memberlist: Refuting a suspect message (from: consul-server-1.dc1)
    2020-12-01T15:02:17.493Z [INFO]  agent.server.memberlist.wan: memberlist: Suspect consul-server-2.dc1 has failed, no acks received
    2020-12-01T15:02:21.970Z [WARN]  agent: Check is now critical: check=service:sandbox-8082
    2020-12-01T15:02:21.984Z [INFO]  agent: Synced check: check=service:sandbox-8082
    2020-12-01T15:02:22.496Z [WARN]  agent.server.memberlist.lan: memberlist: Was able to connect to consul-server-1 but other probes failed, network may be misconfigured
    2020-12-01T15:02:24.445Z [ERROR] agent.server.raft: failed to flush response: error="write tcp 10.233.67.184:8300->10.233.68.21:43522: write: connection reset by peer"
    2020-12-01T15:02:24.498Z [WARN]  agent.server.raft: failed to get previous log: previous-index=1462389 last-index=1462384 error="log not found"
    2020-12-01T15:02:28.495Z [INFO]  agent.server.memberlist.lan: memberlist: Suspect worker2.k8s has failed, no acks received
    2020-12-01T15:02:32.494Z [WARN]  agent.server.memberlist.wan: memberlist: Was able to connect to consul-server-1.dc1 but other probes failed, network may be misconfigured
    2020-12-01T15:02:32.496Z [INFO]  agent.server.memberlist.lan: memberlist: Suspect worker2.k8s has failed, no acks received
    2020-12-01T15:02:32.811Z [WARN]  agent.server.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-2)
    2020-12-01T15:02:37.009Z [INFO]  agent: Synced check: check=service:sandbox-8082
    2020-12-01T15:02:39.496Z [WARN]  agent.server.memberlist.lan: memberlist: Was able to connect to consul-server-1 but other probes failed, network may be misconfigured
    2020-12-01T15:02:40.809Z [INFO]  agent.server.memberlist.lan: memberlist: Marking worker2.k8s as failed, suspect timeout reached (1 peer confirmations)
    2020-12-01T15:02:40.809Z [INFO]  agent.server.serf.lan: serf: EventMemberFailed: worker2.k8s 10.233.68.20
    2020-12-01T15:02:41.081Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20
    2020-12-01T15:02:42.494Z [WARN]  agent.server.memberlist.wan: memberlist: Was able to connect to consul-server-1.dc1 but other probes failed, network may be misconfigured
    2020-12-01T15:02:45.819Z [WARN]  agent.server.raft: heartbeat timeout reached, starting election: last-leader=10.233.68.21:8300
    2020-12-01T15:02:45.819Z [INFO]  agent.server.raft: entering candidate state: node="Node at 10.233.67.184:8300 [Candidate]" term=230
    2020-12-01T15:02:52.057Z [WARN]  agent.server.raft: Election timeout reached, restarting election
    2020-12-01T15:02:52.057Z [INFO]  agent.server.raft: entering candidate state: node="Node at 10.233.67.184:8300 [Candidate]" term=231
    2020-12-01T15:02:58.748Z [INFO]  agent.server: New leader elected: payload=consul-server-2
    2020-12-01T15:03:01.390Z [WARN]  agent.server.raft: Election timeout reached, restarting election
    2020-12-01T15:03:01.390Z [INFO]  agent.server.raft: entering candidate state: node="Node at 10.233.67.184:8300 [Candidate]" term=232
    2020-12-01T15:03:04.569Z [WARN]  agent.server.raft: failed to get previous log: previous-index=1462416 last-index=1462404 error="log not found"
    2020-12-01T15:03:04.569Z [INFO]  agent.server.raft: entering follower state: follower="Node at 10.233.67.184:8300 [Follower]" leader=10.233.68.21:8300
    2020-12-01T15:03:04.813Z [INFO]  agent.server: New leader elected: payload=consul-server-2
    2020-12-01T15:03:05.191Z [INFO]  agent: Synced node info
    2020-12-01T15:03:05.202Z [INFO]  agent: Synced service: service=sandbox-8082
    2020-12-01T15:08:02.496Z [INFO]  agent.server.memberlist.lan: memberlist: Suspect consul-server-1 has failed, no acks received
    2020-12-01T15:08:03.948Z [WARN]  agent.server.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-2)
    2020-12-01T15:08:06.496Z [INFO]  agent.server.memberlist.lan: memberlist: Suspect worker2.k8s has failed, no acks received
    2020-12-01T15:08:07.411Z [INFO]  agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
    2020-12-01T15:08:07.493Z [INFO]  agent.server.memberlist.wan: memberlist: Suspect consul-server-2.dc1 has failed, no acks received
    2020-12-01T15:08:10.811Z [INFO]  agent.server.serf.lan: serf: EventMemberFailed: consul-server-1 10.233.69.201
    2020-12-01T15:08:10.811Z [INFO]  agent.server: Removing LAN server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
    2020-12-01T15:08:10.881Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
    2020-12-01T15:08:11.010Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.201
    2020-12-01T15:08:11.010Z [INFO]  agent.server: Adding LAN server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
    2020-12-01T15:08:13.503Z [WARN]  agent.server.memberlist.lan: memberlist: Was able to connect to consul-server-2 but other probes failed, network may be misconfigured
    2020-12-01T15:08:14.040Z [ERROR] agent.server.raft: failed to decode incoming command: error="read tcp 10.233.67.184:8300->10.233.68.21:37032: read: connection reset by peer"
    2020-12-01T15:08:15.184Z [ERROR] agent.server.rpc: multiplex conn accept failed: conn=from=10.233.68.21:43528 error="keepalive timeout"
    2020-12-01T15:08:15.184Z [ERROR] agent.server: yamux: keepalive failed: i/o deadline reached
    2020-12-01T15:08:17.493Z [WARN]  agent.server.memberlist.wan: memberlist: Was able to connect to consul-server-1.dc1 but other probes failed, network may be misconfigured