Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@james64

james64/consul-server-2

Created December 1, 2020 16:08
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save james64/d34241214554a23aebec3cc4838d5552 to your computer and use it in GitHub Desktop.
Save james64/d34241214554a23aebec3cc4838d5552 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
consul-server-2
Node name: 'consul-server-2'
Datacenter: 'dc1' (Segment: '<all>')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
Cluster Addr: 10.233.68.21 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
Version: '1.9.0'
==> Log data will now stream in as it occurs:
2020-12-01T14:43:32.561Z [WARN] agent: bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
Node ID: '48b9b896-6bb2-839f-209f-978dc8c93bfd'
2020-12-01T14:43:32.569Z [WARN] agent.auto_config: bootstrap_expect > 0: expecting 3 servers
2020-12-01T14:43:32.595Z [INFO] agent.server.raft: restored from snapshot: id=220-1458502-1606817805527
2020-12-01T14:43:32.742Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.233.68.21:8300 [Follower]" leader=
2020-12-01T14:43:32.742Z [INFO] agent.server.raft: initial configuration: index=1462060 servers="[{Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.152:8300} {Suffrage:Voter ID:c8bb9a79-dcf1-edd3-66b9-851265043986 Address:10.233.67.53:8300}]"
2020-12-01T14:43:32.746Z [INFO] agent.router: Initializing LAN area manager
2020-12-01T14:43:32.746Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-2 10.233.68.21
2020-12-01T14:43:32.744Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-2.dc1 10.233.68.21
2020-12-01T14:43:32.747Z [INFO] agent.server: Adding LAN server: server="consul-server-2 (Addr: tcp/10.233.68.21:8300) (DC: dc1)"
2020-12-01T14:43:32.747Z [INFO] agent.server: Raft data found, disabling bootstrap mode
2020-12-01T14:43:32.747Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-2.dc1 area=wan
2020-12-01T14:43:32.750Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=tcp
2020-12-01T14:43:32.750Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=udp
2020-12-01T14:43:32.751Z [INFO] agent: Starting server: address=[::]:8500 network=tcp protocol=http
2020-12-01T14:43:32.751Z [INFO] agent: started state syncer
2020-12-01T14:43:32.751Z [INFO] agent: (LAN) joining: lan_addresses=[consul-server-0.consul-server.consul.svc, consul-server-1.consul-server.consul.svc, consul-server-2.consul-server.consul.svc]
2020-12-01T14:43:32.751Z [WARN] agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them.
==> Consul agent running!
2020-12-01T14:43:32.751Z [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
2020-12-01T14:43:32.751Z [INFO] agent: Joining cluster...: cluster=LAN
2020-12-01T14:43:32.757Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker1.k8s 10.233.69.200
2020-12-01T14:43:32.757Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20
2020-12-01T14:43:32.757Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.53
2020-12-01T14:43:32.757Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.152
2020-12-01T14:43:32.757Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.53:8300) (DC: dc1)"
2020-12-01T14:43:32.757Z [INFO] agent.server: Adding LAN server: server="consul-server-1 (Addr: tcp/10.233.69.152:8300) (DC: dc1)"
2020-12-01T14:43:32.756Z [WARN] agent.server.memberlist.lan: memberlist: Refuting a dead message (from: consul-server-2)
2020-12-01T14:43:32.757Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
2020-12-01T14:43:32.758Z [WARN] agent.server.memberlist.wan: memberlist: Refuting a dead message (from: consul-server-2.dc1)
2020-12-01T14:43:32.758Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-0.dc1 10.233.67.53
2020-12-01T14:43:32.758Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-1.dc1 10.233.69.152
2020-12-01T14:43:32.758Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-0.dc1 area=wan
2020-12-01T14:43:32.758Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-1.dc1 area=wan
2020-12-01T14:43:32.779Z [WARN] agent.server.raft: failed to get previous log: previous-index=1462073 last-index=1462060 error="log not found"
2020-12-01T14:43:32.786Z [INFO] agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=2
2020-12-01T14:43:32.786Z [WARN] agent.server.memberlist.lan: memberlist: Failed to resolve consul-server-2.consul-server.consul.svc: lookup consul-server-2.consul-server.consul.svc on 10.233.0.3:53: no such host
2020-12-01T14:43:32.786Z [INFO] agent: (LAN) joined: number_of_nodes=2
2020-12-01T14:43:33.029Z [INFO] agent: Synced node info
2020-12-01T14:43:41.295Z [INFO] agent.server.serf.wan: serf: EventMemberLeave: consul-server-1.dc1 10.233.69.152
2020-12-01T14:43:41.295Z [INFO] agent.server: Handled event for server in area: event=member-leave server=consul-server-1.dc1 area=wan
2020-12-01T14:43:45.609Z [INFO] agent.server.serf.lan: serf: EventMemberLeave: consul-server-1 10.233.69.152
2020-12-01T14:43:45.609Z [INFO] agent.server: Removing LAN server: server="consul-server-1 (Addr: tcp/10.233.69.152:8300) (DC: dc1)"
2020-12-01T14:43:46.725Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader=10.233.69.152:8300
2020-12-01T14:43:46.725Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.233.68.21:8300 [Candidate]" term=228
2020-12-01T14:43:50.442Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.233.68.21:8300 [Follower]" leader=10.233.67.53:8300
2020-12-01T14:43:50.469Z [INFO] agent.server: New leader elected: payload=consul-server-0
2020-12-01T14:44:06.932Z [INFO] agent.server.memberlist.lan: memberlist: Updating address for left or failed node consul-server-1 from 10.233.69.152:8301 to 10.233.69.201:8301
2020-12-01T14:44:06.932Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.201
2020-12-01T14:44:06.933Z [INFO] agent.server: Adding LAN server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
2020-12-01T14:44:07.067Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-1.dc1 10.233.69.201
2020-12-01T14:44:07.067Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-1.dc1 area=wan
2020-12-01T14:44:07.067Z [INFO] agent.server.memberlist.wan: memberlist: Updating address for left or failed node consul-server-1.dc1 from 10.233.69.152:8302 to 10.233.69.201:8302
2020-12-01T14:44:15.067Z [INFO] agent.server.serf.wan: serf: EventMemberLeave: consul-server-0.dc1 10.233.67.53
2020-12-01T14:44:15.067Z [INFO] agent.server: Handled event for server in area: event=member-leave server=consul-server-0.dc1 area=wan
2020-12-01T14:44:18.880Z [INFO] agent.server.serf.lan: serf: EventMemberLeave: consul-server-0 10.233.67.53
2020-12-01T14:44:18.880Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.53:8300) (DC: dc1)"
2020-12-01T14:44:20.755Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader=10.233.67.53:8300
2020-12-01T14:44:20.755Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.233.68.21:8300 [Candidate]" term=229
2020-12-01T14:44:20.762Z [INFO] agent.server.raft: election won: tally=1
2020-12-01T14:44:20.762Z [INFO] agent.server.raft: entering leader state: leader="Node at 10.233.68.21:8300 [Leader]"
2020-12-01T14:44:20.762Z [INFO] agent.server.raft: added peer, starting replication: peer=3048cdda-bdc9-e93e-54eb-809e8c03ae05
2020-12-01T14:44:20.762Z [INFO] agent.server: cluster leadership acquired
2020-12-01T14:44:20.762Z [INFO] agent.server: New leader elected: payload=consul-server-2
2020-12-01T14:44:20.765Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter 3048cdda-bdc9-e93e-54eb-809e8c03ae05 10.233.69.201:8300}"
2020-12-01T14:44:20.770Z [INFO] agent.leader: started routine: routine="federation state anti-entropy"
2020-12-01T14:44:20.770Z [INFO] agent.leader: started routine: routine="federation state pruning"
2020-12-01T14:44:20.770Z [INFO] agent.leader: started routine: routine="intermediate cert renew watch"
2020-12-01T14:44:20.770Z [INFO] agent.leader: started routine: routine="CA root pruning"
2020-12-01T14:44:20.772Z [INFO] agent.server: deregistering member: member=consul-server-0 reason=left
2020-12-01T14:44:20.775Z [INFO] agent.server: member joined, marking health alive: member=consul-server-1
2020-12-01T14:44:20.778Z [INFO] agent.server: member joined, marking health alive: member=consul-server-2
2020-12-01T14:44:21.200Z [INFO] agent.server: federation state anti-entropy synced
2020-12-01T14:44:30.771Z [INFO] agent.server.autopilot: Promoting server: id=3048cdda-bdc9-e93e-54eb-809e8c03ae05 address=10.233.69.201:8300 name=consul-server-1
2020-12-01T14:44:30.771Z [INFO] agent.server.raft: updating configuration: command=AddStaging server-id=3048cdda-bdc9-e93e-54eb-809e8c03ae05 server-addr=10.233.69.201:8300 servers="[{Suffrage:Voter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300} {Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.201:8300}]"
2020-12-01T14:44:32.517Z [INFO] agent.server.memberlist.wan: memberlist: Updating address for left or failed node consul-server-0.dc1 from 10.233.67.53:8302 to 10.233.67.184:8302
2020-12-01T14:44:32.517Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-0.dc1 10.233.67.184
2020-12-01T14:44:32.517Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-0.dc1 area=wan
2020-12-01T14:44:32.518Z [INFO] agent.server.memberlist.lan: memberlist: Updating address for left or failed node consul-server-0 from 10.233.67.53:8301 to 10.233.67.184:8301
2020-12-01T14:44:32.518Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
2020-12-01T14:44:32.518Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T14:44:32.519Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 server-addr=10.233.67.184:8300 servers="[{Suffrage:Voter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300} {Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.201:8300} {Suffrage:Nonvoter ID:c8bb9a79-dcf1-edd3-66b9-851265043986 Address:10.233.67.184:8300}]"
2020-12-01T14:44:32.521Z [INFO] agent.server.raft: added peer, starting replication: peer=c8bb9a79-dcf1-edd3-66b9-851265043986
2020-12-01T14:44:32.525Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}" next=1462091
2020-12-01T14:44:32.525Z [INFO] agent.server: member joined, marking health alive: member=consul-server-0
2020-12-01T14:44:32.550Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}"
2020-12-01T14:44:50.771Z [INFO] agent.server.autopilot: Promoting server: id=c8bb9a79-dcf1-edd3-66b9-851265043986 address=10.233.67.184:8300 name=consul-server-0
2020-12-01T14:44:50.771Z [INFO] agent.server.raft: updating configuration: command=AddStaging server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 server-addr=10.233.67.184:8300 servers="[{Suffrage:Voter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300} {Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.201:8300} {Suffrage:Voter ID:c8bb9a79-dcf1-edd3-66b9-851265043986 Address:10.233.67.184:8300}]"
2020-12-01T15:01:44.746Z [INFO] agent.server.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received
2020-12-01T15:01:45.293Z [WARN] agent.server.raft: failed to contact: server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 time=2.500142563s
2020-12-01T15:01:45.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:01:47.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:01:47.777Z [WARN] agent.server.raft: failed to contact: server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 time=4.984306253s
2020-12-01T15:01:48.010Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
2020-12-01T15:01:48.011Z [INFO] agent.server: member failed, marking health critical: member=worker3.k8s
2020-12-01T15:01:48.747Z [INFO] agent.server.memberlist.lan: memberlist: Marking consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2020-12-01T15:01:48.747Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
2020-12-01T15:01:48.747Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:01:48.747Z [INFO] agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
2020-12-01T15:01:48.747Z [INFO] agent.server: member failed, marking health critical: member=consul-server-0
2020-12-01T15:01:49.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:01:49.974Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986"
2020-12-01T15:01:50.004Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986"
2020-12-01T15:01:50.013Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986"
2020-12-01T15:01:50.293Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:01:50.293Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
2020-12-01T15:01:50.293Z [INFO] agent.server: member joined, marking health alive: member=consul-server-0
2020-12-01T15:01:52.211Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
2020-12-01T15:01:52.211Z [INFO] agent.server: member joined, marking health alive: member=worker3.k8s
2020-12-01T15:02:01.036Z [WARN] agent.server.kvs: Rejecting lock of key due to lock-delay: key=vault/core/lock expire_time="2020-12-01 15:02:03.020129975 +0000 UTC m=+1110.522970294"
2020-12-01T15:02:13.746Z [INFO] agent.server.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received
2020-12-01T15:02:13.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:02:13.782Z [WARN] agent.server.raft: failed to contact: server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 time=2.50016888s
2020-12-01T15:02:15.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:02:16.262Z [WARN] agent.server.raft: failed to contact: server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 time=4.979693266s
2020-12-01T15:02:17.746Z [INFO] agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
2020-12-01T15:02:17.747Z [INFO] agent.server.memberlist.lan: memberlist: Marking consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2020-12-01T15:02:17.747Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
2020-12-01T15:02:17.747Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:17.747Z [INFO] agent.server: member failed, marking health critical: member=consul-server-0
2020-12-01T15:02:17.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:02:17.892Z [INFO] agent.server.memberlist.lan: memberlist: Marking worker3.k8s as failed, suspect timeout reached (2 peer confirmations)
2020-12-01T15:02:17.892Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
2020-12-01T15:02:17.892Z [INFO] agent.server: member failed, marking health critical: member=worker3.k8s
2020-12-01T15:02:18.723Z [WARN] agent.server.raft: failed to contact: server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 time=7.440531602s
2020-12-01T15:02:19.749Z [INFO] agent.server.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received
2020-12-01T15:02:19.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:02:21.341Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}"
2020-12-01T15:02:21.341Z [ERROR] agent.server.raft: failed to pipeline appendEntries: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}" error="write tcp 10.233.68.21:43522->10.233.67.184:8300: use of closed network connection"
2020-12-01T15:02:21.420Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986"
2020-12-01T15:02:21.504Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.233.67.184:8300 error="read tcp 10.233.68.21:43524->10.233.67.184:8300: i/o timeout"
2020-12-01T15:02:21.515Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986"
2020-12-01T15:02:21.792Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:02:21.792Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=c8bb9a79-dcf1-edd3-66b9-851265043986 name=consul-server-0 address=10.233.67.184:8300
2020-12-01T15:02:21.798Z [INFO] agent.server.serf.lan: serf: EventMemberLeave (forced): consul-server-0 10.233.67.184
2020-12-01T15:02:21.799Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:21.799Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 server-addr= servers="[{Suffrage:Voter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300} {Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.201:8300}]"
2020-12-01T15:02:21.810Z [INFO] agent.server.raft: removed peer, stopping replication: peer=c8bb9a79-dcf1-edd3-66b9-851265043986 last-index=1462385
2020-12-01T15:02:21.823Z [INFO] agent.server.autopilot: removed server: id=c8bb9a79-dcf1-edd3-66b9-851265043986
2020-12-01T15:02:21.823Z [INFO] agent.server: deregistering member: member=consul-server-0 reason=left
2020-12-01T15:02:22.412Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986"
2020-12-01T15:02:22.495Z [WARN] agent.server.memberlist.lan: memberlist: Refuting a suspect message (from: consul-server-1)
2020-12-01T15:02:22.611Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
2020-12-01T15:02:22.611Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
2020-12-01T15:02:22.611Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:22.611Z [INFO] agent.server: member joined, marking health alive: member=worker3.k8s
2020-12-01T15:02:22.618Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 server-addr=10.233.67.184:8300 servers="[{Suffrage:Voter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300} {Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.201:8300} {Suffrage:Nonvoter ID:c8bb9a79-dcf1-edd3-66b9-851265043986 Address:10.233.67.184:8300}]"
2020-12-01T15:02:22.620Z [INFO] agent.server.raft: added peer, starting replication: peer=c8bb9a79-dcf1-edd3-66b9-851265043986
2020-12-01T15:02:22.623Z [INFO] agent.server: member joined, marking health alive: member=consul-server-0
2020-12-01T15:02:23.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:02:24.498Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}" next=1462385
2020-12-01T15:02:24.572Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}"
2020-12-01T15:02:26.717Z [WARN] agent.server.kvs: Rejecting lock of key due to lock-delay: key=vault/core/lock expire_time="2020-12-01 15:02:32.9010502 +0000 UTC m=+1140.403890506"
2020-12-01T15:02:28.746Z [INFO] agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
2020-12-01T15:02:29.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:02:31.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:02:32.744Z [INFO] agent.server.memberlist.wan: memberlist: Suspect consul-server-0.dc1 has failed, no acks received
2020-12-01T15:02:32.746Z [INFO] agent.server.memberlist.lan: memberlist: Marking worker3.k8s as failed, suspect timeout reached (2 peer confirmations)
2020-12-01T15:02:32.747Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
2020-12-01T15:02:32.747Z [INFO] agent.server: member failed, marking health critical: member=worker3.k8s
2020-12-01T15:02:32.747Z [INFO] agent.server.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received
2020-12-01T15:02:33.770Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:02:35.895Z [WARN] agent.server.memberlist.lan: memberlist: Refuting a suspect message (from: worker3.k8s)
2020-12-01T15:02:37.280Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
2020-12-01T15:02:37.280Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:37.280Z [INFO] agent.server: member failed, marking health critical: member=consul-server-0
2020-12-01T15:02:37.286Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986"
2020-12-01T15:02:38.001Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986"
2020-12-01T15:02:38.516Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986"
2020-12-01T15:02:39.512Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986"
2020-12-01T15:02:40.324Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986"
2020-12-01T15:02:40.772Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=c8bb9a79-dcf1-edd3-66b9-851265043986 name=consul-server-0 address=10.233.67.184:8300
2020-12-01T15:02:40.772Z [INFO] agent.server.serf.lan: serf: EventMemberLeave (forced): consul-server-0 10.233.67.184
2020-12-01T15:02:40.772Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 server-addr= servers="[{Suffrage:Voter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300} {Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.201:8300}]"
2020-12-01T15:02:40.772Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:40.776Z [INFO] agent.server.raft: removed peer, stopping replication: peer=c8bb9a79-dcf1-edd3-66b9-851265043986 last-index=1462404
2020-12-01T15:02:40.779Z [INFO] agent.server.autopilot: removed server: id=c8bb9a79-dcf1-edd3-66b9-851265043986
2020-12-01T15:02:40.779Z [INFO] agent.server: deregistering member: member=consul-server-0 reason=left
2020-12-01T15:02:40.776Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}"
2020-12-01T15:02:40.897Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: worker2.k8s 10.233.68.20
2020-12-01T15:02:40.897Z [INFO] agent.server: member failed, marking health critical: member=worker2.k8s
2020-12-01T15:02:41.086Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker2.k8s 10.233.68.20
2020-12-01T15:02:41.086Z [INFO] agent.server: member joined, marking health alive: member=worker2.k8s
2020-12-01T15:02:42.494Z [WARN] agent.server.memberlist.wan: memberlist: Refuting a suspect message (from: consul-server-0.dc1)
2020-12-01T15:02:45.832Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.233.67.184:8300 leader=10.233.68.21:8300
2020-12-01T15:02:46.888Z [INFO] agent.server.memberlist.wan: memberlist: Marking consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2020-12-01T15:02:46.888Z [INFO] agent.server.serf.wan: serf: EventMemberFailed: consul-server-0.dc1 10.233.67.184
2020-12-01T15:02:46.888Z [INFO] agent.server: Handled event for server in area: event=member-failed server=consul-server-0.dc1 area=wan
2020-12-01T15:02:50.571Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-0.dc1 10.233.67.184
2020-12-01T15:02:50.571Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-0.dc1 area=wan
2020-12-01T15:02:52.063Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.233.67.184:8300 leader=10.233.68.21:8300
2020-12-01T15:02:52.480Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
2020-12-01T15:02:52.480Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
2020-12-01T15:02:52.480Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:02:52.480Z [INFO] agent.server: member joined, marking health alive: member=worker3.k8s
2020-12-01T15:02:52.489Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 server-addr=10.233.67.184:8300 servers="[{Suffrage:Voter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300} {Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.201:8300} {Suffrage:Nonvoter ID:c8bb9a79-dcf1-edd3-66b9-851265043986 Address:10.233.67.184:8300}]"
2020-12-01T15:02:52.493Z [INFO] agent.server.raft: added peer, starting replication: peer=c8bb9a79-dcf1-edd3-66b9-851265043986
2020-12-01T15:02:52.494Z [ERROR] agent.server: failed to reconcile member: member="{consul-server-0 10.233.67.184 8301 map[acls:0 build:1.9.0:a417fe51 dc:dc1 expect:3 ft_fs:1 ft_si:1 id:c8bb9a79-dcf1-edd3-66b9-851265043986 port:8300 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:8302] alive 1 5 2 2 5 4}" error="leadership lost while committing log"
2020-12-01T15:02:52.494Z [INFO] agent.server: cluster leadership lost
2020-12-01T15:02:52.494Z [ERROR] agent.server.raft: peer has newer term, stopping replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}"
2020-12-01T15:02:52.494Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.233.68.21:8300 [Follower]" leader=
2020-12-01T15:02:52.494Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter 3048cdda-bdc9-e93e-54eb-809e8c03ae05 10.233.69.201:8300}"
2020-12-01T15:02:52.494Z [ERROR] agent.server.autopilot: failed to add raft non-voting peer: id=c8bb9a79-dcf1-edd3-66b9-851265043986 address=10.233.67.184:8300 error="leadership lost while committing log"
2020-12-01T15:02:54.224Z [WARN] agent.server.coordinate: Batch update failed: error="node is not the leader"
2020-12-01T15:02:58.700Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.233.68.21:8300 [Candidate]" term=230
2020-12-01T15:02:58.700Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader=
2020-12-01T15:02:58.726Z [INFO] agent.server.raft: entering leader state: leader="Node at 10.233.68.21:8300 [Leader]"
2020-12-01T15:02:58.726Z [INFO] agent.server.raft: added peer, starting replication: peer=3048cdda-bdc9-e93e-54eb-809e8c03ae05
2020-12-01T15:02:58.726Z [INFO] agent.server.raft: added peer, starting replication: peer=c8bb9a79-dcf1-edd3-66b9-851265043986
2020-12-01T15:02:58.726Z [INFO] agent.server.raft: election won: tally=2
2020-12-01T15:02:58.727Z [INFO] agent.server: New leader elected: payload=consul-server-2
2020-12-01T15:02:58.726Z [INFO] agent.server: cluster leadership acquired
2020-12-01T15:02:58.727Z [INFO] agent.server.raft: pipelining replication: peer="{Voter 3048cdda-bdc9-e93e-54eb-809e8c03ae05 10.233.69.201:8300}"
2020-12-01T15:02:58.728Z [ERROR] agent.server.raft: peer has newer term, stopping replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}"
2020-12-01T15:02:58.733Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.233.68.21:8300 [Follower]" leader=
2020-12-01T15:02:58.733Z [ERROR] agent.server: failed to wait for barrier: error="leadership lost while committing log"
2020-12-01T15:02:58.733Z [INFO] agent.server: cluster leadership lost
2020-12-01T15:02:58.733Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Voter 3048cdda-bdc9-e93e-54eb-809e8c03ae05 10.233.69.201:8300}"
2020-12-01T15:02:59.224Z [WARN] agent.server.coordinate: Batch update failed: error="node is not the leader"
2020-12-01T15:03:01.424Z [WARN] agent.server.raft: rejecting vote request since our last term is greater: candidate=10.233.67.184:8300 last-term=230 last-candidate-term=229
2020-12-01T15:03:04.502Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.233.68.21:8300 [Candidate]" term=233
2020-12-01T15:03:04.502Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader=
2020-12-01T15:03:04.563Z [INFO] agent.server.raft: election won: tally=2
2020-12-01T15:03:04.563Z [INFO] agent.server.raft: entering leader state: leader="Node at 10.233.68.21:8300 [Leader]"
2020-12-01T15:03:04.563Z [INFO] agent.server.raft: added peer, starting replication: peer=3048cdda-bdc9-e93e-54eb-809e8c03ae05
2020-12-01T15:03:04.563Z [INFO] agent.server.raft: added peer, starting replication: peer=c8bb9a79-dcf1-edd3-66b9-851265043986
2020-12-01T15:03:04.563Z [INFO] agent.server: cluster leadership acquired
2020-12-01T15:03:04.564Z [INFO] agent.server: New leader elected: payload=consul-server-2
2020-12-01T15:03:04.564Z [INFO] agent.server.raft: pipelining replication: peer="{Voter 3048cdda-bdc9-e93e-54eb-809e8c03ae05 10.233.69.201:8300}"
2020-12-01T15:03:04.569Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}" next=1462405
2020-12-01T15:03:04.576Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}"
2020-12-01T15:03:04.585Z [INFO] agent.leader: started routine: routine="federation state anti-entropy"
2020-12-01T15:03:04.585Z [INFO] agent.leader: started routine: routine="intermediate cert renew watch"
2020-12-01T15:03:04.585Z [INFO] agent.leader: started routine: routine="CA root pruning"
2020-12-01T15:03:04.585Z [INFO] agent.leader: started routine: routine="federation state pruning"
2020-12-01T15:03:04.585Z [INFO] agent.server: member joined, marking health alive: member=consul-server-0
2020-12-01T15:03:14.586Z [INFO] agent.server.autopilot: Promoting server: id=c8bb9a79-dcf1-edd3-66b9-851265043986 address=10.233.67.184:8300 name=consul-server-0
2020-12-01T15:03:14.586Z [INFO] agent.server.raft: updating configuration: command=AddStaging server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 server-addr=10.233.67.184:8300 servers="[{Suffrage:Voter ID:48b9b896-6bb2-839f-209f-978dc8c93bfd Address:10.233.68.21:8300} {Suffrage:Voter ID:3048cdda-bdc9-e93e-54eb-809e8c03ae05 Address:10.233.69.201:8300} {Suffrage:Voter ID:c8bb9a79-dcf1-edd3-66b9-851265043986 Address:10.233.67.184:8300}]"
2020-12-01T15:04:50.554Z [ERROR] agent.server: error performing anti-entropy sync of federation state: error="context canceled"
2020-12-01T15:08:03.134Z [WARN] agent.server.raft: failed to contact: server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 time=2.500124934s
2020-12-01T15:08:03.585Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:08:03.746Z [INFO] agent.server.memberlist.lan: memberlist: Suspect consul-server-0 has failed, no acks received
2020-12-01T15:08:05.585Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:08:05.596Z [WARN] agent.server.raft: failed to contact: server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 time=4.962230093s
2020-12-01T15:08:07.292Z [INFO] agent.server.memberlist.lan: memberlist: Marking worker3.k8s as failed, suspect timeout reached (2 peer confirmations)
2020-12-01T15:08:07.292Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: worker3.k8s 10.233.67.183
2020-12-01T15:08:07.292Z [INFO] agent.server: member failed, marking health critical: member=worker3.k8s
2020-12-01T15:08:07.585Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:08:07.746Z [INFO] agent.server.memberlist.lan: memberlist: Suspect worker3.k8s has failed, no acks received
2020-12-01T15:08:07.747Z [INFO] agent.server.memberlist.lan: memberlist: Marking consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2020-12-01T15:08:07.747Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: consul-server-0 10.233.67.184
2020-12-01T15:08:07.747Z [INFO] agent.server: Removing LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:08:07.748Z [INFO] agent.server: member failed, marking health critical: member=consul-server-0
2020-12-01T15:08:08.087Z [WARN] agent.server.raft: failed to contact: server-id=c8bb9a79-dcf1-edd3-66b9-851265043986 time=7.452862656s
2020-12-01T15:08:09.585Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:08:10.670Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}"
2020-12-01T15:08:10.711Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=c8bb9a79-dcf1-edd3-66b9-851265043986 fallback=10.233.67.184:8300 error="Could not find address for server id c8bb9a79-dcf1-edd3-66b9-851265043986"
2020-12-01T15:08:10.734Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.233.67.184
2020-12-01T15:08:10.734Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: worker3.k8s 10.233.67.183
2020-12-01T15:08:10.734Z [INFO] agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.233.67.184:8300) (DC: dc1)"
2020-12-01T15:08:10.735Z [INFO] agent.server: member joined, marking health alive: member=consul-server-0
2020-12-01T15:08:10.741Z [INFO] agent.server: member joined, marking health alive: member=worker3.k8s
2020-12-01T15:08:10.808Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: consul-server-1 10.233.69.201
2020-12-01T15:08:10.808Z [INFO] agent.server: member failed, marking health critical: member=consul-server-1
2020-12-01T15:08:10.808Z [INFO] agent.server: Removing LAN server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
2020-12-01T15:08:10.816Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=3048cdda-bdc9-e93e-54eb-809e8c03ae05 fallback=10.233.69.201:8300 error="Could not find address for server id 3048cdda-bdc9-e93e-54eb-809e8c03ae05"
2020-12-01T15:08:10.892Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-1 10.233.69.201
2020-12-01T15:08:10.892Z [INFO] agent.server: Adding LAN server: server="consul-server-1 (Addr: tcp/10.233.69.201:8300) (DC: dc1)"
2020-12-01T15:08:10.892Z [INFO] agent.server: member joined, marking health alive: member=consul-server-1
2020-12-01T15:08:11.585Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:08:11.608Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.233.67.184:8300 error="read tcp 10.233.68.21:37132->10.233.67.184:8300: i/o timeout"
2020-12-01T15:08:13.460Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter c8bb9a79-dcf1-edd3-66b9-851265043986 10.233.67.184:8300}"
2020-12-01T15:08:13.585Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:08:15.183Z [ERROR] agent: yamux: keepalive failed: i/o deadline reached
2020-12-01T15:08:15.183Z [WARN] agent: error getting server health from server: server=consul-server-0 error="rpc error making call: EOF"
2020-12-01T15:08:15.183Z [WARN] agent: error getting server health from server: server=consul-server-0 error="rpc error making call: EOF"
2020-12-01T15:08:15.183Z [WARN] agent: error getting server health from server: server=consul-server-0 error="rpc error making call: EOF"
2020-12-01T15:08:15.183Z [WARN] agent: error getting server health from server: server=consul-server-0 error="rpc error making call: EOF"
2020-12-01T15:08:15.183Z [WARN] agent: error getting server health from server: server=consul-server-0 error="rpc error making call: EOF"
2020-12-01T15:08:15.183Z [WARN] agent: error getting server health from server: server=consul-server-0 error="rpc error making call: EOF"
2020-12-01T15:08:15.183Z [WARN] agent: error getting server health from server: server=consul-server-0 error="rpc error making call: EOF"
2020-12-01T15:08:15.586Z [WARN] agent: error getting server health from server: server=consul-server-0 error="context deadline exceeded"
2020-12-01T15:08:17.494Z [WARN] agent.server.memberlist.wan: memberlist: Refuting a suspect message (from: consul-server-0.dc1)
2020-12-01T15:08:20.728Z [WARN] agent.server.kvs: Rejecting lock of key due to lock-delay: key=vault/core/lock expire_time="2020-12-01 15:08:22.301795542 +0000 UTC m=+1489.804635841"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment