Skip to content

Instantly share code, notes, and snippets.

@dictvm

dictvm/consul 0.9.2 error with bootstrap-expect

Created August 11, 2017 08:12
Show Gist options
  • Save dictvm/86154cd5d383a35d125c9542df40108f to your computer and use it in GitHub Desktop.
Save dictvm/86154cd5d383a35d125c9542df40108f to your computer and use it in GitHub Desktop.
Download ZIP
Raw
consul 0.9.2 error with bootstrap-expect
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: ==> WARNING: Expect Mode enabled, expecting 5 servers
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: ==> Starting Consul agent...
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: ==> Consul agent running!
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: Version: 'v0.9.2'
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: Node ID: '94adc968-0390-8776-6dc2-00771c4cfff5'
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: Node name: 'ip-172-21-70-171'
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: Datacenter: 'eu-west-1'
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: Server: true (bootstrap: false)
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 8600)
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: Cluster Addr: 172.21.70.171 (LAN: 8301, WAN: 8302)
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: Gossip encrypt: true, RPC-TLS: false, TLS-Incoming: false
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: ==> Log data will now stream in as it occurs:
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:20 [DEBUG] Using random ID "94adc968-0390-8776-6dc2-00771c4cfff5" as node ID
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:20 [INFO] raft: Initial configuration (index=0): []
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:20 [INFO] raft: Node at 172.21.70.171:8300 [Follower] entering Follower state (Leader: "")
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:20 [INFO] serf: EventMemberJoin: ip-172-21-70-171.eu-west-1 172.21.70.171
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:20 [INFO] serf: EventMemberJoin: ip-172-21-70-171 172.21.70.171
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:20 [INFO] consul: Adding LAN server ip-172-21-70-171 (Addr: tcp/172.21.70.171:8300) (DC: eu-west-1)
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:20 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:20 [INFO] consul: Handled member-join event for server "ip-172-21-70-171.eu-west-1" in area "wan"
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:20 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:20 [INFO] agent: Started HTTP server on 127.0.0.1:8500
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:20 [INFO] agent: Retry join is supported for: aws azure gce softlayer
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:20 [INFO] agent: Joining cluster...
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:20 [INFO] discover-aws: Looking up region
Aug 11 08:06:20 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:20 [INFO] discover-aws: Region is eu-west-1
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] discover-aws: Filter instances by App=consul
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] discover-aws: Found 3137322e32312e37312e3934
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] discover-aws: Found 3137322e32312e37302e313731
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] discover-aws: Found 3137322e32312e37312e323534
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] discover-aws: Found 3137322e32312e37322e313036
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] discover-aws: Found 3137322e32312e37322e313430
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] agent: Discovered servers: 172.21.71.94 172.21.70.171 172.21.71.254 172.21.72.106 172.21.72.140
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] agent: (LAN) joining: [172.21.71.94 172.21.70.171 172.21.71.254 172.21.72.106 172.21.72.140]
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.71.94:8301
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] serf: EventMemberJoin: ip-172-21-72-106 172.21.72.106
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] serf: EventMemberJoin: ip-172-21-72-140 172.21.72.140
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] serf: EventMemberJoin: ip-172-21-71-94 172.21.71.94
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] serf: EventMemberJoin: ip-172-21-71-254 172.21.71.254
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] memberlist: Stream connection from=172.21.70.171:37320
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] memberlist: Stream connection from=172.21.71.94:52222
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] serf: EventMemberJoin: ip-172-21-71-94.eu-west-1 172.21.71.94
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] serf: EventMemberJoin: ip-172-21-72-140.eu-west-1 172.21.72.140
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] serf: EventMemberJoin: ip-172-21-72-106.eu-west-1 172.21.72.106
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] serf: EventMemberJoin: ip-172-21-71-254.eu-west-1 172.21.71.254
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] consul: Handled member-join event for server "ip-172-21-71-94.eu-west-1" in area "wan"
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] consul: Handled member-join event for server "ip-172-21-72-140.eu-west-1" in area "wan"
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] consul: Handled member-join event for server "ip-172-21-72-106.eu-west-1" in area "wan"
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] consul: Handled member-join event for server "ip-172-21-71-254.eu-west-1" in area "wan"
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] consul: Adding LAN server ip-172-21-72-106 (Addr: tcp/172.21.72.106:8300) (DC: eu-west-1)
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.70.171:8301
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.71.254:8301
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] consul: Existing Raft peers reported by ip-172-21-72-106, disabling bootstrap mode
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] consul: Adding LAN server ip-172-21-72-140 (Addr: tcp/172.21.72.140:8300) (DC: eu-west-1)
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] consul: Adding LAN server ip-172-21-71-94 (Addr: tcp/172.21.71.94:8300) (DC: eu-west-1)
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] consul: Adding LAN server ip-172-21-71-254 (Addr: tcp/172.21.71.254:8300) (DC: eu-west-1)
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.72.106:8301
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] memberlist: Stream connection from=172.21.71.254:37948
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.72.140:8301
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] memberlist: Stream connection from=172.21.72.106:33716
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] agent: (LAN) joined: 5 Err: <nil>
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] agent: systemd notify failed: No socket
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] agent: Join completed. Synced with 5 initial agents
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] memberlist: Stream connection from=172.21.72.140:36288
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] raft-net: 172.21.70.171:8300 accepted connection from: 172.21.72.106:38314
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [WARN] raft: Failed to get previous log: 99 log not found (last: 0)
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] serf: messageJoinType: ip-172-21-72-106.eu-west-1
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [INFO] agent: Synced node info
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] serf: messageJoinType: ip-172-21-70-171
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] serf: messageJoinType: ip-172-21-70-171
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] serf: messageJoinType: ip-172-21-70-171
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] serf: messageJoinType: ip-172-21-70-171
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] serf: messageJoinType: ip-172-21-70-171
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] serf: messageJoinType: ip-172-21-72-106.eu-west-1
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] serf: messageJoinType: ip-172-21-72-140.eu-west-1
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] serf: messageJoinType: ip-172-21-72-106.eu-west-1
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] serf: messageJoinType: ip-172-21-71-94.eu-west-1
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] memberlist: Stream connection from=172.21.72.140:50372
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] serf: messageJoinType: ip-172-21-71-254.eu-west-1
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] serf: messageJoinType: ip-172-21-71-94.eu-west-1
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] serf: messageJoinType: ip-172-21-72-140.eu-west-1
Aug 11 08:06:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:21 [DEBUG] raft-net: 172.21.70.171:8300 accepted connection from: 172.21.72.106:45303
Aug 11 08:06:22 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:22 [DEBUG] serf: messageJoinType: ip-172-21-71-254.eu-west-1
Aug 11 08:06:22 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:22 [DEBUG] serf: messageJoinType: ip-172-21-71-94.eu-west-1
Aug 11 08:06:22 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:22 [DEBUG] serf: messageJoinType: ip-172-21-71-254.eu-west-1
Aug 11 08:06:22 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:22 [DEBUG] serf: messageJoinType: ip-172-21-72-140.eu-west-1
Aug 11 08:06:48 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:48 [DEBUG] memberlist: Stream connection from=172.21.72.106:58058
Aug 11 08:06:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:56 [DEBUG] http: Request GET /v1/status/peers (56.234µs) from=127.0.0.1:33480
Aug 11 08:06:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:56 [DEBUG] http: Request GET /v1/status/leader (18.21µs) from=127.0.0.1:33480
Aug 11 08:06:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:56 [DEBUG] http: Request GET /v1/catalog/nodes (1.235915ms) from=127.0.0.1:33480
Aug 11 08:06:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:56 [DEBUG] http: Request GET /v1/catalog/services (1.959258ms) from=127.0.0.1:33480
Aug 11 08:06:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:56 [DEBUG] http: Request GET /v1/health/service/consul (1.281145ms) from=127.0.0.1:33480
Aug 11 08:06:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:06:56 [DEBUG] http: Request GET /v1/health/state/any (884.05µs) from=127.0.0.1:33480
Aug 11 08:07:04 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:04 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.71.254:8301
Aug 11 08:07:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:11 [DEBUG] http: Request GET /v1/status/peers (42.959µs) from=127.0.0.1:33480
Aug 11 08:07:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:11 [DEBUG] http: Request GET /v1/status/leader (12.069µs) from=127.0.0.1:33480
Aug 11 08:07:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:11 [DEBUG] http: Request GET /v1/catalog/nodes (1.158659ms) from=127.0.0.1:33480
Aug 11 08:07:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:11 [DEBUG] http: Request GET /v1/catalog/services (899.95µs) from=127.0.0.1:33480
Aug 11 08:07:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:11 [DEBUG] http: Request GET /v1/health/service/consul (1.121331ms) from=127.0.0.1:33480
Aug 11 08:07:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:11 [DEBUG] http: Request GET /v1/health/state/any (1.813354ms) from=127.0.0.1:33480
Aug 11 08:07:12 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:12 [DEBUG] memberlist: Stream connection from=172.21.71.254:37956
Aug 11 08:07:18 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:18 [DEBUG] memberlist: Stream connection from=172.21.72.106:58064
Aug 11 08:07:23 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:23 [WARN] consul: error getting server health from "ip-172-21-71-254": rpc error: stream closed
Aug 11 08:07:24 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:24 [WARN] consul: error getting server health from "ip-172-21-71-254": context deadline exceeded
Aug 11 08:07:24 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:24 [WARN] consul: error getting server health from "ip-172-21-72-106": context deadline exceeded
Aug 11 08:07:24 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:24 [WARN] consul: error getting server health from "ip-172-21-71-94": context deadline exceeded
Aug 11 08:07:25 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:25 [DEBUG] memberlist: Failed ping: ip-172-21-71-254 (timeout reached)
Aug 11 08:07:26 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:26 [DEBUG] http: Request GET /v1/status/peers (64.062µs) from=127.0.0.1:33480
Aug 11 08:07:26 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:26 [DEBUG] http: Request GET /v1/status/leader (13.164µs) from=127.0.0.1:33480
Aug 11 08:07:26 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:26 [DEBUG] http: Request GET /v1/catalog/nodes (1.102305ms) from=127.0.0.1:33480
Aug 11 08:07:26 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:26 [DEBUG] http: Request GET /v1/catalog/services (937.613µs) from=127.0.0.1:33480
Aug 11 08:07:26 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:26 [DEBUG] http: Request GET /v1/health/service/consul (1.754852ms) from=127.0.0.1:33480
Aug 11 08:07:26 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:26 [DEBUG] http: Request GET /v1/health/state/any (1.048728ms) from=127.0.0.1:33480
Aug 11 08:07:26 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:26 [WARN] consul: error getting server health from "ip-172-21-71-254": context deadline exceeded
Aug 11 08:07:26 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:26 [WARN] consul: error getting server health from "ip-172-21-72-106": context deadline exceeded
Aug 11 08:07:26 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:26 [WARN] consul: error getting server health from "ip-172-21-71-94": context deadline exceeded
Aug 11 08:07:27 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:27 [WARN] consul: error getting server health from "ip-172-21-71-254": last request still outstanding
Aug 11 08:07:29 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:29 [WARN] consul: error getting server health from "ip-172-21-71-254": last request still outstanding
Aug 11 08:07:31 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:31 [WARN] consul: error getting server health from "ip-172-21-71-254": last request still outstanding
Aug 11 08:07:33 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:33 [WARN] consul: error getting server health from "ip-172-21-71-254": last request still outstanding
Aug 11 08:07:34 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:34 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.71.94:8301
Aug 11 08:07:34 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:34 [INFO] memberlist: Marking ip-172-21-71-254 as failed, suspect timeout reached (2 peer confirmations)
Aug 11 08:07:34 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:34 [INFO] serf: EventMemberFailed: ip-172-21-71-254 172.21.71.254
Aug 11 08:07:34 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:34 [INFO] consul: Removing LAN server ip-172-21-71-254 (Addr: tcp/172.21.71.254:8300) (DC: eu-west-1)
Aug 11 08:07:35 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:35 [INFO] memberlist: Suspect ip-172-21-71-254 has failed, no acks received
Aug 11 08:07:35 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:35 [WARN] consul: error getting server health from "ip-172-21-71-254": last request still outstanding
Aug 11 08:07:35 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:35 [WARN] consul: error getting server health from "ip-172-21-71-254": rpc error: failed to get conn: dial tcp 172.21.70.171:0->172.21.71.254:8300: i/o timeout
Aug 11 08:07:38 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:38 [WARN] consul: error getting server health from "ip-172-21-71-254": context deadline exceeded
Aug 11 08:07:38 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:38 [WARN] consul: error getting server health from "ip-172-21-71-94": context deadline exceeded
Aug 11 08:07:38 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:38 [WARN] consul: error getting server health from "ip-172-21-70-171": context deadline exceeded
Aug 11 08:07:39 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:39 [DEBUG] memberlist: Failed ping: ip-172-21-71-254.eu-west-1 (timeout reached)
Aug 11 08:07:39 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:39 [WARN] consul: error getting server health from "ip-172-21-71-254": last request still outstanding
Aug 11 08:07:41 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:41 [DEBUG] http: Request GET /v1/status/peers (37.498µs) from=127.0.0.1:33480
Aug 11 08:07:41 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:41 [DEBUG] http: Request GET /v1/status/leader (11.385µs) from=127.0.0.1:33480
Aug 11 08:07:41 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:41 [DEBUG] http: Request GET /v1/catalog/nodes (1.049872ms) from=127.0.0.1:33480
Aug 11 08:07:41 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:41 [DEBUG] http: Request GET /v1/catalog/services (884.452µs) from=127.0.0.1:33480
Aug 11 08:07:41 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:41 [DEBUG] http: Request GET /v1/health/service/consul (1.12487ms) from=127.0.0.1:33480
Aug 11 08:07:41 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:41 [DEBUG] http: Request GET /v1/health/state/any (974.868µs) from=127.0.0.1:33480
Aug 11 08:07:41 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:41 [WARN] consul: error getting server health from "ip-172-21-71-254": last request still outstanding
Aug 11 08:07:42 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:42 [DEBUG] serf: messageLeaveType: ip-172-21-71-254
Aug 11 08:07:42 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:42 [INFO] serf: EventMemberLeave (forced): ip-172-21-71-254 172.21.71.254
Aug 11 08:07:42 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:42 [INFO] consul: Removing LAN server ip-172-21-71-254 (Addr: tcp/172.21.71.254:8300) (DC: eu-west-1)
Aug 11 08:07:42 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:42 [DEBUG] serf: messageLeaveType: ip-172-21-71-254
Aug 11 08:07:42 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:42 [DEBUG] serf: messageLeaveType: ip-172-21-71-254
Aug 11 08:07:42 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:42 [DEBUG] serf: messageLeaveType: ip-172-21-71-254
Aug 11 08:07:42 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:42 [DEBUG] serf: messageLeaveType: ip-172-21-71-254
Aug 11 08:07:47 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:47 [WARN] consul: error getting server health from "ip-172-21-71-254": rpc error: failed to get conn: dial tcp 172.21.70.171:0->172.21.71.254:8300: i/o timeout
Aug 11 08:07:48 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:48 [DEBUG] agent: Node info in sync
Aug 11 08:07:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:56 [DEBUG] http: Request GET /v1/status/peers (85.895µs) from=127.0.0.1:33480
Aug 11 08:07:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:56 [DEBUG] http: Request GET /v1/status/leader (10.928µs) from=127.0.0.1:33480
Aug 11 08:07:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:56 [DEBUG] http: Request GET /v1/catalog/nodes (1.138312ms) from=127.0.0.1:33480
Aug 11 08:07:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:56 [DEBUG] http: Request GET /v1/catalog/services (849.329µs) from=127.0.0.1:33480
Aug 11 08:07:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:56 [DEBUG] http: Request GET /v1/health/service/consul (1.344921ms) from=127.0.0.1:33480
Aug 11 08:07:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:07:56 [DEBUG] http: Request GET /v1/health/state/any (865.809µs) from=127.0.0.1:33480
Aug 11 08:08:04 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:04 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.72.140:8302
Aug 11 08:08:04 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:04 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.71.94:8301
Aug 11 08:08:09 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:09 [INFO] memberlist: Suspect ip-172-21-71-254.eu-west-1 has failed, no acks received
Aug 11 08:08:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:11 [DEBUG] http: Request GET /v1/status/peers (41.364µs) from=127.0.0.1:33480
Aug 11 08:08:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:11 [DEBUG] http: Request GET /v1/status/leader (11.709µs) from=127.0.0.1:33480
Aug 11 08:08:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:11 [DEBUG] http: Request GET /v1/catalog/nodes (1.073689ms) from=127.0.0.1:33480
Aug 11 08:08:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:11 [DEBUG] http: Request GET /v1/catalog/services (826.37µs) from=127.0.0.1:33480
Aug 11 08:08:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:11 [DEBUG] http: Request GET /v1/health/service/consul (1.151861ms) from=127.0.0.1:33480
Aug 11 08:08:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:11 [DEBUG] http: Request GET /v1/health/state/any (888.808µs) from=127.0.0.1:33480
Aug 11 08:08:17 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:17 [DEBUG] memberlist: Stream connection from=172.21.71.126:55188
Aug 11 08:08:17 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:17 [INFO] serf: EventMemberJoin: ip-172-21-71-126 172.21.71.126
Aug 11 08:08:17 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:17 [INFO] consul: Adding LAN server ip-172-21-71-126 (Addr: tcp/172.21.71.126:8300) (DC: eu-west-1)
Aug 11 08:08:17 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:17 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.71.126:8302
Aug 11 08:08:17 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:17 [INFO] serf: EventMemberJoin: ip-172-21-71-126.eu-west-1 172.21.71.126
Aug 11 08:08:17 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:17 [DEBUG] consul: Successfully performed flood-join for "ip-172-21-71-126" at 172.21.71.126:8302
Aug 11 08:08:17 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:17 [INFO] consul: Handled member-join event for server "ip-172-21-71-126.eu-west-1" in area "wan"
Aug 11 08:08:17 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:17 [DEBUG] serf: messageJoinType: ip-172-21-71-126
Aug 11 08:08:17 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:17 [DEBUG] serf: messageJoinType: ip-172-21-72-140.eu-west-1
Aug 11 08:08:17 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:17 [DEBUG] serf: messageJoinType: ip-172-21-71-126
Aug 11 08:08:17 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:17 [DEBUG] serf: messageJoinType: ip-172-21-71-126
Aug 11 08:08:17 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:17 [DEBUG] serf: messageJoinType: ip-172-21-71-126
Aug 11 08:08:17 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:17 [DEBUG] serf: messageJoinType: ip-172-21-71-126
Aug 11 08:08:18 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:18 [DEBUG] serf: messageJoinType: ip-172-21-70-171.eu-west-1
Aug 11 08:08:18 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:18 [DEBUG] serf: messageJoinType: ip-172-21-71-94.eu-west-1
Aug 11 08:08:18 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:18 [DEBUG] serf: messageJoinType: ip-172-21-72-140.eu-west-1
Aug 11 08:08:18 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:18 [DEBUG] serf: messageJoinType: ip-172-21-72-106.eu-west-1
Aug 11 08:08:18 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:18 [DEBUG] serf: messageJoinType: ip-172-21-70-171.eu-west-1
Aug 11 08:08:18 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:18 [DEBUG] serf: messageJoinType: ip-172-21-72-106.eu-west-1
Aug 11 08:08:18 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:18 [DEBUG] serf: messageJoinType: ip-172-21-70-171.eu-west-1
Aug 11 08:08:19 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:19 [DEBUG] memberlist: Failed ping: ip-172-21-71-254.eu-west-1 (timeout reached)
Aug 11 08:08:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:21 [DEBUG] manager: Rebalanced 6 servers, next active server is ip-172-21-72-140.eu-west-1 (Addr: tcp/172.21.72.140:8300) (DC: eu-west-1)
Aug 11 08:08:26 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:26 [DEBUG] http: Request GET /v1/status/peers (35.841µs) from=127.0.0.1:33480
Aug 11 08:08:26 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:26 [DEBUG] http: Request GET /v1/status/leader (11.452µs) from=127.0.0.1:33480
Aug 11 08:08:26 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:26 [DEBUG] http: Request GET /v1/catalog/nodes (1.029863ms) from=127.0.0.1:33480
Aug 11 08:08:26 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:26 [DEBUG] http: Request GET /v1/catalog/services (903.097µs) from=127.0.0.1:33480
Aug 11 08:08:26 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:26 [DEBUG] http: Request GET /v1/health/service/consul (1.128185ms) from=127.0.0.1:33480
Aug 11 08:08:26 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:26 [DEBUG] http: Request GET /v1/health/state/any (934.513µs) from=127.0.0.1:33480
Aug 11 08:08:30 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:30 [INFO] memberlist: Marking ip-172-21-71-254.eu-west-1 as failed, suspect timeout reached (0 peer confirmations)
Aug 11 08:08:30 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:30 [INFO] serf: EventMemberFailed: ip-172-21-71-254.eu-west-1 172.21.71.254
Aug 11 08:08:30 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:30 [INFO] consul: Handled member-failed event for server "ip-172-21-71-254.eu-west-1" in area "wan"
Aug 11 08:08:34 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:34 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.71.126:8301
Aug 11 08:08:41 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:41 [DEBUG] http: Request GET /v1/status/peers (80.635µs) from=127.0.0.1:33480
Aug 11 08:08:41 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:41 [DEBUG] http: Request GET /v1/status/leader (12.718µs) from=127.0.0.1:33480
Aug 11 08:08:41 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:41 [DEBUG] http: Request GET /v1/catalog/nodes (1.135124ms) from=127.0.0.1:33480
Aug 11 08:08:41 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:41 [DEBUG] http: Request GET /v1/catalog/services (879.401µs) from=127.0.0.1:33480
Aug 11 08:08:41 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:41 [DEBUG] http: Request GET /v1/health/service/consul (1.217432ms) from=127.0.0.1:33480
Aug 11 08:08:41 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:41 [DEBUG] http: Request GET /v1/health/state/any (3.0158ms) from=127.0.0.1:33480
Aug 11 08:08:48 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:48 [DEBUG] memberlist: Stream connection from=172.21.72.106:58082
Aug 11 08:08:49 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:49 [INFO] memberlist: Suspect ip-172-21-71-254.eu-west-1 has failed, no acks received
Aug 11 08:08:51 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:51 [DEBUG] serf: forgoing reconnect for random throttling
Aug 11 08:08:52 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:52 [DEBUG] memberlist: Stream connection from=172.21.72.140:50398
Aug 11 08:08:55 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:55 [DEBUG] memberlist: Stream connection from=172.21.72.140:36318
Aug 11 08:08:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:56 [DEBUG] http: Request GET /v1/status/peers (41.154µs) from=127.0.0.1:33480
Aug 11 08:08:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:56 [DEBUG] http: Request GET /v1/status/leader (11.538µs) from=127.0.0.1:33480
Aug 11 08:08:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:56 [DEBUG] http: Request GET /v1/catalog/nodes (1.13339ms) from=127.0.0.1:33480
Aug 11 08:08:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:56 [DEBUG] http: Request GET /v1/catalog/services (927.173µs) from=127.0.0.1:33480
Aug 11 08:08:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:56 [DEBUG] http: Request GET /v1/health/service/consul (1.138461ms) from=127.0.0.1:33480
Aug 11 08:08:56 ip-172-21-70-171 consul[1396]: 2017/08/11 08:08:56 [DEBUG] http: Request GET /v1/health/state/any (923.485µs) from=127.0.0.1:33480
Aug 11 08:09:01 ip-172-21-70-171 consul[1396]: 2017/08/11 08:09:01 [DEBUG] memberlist: Stream connection from=172.21.72.106:33744
Aug 11 08:09:04 ip-172-21-70-171 consul[1396]: 2017/08/11 08:09:04 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.72.140:8302
Aug 11 08:09:04 ip-172-21-70-171 consul[1396]: 2017/08/11 08:09:04 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.72.140:8301
Aug 11 08:09:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:09:11 [DEBUG] http: Request GET /v1/status/peers (72.839µs) from=127.0.0.1:33480
Aug 11 08:09:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:09:11 [DEBUG] http: Request GET /v1/status/leader (12.127µs) from=127.0.0.1:33480
Aug 11 08:09:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:09:11 [DEBUG] http: Request GET /v1/catalog/nodes (1.08335ms) from=127.0.0.1:33480
Aug 11 08:09:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:09:11 [DEBUG] http: Request GET /v1/catalog/services (897.232µs) from=127.0.0.1:33480
Aug 11 08:09:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:09:11 [DEBUG] http: Request GET /v1/health/service/consul (1.111294ms) from=127.0.0.1:33480
Aug 11 08:09:11 ip-172-21-70-171 consul[1396]: 2017/08/11 08:09:11 [DEBUG] http: Request GET /v1/health/state/any (905.885µs) from=127.0.0.1:33480
Aug 11 08:09:18 ip-172-21-70-171 consul[1396]: 2017/08/11 08:09:18 [DEBUG] memberlist: Stream connection from=172.21.72.106:58088
Aug 11 08:09:19 ip-172-21-70-171 consul[1396]: 2017/08/11 08:09:19 [DEBUG] memberlist: Stream connection from=172.21.71.94:52248
Aug 11 08:09:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:09:21 [DEBUG] serf: forgoing reconnect for random throttling
Aug 11 08:09:21 ip-172-21-70-171 consul[1396]: 2017/08/11 08:09:21 [WARN] consul: error getting server health from "ip-172-21-72-106": rpc error: stream closed
Aug 11 08:09:22 ip-172-21-70-171 consul[1396]: 2017/08/11 08:09:22 [WARN] consul: error getting server health from "ip-172-21-72-106": context deadline exceeded
Aug 11 08:09:22 ip-172-21-70-171 consul[1396]: 2017/08/11 08:09:22 [WARN] consul: error getting server health from "ip-172-21-70-171": context deadline exceeded
Aug 11 08:09:22 ip-172-21-70-171 consul[1396]: 2017/08/11 08:09:22 [WARN] consul: error getting server health from "ip-172-21-71-126": context deadline exceeded
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment