Skip to content

Instantly share code, notes, and snippets.

@dgulinobw

dgulinobw/BAD: syslog with raft_protocol=3

Last active August 8, 2017 19:21
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dgulinobw/dc84121837952701d4ff2e1d2b6d4a1a to your computer and use it in GitHub Desktop.
Save dgulinobw/dc84121837952701d4ff2e1d2b6d4a1a to your computer and use it in GitHub Desktop.
Download ZIP
Cluster of 5 running 0.9.0 will not elect a leader w/raft_protocol = 3
Raw
BAD: syslog with raft_protocol=3
Aug 8 16:46:21 consul-qa-aws05 systemd[1]: Starting Consul service discovery agent...
Aug 8 16:46:21 consul-qa-aws05 systemd[1]: Started Consul service discovery agent.
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: ==> WARNING: LAN keyring exists but -encrypt given, using keyring
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: ==> WARNING: WAN keyring exists but -encrypt given, using keyring
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: ==> WARNING: Expect Mode enabled, expecting 5 servers
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: ==> Starting Consul agent...
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: raft: Restored from snapshot 4-295004-1502204270457
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: raft: Initial configuration (index=298061): [{Suffrage:Voter ID:01b365e4-dc5f-a730-0744-2315187d31c7 Address:34.228.202.214:8300} {Suffrage:Voter ID:ac31e8ef-a1b9-3ecd-ee1f-b4e2326ac5ee Address:52.1.85.126:8300} {Suffrage:Voter ID:6d2a07da-d73e-f007-7e8f-f22d8e93b0c0 Address:34.207.18.6:8300} {Suffrage:Nonvoter ID:4ab193be-a6ba-5300-23de-0d25151983fe Address:34.206.132.179:8300}]
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: raft: Node at 34.207.18.6:8300 [Follower] entering Follower state (Leader: "")
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: consul-qa-aws05.us-east-1 34.207.18.6
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: Attempting re-join to previously known node: consul-qa-aws01.us-east-1: 34.228.202.214:8302
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: memberlist: Failed to join 34.228.202.214: dial tcp 34.228.202.214:8302: getsockopt: connection refused
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: Attempting re-join to previously known node: consul-qa-aws03.us-east-1: 34.206.132.179:8302
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: memberlist: Initiating push/pull sync with: 34.206.132.179:8302
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: consul-qa-aws03.us-east-1 34.206.132.179
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: Re-joined to previously known node: consul-qa-aws03.us-east-1: 34.206.132.179:8302
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: consul-qa-aws05 34.207.18.6
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: consul: Adding LAN server consul-qa-aws05 (Addr: tcp/34.207.18.6:8300) (DC: us-east-1)
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: consul: Raft data found, disabling bootstrap mode
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: Attempting re-join to previously known node: isw-proxy-qa-aws03: 54.83.1.188:8301
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: consul: Handled member-join event for server "consul-qa-aws05.us-east-1" in area "wan"
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: consul: Handled member-join event for server "consul-qa-aws03.us-east-1" in area "wan"
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: agent: Started DNS server 127.0.0.1:8600 (udp)
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: agent: Started DNS server 127.0.0.1:8600 (tcp)
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: memberlist: Initiating push/pull sync with: 54.83.1.188:8301
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: agent: Started HTTP server on 127.0.0.1:8500
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: ==> Consul agent running!
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: Version: 'v0.9.0'
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: Node ID: '6d2a07da-d73e-f007-7e8f-f22d8e93b0c0'
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: Node name: 'consul-qa-aws05'
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: Datacenter: 'us-east-1'
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: Server: true (bootstrap: false)
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 8600)
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: Cluster Addr: 10.1.24.66 (LAN: 8301, WAN: 8302)
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: Gossip encrypt: true, RPC-TLS: false, TLS-Incoming: false
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: ==> Log data will now stream in as it occurs:
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] raft: Restored from snapshot 4-295004-1502204270457
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] raft: Initial configuration (index=298061): [{Suffrage:Voter ID:01b365e4-dc5f-a730-0744-2315187d31c7 Address:34.228.202.214:8300} {Suffrage:Voter ID:ac31e8ef-a1b9-3ecd-ee1f-b4e2326ac5ee Address:52.1.85.126:8300} {Suffrage:Voter ID:6d2a07da-d73e-f007-7e8f-f22d8e93b0c0 Address:34.207.18.6:8300} {Suffrage:Nonvoter ID:4ab193be-a6ba-5300-23de-0d25151983fe Address:34.206.132.179:8300}]
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] raft: Node at 34.207.18.6:8300 [Follower] entering Follower state (Leader: "")
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: consul-qa-aws05.us-east-1 34.207.18.6
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: Attempting re-join to previously known node: consul-qa-aws01.us-east-1: 34.228.202.214:8302
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] memberlist: Failed to join 34.228.202.214: dial tcp 34.228.202.214:8302: getsockopt: connection refused
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: Attempting re-join to previously known node: consul-qa-aws03.us-east-1: 34.206.132.179:8302
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] memberlist: Initiating push/pull sync with: 34.206.132.179:8302
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: consul-qa-aws03.us-east-1 34.206.132.179
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: Re-joined to previously known node: consul-qa-aws03.us-east-1: 34.206.132.179:8302
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: consul-qa-aws05 34.207.18.6
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] consul: Adding LAN server consul-qa-aws05 (Addr: tcp/34.207.18.6:8300) (DC: us-east-1)
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] consul: Raft data found, disabling bootstrap mode
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: Attempting re-join to previously known node: isw-proxy-qa-aws03: 54.83.1.188:8301
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] consul: Handled member-join event for server "consul-qa-aws05.us-east-1" in area "wan"
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] consul: Handled member-join event for server "consul-qa-aws03.us-east-1" in area "wan"
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] memberlist: Initiating push/pull sync with: 54.83.1.188:8301
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] agent: Started HTTP server on 127.0.0.1:8500
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] agent: Joining cluster...
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] agent: (LAN) joining: [34.228.202.214 52.1.85.126 34.206.132.179 34.194.13.219 34.207.18.6]
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: agent: Joining cluster...
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: agent: (LAN) joining: [34.228.202.214 52.1.85.126 34.206.132.179 34.194.13.219 34.207.18.6]
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: isw-proxy-qa-aws01 174.129.26.223
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: consul-qa-aws04 34.194.13.219
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: isw-qa-aws01 52.70.226.202
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: isw-proxy-qa-aws01 174.129.26.223
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: consul-qa-aws04 34.194.13.219
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: isw-qa-aws01 52.70.226.202
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: isw-qa-aws05 52.23.26.213
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: isw-proxy-qa-aws03 54.83.1.188
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: consul-qa-aws03 34.206.132.179
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: p19echo-qa-aws01 23.23.160.51
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: isw-qa-aws03 52.22.206.120
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: isw-qa-aws05 52.23.26.213
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: isw-proxy-qa-aws03 54.83.1.188
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: consul-qa-aws03 34.206.132.179
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: p19echo-qa-aws01 23.23.160.51
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: isw-qa-aws03 52.22.206.120
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: isw-proxy-qa-aws02 54.227.252.144
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: isw-qa-aws02 52.23.145.237
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: isw-qa-aws07 52.23.41.160
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: isw-qa-aws04 52.23.32.193
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: igate-qa-aws01 52.5.230.199
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] consul: Adding LAN server consul-qa-aws04 (Addr: tcp/34.194.13.219:8300) (DC: us-east-1)
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] consul: Adding LAN server consul-qa-aws03 (Addr: tcp/34.206.132.179:8300) (DC: us-east-1)
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] memberlist: Initiating push/pull sync with: 34.228.202.214:8301
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: isw-proxy-qa-aws02 54.227.252.144
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: isw-qa-aws02 52.23.145.237
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: isw-qa-aws07 52.23.41.160
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: isw-qa-aws04 52.23.32.193
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: igate-qa-aws01 52.5.230.199
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] memberlist: Failed to join 34.194.13.219: dial tcp 34.194.13.219:8302: getsockopt: connection refused
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] consul: Failed to flood-join "consul-qa-aws04" at 34.194.13.219:8302: 1 error(s) occurred:
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: * Failed to join 34.194.13.219: dial tcp 34.194.13.219:8302: getsockopt: connection refused
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: Re-joined to previously known node: isw-proxy-qa-aws03: 54.83.1.188:8301
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: consul: Adding LAN server consul-qa-aws04 (Addr: tcp/34.194.13.219:8300) (DC: us-east-1)
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: consul: Adding LAN server consul-qa-aws03 (Addr: tcp/34.206.132.179:8300) (DC: us-east-1)
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: memberlist: Initiating push/pull sync with: 34.228.202.214:8301
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: memberlist: Failed to join 34.194.13.219: dial tcp 34.194.13.219:8302: getsockopt: connection refused
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: consul: Failed to flood-join "consul-qa-aws04" at 34.194.13.219:8302: 1 error(s) occurred:#012#012* Failed to join 34.194.13.219: dial tcp 34.194.13.219:8302: getsockopt: connection refused
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: Re-joined to previously known node: isw-proxy-qa-aws03: 54.83.1.188:8301
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: consul-qa-aws01 34.228.202.214
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] consul: Adding LAN server consul-qa-aws01 (Addr: tcp/34.228.202.214:8300) (DC: us-east-1)
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: consul-qa-aws01 34.228.202.214
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] memberlist: Stream connection from=34.206.132.179:65048
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] memberlist: Initiating push/pull sync with: 52.1.85.126:8301
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] memberlist: Initiating push/pull sync with: 34.194.13.219:8302
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: consul: Adding LAN server consul-qa-aws01 (Addr: tcp/34.228.202.214:8300) (DC: us-east-1)
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: memberlist: Stream connection from=34.206.132.179:65048
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: memberlist: Initiating push/pull sync with: 52.1.85.126:8301
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: memberlist: Initiating push/pull sync with: 34.194.13.219:8302
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: consul-qa-aws02 52.1.85.126
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] consul: Adding LAN server consul-qa-aws02 (Addr: tcp/52.1.85.126:8300) (DC: us-east-1)
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: consul-qa-aws02 52.1.85.126
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: consul: Adding LAN server consul-qa-aws02 (Addr: tcp/52.1.85.126:8300) (DC: us-east-1)
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] consul: Successfully performed flood-join for "consul-qa-aws04" at 34.194.13.219:8302
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: consul: Successfully performed flood-join for "consul-qa-aws04" at 34.194.13.219:8302
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] memberlist: Initiating push/pull sync with: 34.206.132.179:8301
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: memberlist: Initiating push/pull sync with: 34.206.132.179:8301
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] memberlist: Initiating push/pull sync with: 34.228.202.214:8302
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: memberlist: Initiating push/pull sync with: 34.228.202.214:8302
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] memberlist: Failed to join 34.194.13.219: dial tcp 34.194.13.219:8301: getsockopt: connection refused
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: consul-qa-aws01.us-east-1 34.228.202.214
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: memberlist: Failed to join 34.194.13.219: dial tcp 34.194.13.219:8301: getsockopt: connection refused
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: consul-qa-aws01.us-east-1 34.228.202.214
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberJoin: consul-qa-aws02.us-east-1 52.1.85.126
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] memberlist: Stream connection from=34.207.18.6:9218
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: memberlist: Stream connection from=34.207.18.6:9218
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] memberlist: Initiating push/pull sync with: 34.207.18.6:8301
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: memberlist: Initiating push/pull sync with: 34.207.18.6:8301
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: consul-qa-aws02.us-east-1 52.1.85.126
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] consul: Successfully performed flood-join for "consul-qa-aws01" at 34.228.202.214:8302
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: consul: Successfully performed flood-join for "consul-qa-aws01" at 34.228.202.214:8302
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] consul: Handled member-join event for server "consul-qa-aws01.us-east-1" in area "wan"
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: consul: Handled member-join event for server "consul-qa-aws01.us-east-1" in area "wan"
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] consul: Handled member-join event for server "consul-qa-aws02.us-east-1" in area "wan"
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: consul: Handled member-join event for server "consul-qa-aws02.us-east-1" in area "wan"
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] agent: (LAN) joined: 4 Err: <nil>
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: agent: (LAN) joined: 4 Err: <nil>
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] agent: systemd notify failed: No socket
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: agent: systemd notify failed: No socket
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] agent: Join completed. Synced with 4 initial agents
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: agent: Join completed. Synced with 4 initial agents
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] memberlist: Stream connection from=52.1.85.126:58388
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: memberlist: Stream connection from=52.1.85.126:58388
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] memberlist: Stream connection from=34.228.202.214:33310
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: memberlist: Stream connection from=34.228.202.214:33310
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [DEBUG] memberlist: Stream connection from=34.194.13.219:47014
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: memberlist: Stream connection from=34.194.13.219:47014
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:21 [INFO] serf: EventMemberUpdate: consul-qa-aws04
Aug 8 16:46:21 consul-qa-aws05 consul[6268]: serf: EventMemberUpdate: consul-qa-aws04
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws01
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws01
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws02
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws02
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws05
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws05
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws03
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws03
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [INFO] serf: EventMemberJoin: consul-qa-aws04.us-east-1 34.194.13.219
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: EventMemberJoin: consul-qa-aws04.us-east-1 34.194.13.219
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws03.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws03.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [INFO] consul: Handled member-join event for server "consul-qa-aws04.us-east-1" in area "wan"
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: consul: Handled member-join event for server "consul-qa-aws04.us-east-1" in area "wan"
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws03.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws03.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws02.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws02.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws02.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws02.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws05.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws05.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws02.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws02.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws03.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws03.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws05.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws05.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws02.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws02.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws03.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws03.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws01
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws01
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws04
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws04
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws02
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws02
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws02
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws02
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws01
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws01
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws04
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws04
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws03
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws03
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws05
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws05
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws05.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws05.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws05.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws05.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws02.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws02.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws05.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws05.us-east-1
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws01
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws01
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws04
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws04
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:22 [DEBUG] serf: messageJoinType: consul-qa-aws05
Aug 8 16:46:22 consul-qa-aws05 consul[6268]: serf: messageJoinType: consul-qa-aws05
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:27 [DEBUG] raft-net: 34.207.18.6:8300 accepted connection from: 52.1.85.126:4489
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: raft-net: 34.207.18.6:8300 accepted connection from: 52.1.85.126:4489
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:27 [DEBUG] raft-net: 34.207.18.6:8300 accepted connection from: 52.1.85.126:33205
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: raft-net: 34.207.18.6:8300 accepted connection from: 52.1.85.126:33205
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:27 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:27 [INFO] consul: New leader elected: consul-qa-aws02
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: consul: New leader elected: consul-qa-aws02
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:27 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:27 [INFO] agent: Synced node info
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: agent: Synced node info
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:27 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:27 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:27 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:27 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:27 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:31 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:31 [DEBUG] memberlist: Stream connection from=52.23.145.237:34884
Aug 8 16:46:31 consul-qa-aws05 consul[6268]: memberlist: Stream connection from=52.23.145.237:34884
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [INFO] consul: New leader elected: consul-qa-aws02
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: consul: New leader elected: consul-qa-aws02
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:35 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:35 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:41 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:41 [DEBUG] raft-net: 34.207.18.6:8300 accepted connection from: 52.1.85.126:33832
Aug 8 16:46:41 consul-qa-aws05 consul[6268]: raft-net: 34.207.18.6:8300 accepted connection from: 52.1.85.126:33832
Aug 8 16:46:41 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:41 [DEBUG] raft-net: 34.207.18.6:8300 accepted connection from: 52.1.85.126:22389
Aug 8 16:46:41 consul-qa-aws05 consul[6268]: raft-net: 34.207.18.6:8300 accepted connection from: 52.1.85.126:22389
Aug 8 16:46:41 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:41 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:41 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:41 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:41 [INFO] consul: New leader elected: consul-qa-aws02
Aug 8 16:46:41 consul-qa-aws05 consul[6268]: consul: New leader elected: consul-qa-aws02
Aug 8 16:46:41 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:41 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:41 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:41 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:41 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:41 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:41 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:41 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:41 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:42 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:42 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:42 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:42 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:42 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:42 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:47 [ERR] agent: Coordinate update error: rpc error: No cluster leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: agent: Coordinate update error: rpc error: No cluster leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:47 [WARN] raft: Failed to get previous log: 298090 log not found (last: 298088)
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: raft: Failed to get previous log: 298090 log not found (last: 298088)
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:47 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:47 [INFO] consul: New leader elected: consul-qa-aws02
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: consul: New leader elected: consul-qa-aws02
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:47 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:47 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:47 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:47 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:47 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:47 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:47 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:47 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:48 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:48 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:48 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:48 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:48 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:48 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:54 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:54 [ERR] http: Request GET /v1/operator/autopilot/configuration, error: rpc error: No cluster leader from=127.0.0.1:41192
Aug 8 16:46:54 consul-qa-aws05 consul[6268]: http: Request GET /v1/operator/autopilot/configuration, error: rpc error: No cluster leader from=127.0.0.1:41192
Aug 8 16:46:54 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:54 [DEBUG] http: Request GET /v1/operator/autopilot/configuration (7.280070555s) from=127.0.0.1:41192
Aug 8 16:46:54 consul-qa-aws05 consul[6268]: http: Request GET /v1/operator/autopilot/configuration (7.280070555s) from=127.0.0.1:41192
Aug 8 16:46:55 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:55 [WARN] raft: Heartbeat timeout from "52.1.85.126:8300" reached, starting election
Aug 8 16:46:55 consul-qa-aws05 consul[6268]: raft: Heartbeat timeout from "52.1.85.126:8300" reached, starting election
Aug 8 16:46:55 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:55 [INFO] raft: Node at 34.207.18.6:8300 [Candidate] entering Candidate state in term 16
Aug 8 16:46:55 consul-qa-aws05 consul[6268]: raft: Node at 34.207.18.6:8300 [Candidate] entering Candidate state in term 16
Aug 8 16:46:55 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:55 [DEBUG] raft: Votes needed: 2
Aug 8 16:46:55 consul-qa-aws05 consul[6268]: raft: Votes needed: 2
Aug 8 16:46:55 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:55 [DEBUG] raft: Vote granted from 6d2a07da-d73e-f007-7e8f-f22d8e93b0c0 in term 16. Tally: 1
Aug 8 16:46:55 consul-qa-aws05 consul[6268]: raft: Vote granted from 6d2a07da-d73e-f007-7e8f-f22d8e93b0c0 in term 16. Tally: 1
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:56 [INFO] raft: Node at 34.207.18.6:8300 [Follower] entering Follower state (Leader: "")
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: raft: Node at 34.207.18.6:8300 [Follower] entering Follower state (Leader: "")
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:56 [WARN] raft: Failed to get previous log: 298092 log not found (last: 298090)
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: raft: Failed to get previous log: 298092 log not found (last: 298090)
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:56 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:56 [INFO] consul: New leader elected: consul-qa-aws02
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: consul: New leader elected: consul-qa-aws02
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:56 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:56 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:56 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:56 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:56 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:56 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:56 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:56 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:56 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: 2017/08/08 16:46:56 [DEBUG] serf: messageUserEventType: consul:new-leader
Aug 8 16:46:56 consul-qa-aws05 consul[6268]: serf: messageUserEventType: consul:new-leader
Raw
consul.json
syslog w/raft_protocol=2
{
"bootstrap_expect": 5,
"retry_join": [ "a.a.a.a", "b.b.b.b", "c.c.c.c", "d.d.d.d", "e.e.e.e"],
"server": true,
"rejoin_after_leave": true,
"enable_syslog": true,
"data_dir": "/var/consul/data",
"datacenter": "us-east-1",
"recursor": "10.0.0.2",
"advertise_addrs": {
"serf_lan": "a.a.a.a:8301",
"serf_wan": "z.z.z.z:8302",
"rpc": "a.a.a.a:8300"
},
"bind_addr": "a.a.a.a",
"encrypt": "NANANANA",
"ui": true,
"encrypt_verify_incoming": true,
"encrypt_verify_outgoing": true,
"key_file": "/var/consul/data/pki/private/server.key",
"cert_file": "/var/consul/data/pki/certs/server.crt",
"ca_file": "/var/consul/data/pki/certs/ca.crt",
"log_level": "DEBUG",
"raft_protocol": 2
}
Raw
GOOD: syslog with raft_protocol=2
Aug 8 16:38:58 consul-qa-aws05 consul[6152]: 2017/08/08 16:38:58 [DEBUG] memberlist: Stream connection from=34.228.202.214:33132
Aug 8 16:38:58 consul-qa-aws05 consul[6152]: memberlist: Stream connection from=34.228.202.214:33132
Aug 8 16:38:58 consul-qa-aws05 consul[6152]: 2017/08/08 16:38:58 [INFO] serf: EventMemberUpdate: consul-qa-aws01
Aug 8 16:38:58 consul-qa-aws05 consul[6152]: serf: EventMemberUpdate: consul-qa-aws01
Aug 8 16:38:58 consul-qa-aws05 consul[6152]: 2017/08/08 16:38:58 [DEBUG] memberlist: Stream connection from=52.1.85.126:58224
Aug 8 16:38:58 consul-qa-aws05 consul[6152]: memberlist: Stream connection from=52.1.85.126:58224
Aug 8 16:38:58 consul-qa-aws05 consul[6152]: 2017/08/08 16:38:58 [DEBUG] serf: messageJoinType: consul-qa-aws01
Aug 8 16:38:58 consul-qa-aws05 consul[6152]: serf: messageJoinType: consul-qa-aws01
Aug 8 16:38:58 consul-qa-aws05 consul[6152]: 2017/08/08 16:38:58 [DEBUG] serf: messageJoinType: consul-qa-aws01
Aug 8 16:38:58 consul-qa-aws05 consul[6152]: serf: messageJoinType: consul-qa-aws01
Aug 8 16:38:59 consul-qa-aws05 consul[6152]: 2017/08/08 16:38:59 [DEBUG] serf: messageJoinType: consul-qa-aws01
Aug 8 16:38:59 consul-qa-aws05 consul[6152]: serf: messageJoinType: consul-qa-aws01
Aug 8 16:38:59 consul-qa-aws05 consul[6152]: 2017/08/08 16:38:59 [INFO] serf: EventMemberUpdate: consul-qa-aws01.us-east-1
Aug 8 16:38:59 consul-qa-aws05 consul[6152]: serf: EventMemberUpdate: consul-qa-aws01.us-east-1
Aug 8 16:38:59 consul-qa-aws05 consul[6152]: 2017/08/08 16:38:59 [DEBUG] serf: messageJoinType: consul-qa-aws01
Aug 8 16:38:59 consul-qa-aws05 consul[6152]: serf: messageJoinType: consul-qa-aws01
Aug 8 16:38:59 consul-qa-aws05 consul[6152]: 2017/08/08 16:38:59 [DEBUG] serf: messageJoinType: consul-qa-aws01
Aug 8 16:38:59 consul-qa-aws05 consul[6152]: serf: messageJoinType: consul-qa-aws01
Aug 8 16:39:14 consul-qa-aws05 consul[6152]: 2017/08/08 16:39:14 [DEBUG] memberlist: Initiating push/pull sync with: 52.5.230.199:8301
Aug 8 16:39:14 consul-qa-aws05 consul[6152]: memberlist: Initiating push/pull sync with: 52.5.230.199:8301
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment