Last active
March 28, 2017 12:00
-
-
Save ybubnov/e11a2ac5f16fc1dc3562db213a76935d to your computer and use it in GitHub Desktop.
Logs from the consul cluster
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ==> WARNING: the 'dc' flag has been deprecated. Use 'datacenter' instead | |
| ==> WARNING: Expect Mode enabled, expecting 3 servers | |
| ==> Starting Consul agent... | |
| ==> Starting Consul agent RPC... | |
| ==> Consul agent running! | |
| Version: 'v0.7.0' | |
| Node name: '10.2.1.0/24,172.28.5.71,3fc622a3e09b' | |
| Datacenter: 'dc1-3-santa_clara' | |
| Server: true (bootstrap: false) | |
| Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400) | |
| Cluster Addr: 10.2.1.1 (LAN: 8301, WAN: 8302) | |
| Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false | |
| Atlas: <disabled> | |
| ==> Log data will now stream in as it occurs: | |
| 2017/03/28 11:06:06 [INFO] raft: Initial configuration (index=0): [] | |
| 2017/03/28 11:06:06 [INFO] serf: EventMemberJoin: 10.2.1.0/24,172.28.5.71,3fc622a3e09b 10.2.1.1 | |
| 2017/03/28 11:06:06 [INFO] serf: EventMemberJoin: 10.2.1.0/24,172.28.5.71,3fc622a3e09b.dc1-3-santa_clara 10.2.1.1 | |
| 2017/03/28 11:06:06 [INFO] raft: Node at 10.2.1.1:8300 [Follower] entering Follower state (Leader: "") | |
| 2017/03/28 11:06:06 [INFO] consul: Adding LAN server 10.2.1.0/24,172.28.5.71,3fc622a3e09b (Addr: tcp/10.2.1.1:8300) (DC: dc1-3-santa_clara) | |
| 2017/03/28 11:06:06 [INFO] consul: Adding WAN server 10.2.1.0/24,172.28.5.71,3fc622a3e09b.dc1-3-santa_clara (Addr: tcp/10.2.1.1:8300) (DC: dc1-3-santa_clara) | |
| 2017/03/28 11:06:06 [DEBUG] agent: pausing 26.09298699s before first invocation of /bin/sh -c /usr/bin/agent healthcheck in container agent | |
| 2017/03/28 11:06:06 [DEBUG] Automatically reaping child processes | |
| 2017/03/28 11:06:06 [INFO] agent: Joining cluster... | |
| 2017/03/28 11:06:06 [INFO] agent: (LAN) joining: [10.2.1.1 10.2.2.1 10.2.3.1] | |
| 2017/03/28 11:06:06 [DEBUG] memberlist: TCP connection from=172.28.5.71:46750 | |
| 2017/03/28 11:06:06 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:06:06 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:06:06 [INFO] serf: EventMemberJoin: 10.2.2.0/24,172.28.5.16,19883068ab3c 10.2.2.1 | |
| 2017/03/28 11:06:06 [INFO] serf: EventMemberJoin: 10.2.3.0/24,172.28.5.211,808195027ade 10.2.3.1 | |
| 2017/03/28 11:06:06 [INFO] consul: Adding LAN server 10.2.2.0/24,172.28.5.16,19883068ab3c (Addr: tcp/10.2.2.1:8300) (DC: dc1-3-santa_clara) | |
| 2017/03/28 11:06:06 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:06:06 [INFO] agent: (LAN) joined: 3 Err: <nil> | |
| 2017/03/28 11:06:06 [INFO] agent: Join completed. Synced with 3 initial agents | |
| 2017/03/28 11:06:06 [INFO] consul: Existing Raft peers reported by 10.2.2.0/24,172.28.5.16,19883068ab3c, disabling bootstrap mode | |
| 2017/03/28 11:06:06 [INFO] consul: Adding LAN server 10.2.3.0/24,172.28.5.211,808195027ade (Addr: tcp/10.2.3.1:8300) (DC: dc1-3-santa_clara) | |
| 2017/03/28 11:06:06 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.211:55872 | |
| 2017/03/28 11:06:06 [WARN] raft: Failed to get previous log: 247 log not found (last: 0) | |
| 2017/03/28 11:06:07 [DEBUG] serf: messageJoinType: 10.2.1.0/24,172.28.5.71,3fc622a3e09b | |
| 2017/03/28 11:06:07 [INFO] agent: Synced service 'consul' | |
| 2017/03/28 11:06:07 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:06:07 [DEBUG] serf: messageJoinType: 10.2.1.0/24,172.28.5.71,3fc622a3e09b | |
| 2017/03/28 11:06:07 [DEBUG] serf: messageJoinType: 10.2.1.0/24,172.28.5.71,3fc622a3e09b | |
| 2017/03/28 11:06:07 [DEBUG] serf: messageJoinType: 10.2.1.0/24,172.28.5.71,3fc622a3e09b | |
| 2017/03/28 11:06:07 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.211:55874 | |
| 2017/03/28 11:06:11 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:06:11 [INFO] agent: Deregistered service '4655871dc1a52c6a90b654df9ae89977e28816808540a13b76e52a9b95b4a74e' | |
| 2017/03/28 11:06:11 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:06:11 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:06:11 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:06:35 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:06:35 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:06:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:06:48 [WARN] raft: Failed to get previous log: 261 log not found (last: 260) | |
| 2017/03/28 11:06:49 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.211:55940 | |
| 2017/03/28 11:07:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:07:17 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:07:17 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:07:27 [DEBUG] memberlist: TCP connection from=172.28.5.211:50944 | |
| 2017/03/28 11:07:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:48616 | |
| 2017/03/28 11:07:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:07:48 [WARN] raft: Failed to get previous log: 269 log not found (last: 268) | |
| 2017/03/28 11:07:49 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.211:56046 | |
| 2017/03/28 11:08:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:08:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:48724 | |
| 2017/03/28 11:08:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:08:48 [WARN] raft: Failed to get previous log: 278 log not found (last: 277) | |
| 2017/03/28 11:08:49 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.211:56152 | |
| 2017/03/28 11:08:57 [DEBUG] memberlist: TCP connection from=172.28.5.211:51104 | |
| 2017/03/28 11:09:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:48778 | |
| 2017/03/28 11:09:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:09:12 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:09:12 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:09:27 [DEBUG] memberlist: TCP connection from=172.28.5.211:51166 | |
| 2017/03/28 11:09:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:48842 | |
| 2017/03/28 11:09:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:09:49 [WARN] raft: Failed to get previous log: 288 log not found (last: 287) | |
| 2017/03/28 11:09:49 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.211:56268 | |
| 2017/03/28 11:10:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:48898 | |
| 2017/03/28 11:10:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:10:27 [DEBUG] memberlist: TCP connection from=172.28.5.211:51278 | |
| 2017/03/28 11:10:36 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:10:36 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:10:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:10:49 [WARN] raft: Failed to get previous log: 298 log not found (last: 297) | |
| 2017/03/28 11:10:49 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.211:56330 | |
| 2017/03/28 11:10:57 [DEBUG] memberlist: TCP connection from=172.28.5.211:51334 | |
| 2017/03/28 11:11:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:49006 | |
| 2017/03/28 11:11:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:11:12 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:11:12 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:11:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:49056 | |
| 2017/03/28 11:11:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:11:53 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:11:53 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:11:59 [WARN] raft: Heartbeat timeout from "10.2.3.1:8300" reached, starting election | |
| 2017/03/28 11:11:59 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 5 | |
| 2017/03/28 11:11:59 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:12:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:49110 | |
| 2017/03/28 11:12:06 [ERR] agent: coordinate update error: No cluster leader | |
| 2017/03/28 11:12:07 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:12:07 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 6 | |
| 2017/03/28 11:12:07 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:12:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:12:16 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:12:16 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 7 | |
| 2017/03/28 11:12:16 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:12:26 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:12:26 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 8 | |
| 2017/03/28 11:12:26 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:12:27 [DEBUG] memberlist: TCP connection from=172.28.5.211:51442 | |
| 2017/03/28 11:12:34 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:12:34 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 9 | |
| 2017/03/28 11:12:34 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:12:37 [ERR] agent: coordinate update error: No cluster leader | |
| 2017/03/28 11:12:40 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:12:40 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 10 | |
| 2017/03/28 11:12:40 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:12:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:12:47 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:12:47 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 11 | |
| 2017/03/28 11:12:47 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:12:54 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.16:37224 | |
| 2017/03/28 11:12:54 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:12:54 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 12 | |
| 2017/03/28 11:12:54 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:12:57 [DEBUG] memberlist: TCP connection from=172.28.5.211:51494 | |
| 2017/03/28 11:13:00 [INFO] raft: Node at 10.2.1.1:8300 [Follower] entering Follower state (Leader: "") | |
| 2017/03/28 11:13:00 [WARN] raft: Failed to get previous log: 315 log not found (last: 308) | |
| 2017/03/28 11:13:00 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:13:00 [INFO] consul: New leader elected: 10.2.2.0/24,172.28.5.16,19883068ab3c | |
| 2017/03/28 11:13:00 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:13:00 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:13:03 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:13:03 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:13:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:49218 | |
| 2017/03/28 11:13:08 [WARN] raft: Heartbeat timeout from "10.2.2.1:8300" reached, starting election | |
| 2017/03/28 11:13:08 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 14 | |
| 2017/03/28 11:13:08 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:13:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:13:14 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:13:14 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 15 | |
| 2017/03/28 11:13:14 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:13:20 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:13:20 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 16 | |
| 2017/03/28 11:13:20 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:13:28 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:13:28 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 17 | |
| 2017/03/28 11:13:28 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:13:37 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:13:37 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 18 | |
| 2017/03/28 11:13:37 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:13:38 [ERR] agent: coordinate update error: No cluster leader | |
| 2017/03/28 11:13:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:13:43 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:13:43 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 19 | |
| 2017/03/28 11:13:43 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:13:51 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:13:51 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 20 | |
| 2017/03/28 11:13:51 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:13:57 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:13:57 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 21 | |
| 2017/03/28 11:13:57 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:14:00 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.16:37336 | |
| 2017/03/28 11:14:01 [ERR] agent: coordinate update error: No cluster leader | |
| 2017/03/28 11:14:04 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:14:04 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 22 | |
| 2017/03/28 11:14:04 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:14:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:49324 | |
| 2017/03/28 11:14:07 [INFO] raft: Node at 10.2.1.1:8300 [Follower] entering Follower state (Leader: "") | |
| 2017/03/28 11:14:07 [WARN] raft: Failed to get previous log: 327 log not found (last: 318) | |
| 2017/03/28 11:14:07 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.16:37388 | |
| 2017/03/28 11:14:07 [WARN] raft: Failed to get previous log: 331 log not found (last: 330) | |
| 2017/03/28 11:14:07 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:14:07 [INFO] consul: New leader elected: 10.2.2.0/24,172.28.5.16,19883068ab3c | |
| 2017/03/28 11:14:07 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:14:07 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:14:07 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:14:08 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.16:37402 | |
| 2017/03/28 11:14:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:14:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:49344 | |
| 2017/03/28 11:14:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:14:57 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:14:57 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:14:57 [DEBUG] memberlist: TCP connection from=172.28.5.211:51724 | |
| 2017/03/28 11:15:07 [WARN] raft: Failed to get previous log: 342 log not found (last: 341) | |
| 2017/03/28 11:15:08 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.16:37462 | |
| 2017/03/28 11:15:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:15:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:16:03 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:16:03 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:16:07 [WARN] raft: Failed to get previous log: 353 log not found (last: 352) | |
| 2017/03/28 11:16:08 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.16:37572 | |
| 2017/03/28 11:16:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:16:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:16:57 [DEBUG] memberlist: TCP connection from=172.28.5.211:51938 | |
| 2017/03/28 11:17:07 [WARN] raft: Failed to get previous log: 362 log not found (last: 361) | |
| 2017/03/28 11:17:08 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.16:37680 | |
| 2017/03/28 11:17:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:17:26 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:17:26 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:17:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:49672 | |
| 2017/03/28 11:17:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:17:57 [DEBUG] memberlist: TCP connection from=172.28.5.211:52044 | |
| 2017/03/28 11:18:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:49726 | |
| 2017/03/28 11:18:07 [WARN] raft: Failed to get previous log: 372 log not found (last: 371) | |
| 2017/03/28 11:18:08 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.16:37788 | |
| 2017/03/28 11:18:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:18:27 [DEBUG] memberlist: TCP connection from=172.28.5.211:52094 | |
| 2017/03/28 11:18:31 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:18:31 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:18:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:49782 | |
| 2017/03/28 11:18:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:18:45 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:18:45 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:18:57 [DEBUG] memberlist: TCP connection from=172.28.5.211:52146 | |
| 2017/03/28 11:19:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:49836 | |
| 2017/03/28 11:19:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:19:15 [WARN] raft: Heartbeat timeout from "10.2.2.1:8300" reached, starting election | |
| 2017/03/28 11:19:15 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 24 | |
| 2017/03/28 11:19:15 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:19:20 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:19:20 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 25 | |
| 2017/03/28 11:19:20 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:19:27 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:19:27 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 26 | |
| 2017/03/28 11:19:27 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:19:27 [DEBUG] memberlist: TCP connection from=172.28.5.211:52208 | |
| 2017/03/28 11:19:32 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:19:32 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 27 | |
| 2017/03/28 11:19:32 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:19:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:49898 | |
| 2017/03/28 11:19:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:19:40 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:19:40 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 28 | |
| 2017/03/28 11:19:40 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:19:43 [ERR] agent: coordinate update error: No cluster leader | |
| 2017/03/28 11:19:49 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:19:49 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 29 | |
| 2017/03/28 11:19:49 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:19:57 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:19:57 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 30 | |
| 2017/03/28 11:19:57 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:19:57 [DEBUG] memberlist: TCP connection from=172.28.5.211:52210 | |
| 2017/03/28 11:20:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:49958 | |
| 2017/03/28 11:20:05 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:20:05 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 31 | |
| 2017/03/28 11:20:05 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:20:07 [ERR] agent: coordinate update error: No cluster leader | |
| 2017/03/28 11:20:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:20:14 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:20:14 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 32 | |
| 2017/03/28 11:20:14 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:20:21 [INFO] raft: Node at 10.2.1.1:8300 [Follower] entering Follower state (Leader: "") | |
| 2017/03/28 11:20:21 [WARN] raft: Failed to get previous log: 388 log not found (last: 381) | |
| 2017/03/28 11:20:21 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:20:21 [INFO] consul: New leader elected: 10.2.3.0/24,172.28.5.211,808195027ade | |
| 2017/03/28 11:20:21 [ERR] raft-net: Failed to flush response: write tcp 10.2.1.2:8300->172.28.5.211:56330: write: broken pipe | |
| 2017/03/28 11:20:21 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.211:57314 | |
| 2017/03/28 11:20:21 [WARN] raft: Failed to get previous log: 392 log not found (last: 391) | |
| 2017/03/28 11:20:21 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:20:21 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:20:21 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:20:21 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:20:21 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.211:57318 | |
| 2017/03/28 11:20:27 [DEBUG] memberlist: TCP connection from=172.28.5.211:52270 | |
| 2017/03/28 11:20:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:21:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:21:27 [WARN] raft: Heartbeat timeout from "10.2.3.1:8300" reached, starting election | |
| 2017/03/28 11:21:27 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 34 | |
| 2017/03/28 11:21:27 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:21:27 [DEBUG] memberlist: TCP connection from=172.28.5.211:52380 | |
| 2017/03/28 11:21:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:50126 | |
| 2017/03/28 11:21:36 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:21:36 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 35 | |
| 2017/03/28 11:21:36 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:21:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:21:42 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:21:42 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 36 | |
| 2017/03/28 11:21:42 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:21:47 [ERR] agent: coordinate update error: No cluster leader | |
| 2017/03/28 11:21:49 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:21:49 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 37 | |
| 2017/03/28 11:21:49 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:21:55 [ERR] agent: failed to sync remote state: No cluster leader | |
| 2017/03/28 11:21:55 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:21:55 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 38 | |
| 2017/03/28 11:21:55 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:22:03 [ERR] agent: failed to sync remote state: No cluster leader | |
| 2017/03/28 11:22:05 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:22:05 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 39 | |
| 2017/03/28 11:22:05 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:22:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:22:14 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:22:14 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 40 | |
| 2017/03/28 11:22:14 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:22:17 [ERR] agent: coordinate update error: No cluster leader | |
| 2017/03/28 11:22:22 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:22:22 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 41 | |
| 2017/03/28 11:22:22 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:22:27 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.211:57538 | |
| 2017/03/28 11:22:27 [INFO] raft: Node at 10.2.1.1:8300 [Follower] entering Follower state (Leader: "") | |
| 2017/03/28 11:22:27 [WARN] raft: Failed to get previous log: 409 log not found (last: 402) | |
| 2017/03/28 11:22:27 [WARN] raft: Failed to get previous log: 413 log not found (last: 402) | |
| 2017/03/28 11:22:27 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:22:27 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:22:27 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:22:27 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:22:27 [INFO] consul: New leader elected: 10.2.3.0/24,172.28.5.211,808195027ade | |
| 2017/03/28 11:22:27 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:22:27 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:22:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:23:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:23:27 [DEBUG] memberlist: TCP connection from=172.28.5.211:52596 | |
| 2017/03/28 11:23:32 [WARN] raft: Heartbeat timeout from "10.2.3.1:8300" reached, starting election | |
| 2017/03/28 11:23:32 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 43 | |
| 2017/03/28 11:23:32 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:23:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:23:40 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:23:40 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 44 | |
| 2017/03/28 11:23:40 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:23:49 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:23:49 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 45 | |
| 2017/03/28 11:23:49 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:23:50 [ERR] agent: coordinate update error: No cluster leader | |
| 2017/03/28 11:23:56 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:23:56 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 46 | |
| 2017/03/28 11:23:56 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:23:57 [DEBUG] memberlist: TCP connection from=172.28.5.211:52648 | |
| 2017/03/28 11:24:01 [ERR] agent: failed to sync remote state: No cluster leader | |
| 2017/03/28 11:24:02 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:24:02 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 47 | |
| 2017/03/28 11:24:02 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:24:08 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:24:08 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 48 | |
| 2017/03/28 11:24:08 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:24:09 [ERR] agent: failed to sync remote state: No cluster leader | |
| 2017/03/28 11:24:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:24:15 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:24:15 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 49 | |
| 2017/03/28 11:24:15 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:24:21 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:24:21 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 50 | |
| 2017/03/28 11:24:21 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:24:23 [ERR] agent: coordinate update error: No cluster leader | |
| 2017/03/28 11:24:26 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:24:26 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 51 | |
| 2017/03/28 11:24:26 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:24:32 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:24:32 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 52 | |
| 2017/03/28 11:24:32 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:24:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:50406 | |
| 2017/03/28 11:24:35 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.211:57766 | |
| 2017/03/28 11:24:35 [INFO] raft: Node at 10.2.1.1:8300 [Follower] entering Follower state (Leader: "") | |
| 2017/03/28 11:24:35 [WARN] raft: Failed to get previous log: 430 log not found (last: 422) | |
| 2017/03/28 11:24:35 [WARN] raft: Failed to get previous log: 434 log not found (last: 430) | |
| 2017/03/28 11:24:35 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:24:35 [INFO] consul: New leader elected: 10.2.3.0/24,172.28.5.211,808195027ade | |
| 2017/03/28 11:24:35 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:24:35 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:24:35 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:24:36 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:24:36 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:24:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:24:57 [DEBUG] memberlist: TCP connection from=172.28.5.211:52770 | |
| 2017/03/28 11:25:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:50462 | |
| 2017/03/28 11:25:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:25:29 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:25:29 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:25:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:50516 | |
| 2017/03/28 11:25:35 [WARN] raft: Failed to get previous log: 445 log not found (last: 444) | |
| 2017/03/28 11:25:36 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.211:57880 | |
| 2017/03/28 11:25:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:26:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:26:25 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:26:25 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:26:27 [DEBUG] memberlist: TCP connection from=172.28.5.211:52934 | |
| 2017/03/28 11:26:35 [WARN] raft: Failed to get previous log: 455 log not found (last: 454) | |
| 2017/03/28 11:26:36 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.211:57986 | |
| 2017/03/28 11:26:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 11:26:57 [DEBUG] memberlist: TCP connection from=172.28.5.211:52988 | |
| 2017/03/28 11:27:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:50682 | |
| 2017/03/28 11:27:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:27:27 [DEBUG] memberlist: TCP connection from=172.28.5.211:53040 | |
| 2017/03/28 11:27:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:50738 | |
| 2017/03/28 11:27:40 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:27:40 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:27:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:27:41 [WARN] raft: Heartbeat timeout from "10.2.3.1:8300" reached, starting election | |
| 2017/03/28 11:27:41 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 54 | |
| 2017/03/28 11:27:41 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:27:46 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:27:46 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 55 | |
| 2017/03/28 11:27:46 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:27:55 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:27:55 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 56 | |
| 2017/03/28 11:27:55 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:27:57 [DEBUG] memberlist: TCP connection from=172.28.5.211:53042 | |
| 2017/03/28 11:28:00 [ERR] agent: coordinate update error: No cluster leader | |
| 2017/03/28 11:28:00 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:28:00 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 57 | |
| 2017/03/28 11:28:00 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:28:07 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:28:07 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 58 | |
| 2017/03/28 11:28:07 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:28:10 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:28:15 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:28:15 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 59 | |
| 2017/03/28 11:28:15 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:28:22 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:28:22 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 60 | |
| 2017/03/28 11:28:22 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:28:29 [ERR] agent: coordinate update error: No cluster leader | |
| 2017/03/28 11:28:31 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:28:31 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 61 | |
| 2017/03/28 11:28:31 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:28:37 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:28:37 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 62 | |
| 2017/03/28 11:28:37 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:28:40 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:28:43 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 11:28:43 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 63 | |
| 2017/03/28 11:28:43 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:28:44 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.211:58202 | |
| 2017/03/28 11:28:44 [INFO] raft: Node at 10.2.1.1:8300 [Follower] entering Follower state (Leader: "") | |
| 2017/03/28 11:28:44 [WARN] raft: Failed to get previous log: 472 log not found (last: 465) | |
| 2017/03/28 11:28:44 [ERR] raft-net: Failed to flush response: write tcp 10.2.1.2:8300->172.28.5.211:58202: write: broken pipe | |
| 2017/03/28 11:28:44 [WARN] raft: Failed to get previous log: 476 log not found (last: 475) | |
| 2017/03/28 11:28:44 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:28:44 [INFO] consul: New leader elected: 10.2.3.0/24,172.28.5.211,808195027ade | |
| 2017/03/28 11:28:44 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:28:44 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:28:44 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:28:44 [DEBUG] raft-net: 10.2.1.1:8300 accepted connection from: 172.28.5.211:58204 | |
| 2017/03/28 11:29:01 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:29:01 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:29:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:50898 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ==> WARNING: the 'dc' flag has been deprecated. Use 'datacenter' instead | |
| ==> WARNING: Expect Mode enabled, expecting 3 servers | |
| ==> Starting Consul agent... | |
| ==> Starting Consul agent RPC... | |
| ==> Consul agent running! | |
| Version: 'v0.7.0' | |
| Node name: '10.2.1.0/24,172.28.5.71,c2bd8b49b4ae' | |
| Datacenter: 'dc1-3-santa_clara' | |
| Server: true (bootstrap: false) | |
| Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400) | |
| Cluster Addr: 10.2.1.1 (LAN: 8301, WAN: 8302) | |
| Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false | |
| Atlas: <disabled> | |
| ==> Log data will now stream in as it occurs: | |
| 2017/03/28 10:33:37 [INFO] raft: Initial configuration (index=0): [] | |
| 2017/03/28 10:33:37 [INFO] serf: EventMemberJoin: 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae 10.2.1.1 | |
| 2017/03/28 10:33:37 [INFO] raft: Node at 10.2.1.1:8300 [Follower] entering Follower state (Leader: "") | |
| 2017/03/28 10:33:37 [INFO] serf: EventMemberJoin: 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae.dc1-3-santa_clara 10.2.1.1 | |
| 2017/03/28 10:33:37 [INFO] consul: Adding LAN server 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae (Addr: tcp/10.2.1.1:8300) (DC: dc1-3-santa_clara) | |
| 2017/03/28 10:33:37 [INFO] consul: Adding WAN server 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae.dc1-3-santa_clara (Addr: tcp/10.2.1.1:8300) (DC: dc1-3-santa_clara) | |
| 2017/03/28 10:33:37 [DEBUG] agent: pausing 6.754290957s before first invocation of /bin/sh -c /usr/bin/agent healthcheck in container agent | |
| 2017/03/28 10:33:37 [DEBUG] Automatically reaping child processes | |
| 2017/03/28 10:33:37 [INFO] agent: Joining cluster... | |
| 2017/03/28 10:33:37 [INFO] agent: (LAN) joining: [10.2.1.1 10.2.2.1 10.2.3.1] | |
| 2017/03/28 10:33:37 [DEBUG] memberlist: TCP connection from=172.28.5.71:45772 | |
| 2017/03/28 10:33:37 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 10:33:37 [DEBUG] memberlist: Failed to join 10.2.2.1: dial tcp 10.2.2.1:8301: getsockopt: connection refused | |
| 2017/03/28 10:33:37 [DEBUG] memberlist: Failed to join 10.2.3.1: dial tcp 10.2.3.1:8301: getsockopt: connection refused | |
| 2017/03/28 10:33:37 [INFO] agent: (LAN) joined: 1 Err: <nil> | |
| 2017/03/28 10:33:37 [INFO] agent: Join completed. Synced with 1 initial agents | |
| 2017/03/28 10:33:42 [WARN] raft: no known peers, aborting election | |
| 2017/03/28 10:33:44 [ERR] agent: failed to sync remote state: No cluster leader | |
| 2017/03/28 10:33:48 [ERR] agent: failed to sync changes: No cluster leader | |
| 2017/03/28 10:33:53 [DEBUG] memberlist: TCP connection from=172.28.5.16:45004 | |
| 2017/03/28 10:33:53 [INFO] serf: EventMemberJoin: 10.2.2.0/24,172.28.5.16,19883068ab3c 10.2.2.1 | |
| 2017/03/28 10:33:53 [INFO] consul: Adding LAN server 10.2.2.0/24,172.28.5.16,19883068ab3c (Addr: tcp/10.2.2.1:8300) (DC: dc1-3-santa_clara) | |
| 2017/03/28 10:33:56 [ERR] agent: failed to sync changes: No cluster leader | |
| 2017/03/28 10:34:03 [DEBUG] serf: messageJoinType: 10.2.2.0/24,172.28.5.16,19883068ab3c | |
| 2017/03/28 10:34:04 [DEBUG] serf: messageJoinType: 10.2.2.0/24,172.28.5.16,19883068ab3c | |
| 2017/03/28 10:34:04 [DEBUG] serf: messageJoinType: 10.2.2.0/24,172.28.5.16,19883068ab3c | |
| 2017/03/28 10:34:04 [DEBUG] serf: messageJoinType: 10.2.2.0/24,172.28.5.16,19883068ab3c | |
| 2017/03/28 10:34:07 [ERR] agent: coordinate update error: No cluster leader | |
| 2017/03/28 10:34:09 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:34:11 [ERR] agent: failed to sync remote state: No cluster leader | |
| 2017/03/28 10:34:13 [ERR] agent: failed to sync changes: No cluster leader | |
| 2017/03/28 10:34:16 [DEBUG] memberlist: TCP connection from=172.28.5.211:47478 | |
| 2017/03/28 10:34:16 [INFO] serf: EventMemberJoin: 10.2.3.0/24,172.28.5.211,808195027ade 10.2.3.1 | |
| 2017/03/28 10:34:16 [INFO] consul: Adding LAN server 10.2.3.0/24,172.28.5.211,808195027ade (Addr: tcp/10.2.3.1:8300) (DC: dc1-3-santa_clara) | |
| 2017/03/28 10:34:16 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.2.1.1:8300,10.2.2.1:8300,10.2.3.1:8300 | |
| 2017/03/28 10:34:16 [DEBUG] serf: messageJoinType: 10.2.3.0/24,172.28.5.211,808195027ade | |
| 2017/03/28 10:34:16 [DEBUG] serf: messageJoinType: 10.2.3.0/24,172.28.5.211,808195027ade | |
| 2017/03/28 10:34:16 [DEBUG] serf: messageJoinType: 10.2.3.0/24,172.28.5.211,808195027ade | |
| 2017/03/28 10:34:17 [DEBUG] serf: messageJoinType: 10.2.3.0/24,172.28.5.211,808195027ade | |
| 2017/03/28 10:34:21 [WARN] raft: Heartbeat timeout from "" reached, starting election | |
| 2017/03/28 10:34:21 [INFO] raft: Node at 10.2.1.1:8300 [Candidate] entering Candidate state in term 2 | |
| 2017/03/28 10:34:21 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 10:34:21 [DEBUG] raft: Vote granted from 10.2.1.1:8300 in term 2. Tally: 1 | |
| 2017/03/28 10:34:21 [DEBUG] raft: Vote granted from 10.2.2.1:8300 in term 2. Tally: 2 | |
| 2017/03/28 10:34:21 [INFO] raft: Election won. Tally: 2 | |
| 2017/03/28 10:34:21 [INFO] raft: Node at 10.2.1.1:8300 [Leader] entering Leader state | |
| 2017/03/28 10:34:21 [INFO] raft: Added peer 10.2.2.1:8300, starting replication | |
| 2017/03/28 10:34:21 [INFO] raft: Added peer 10.2.3.1:8300, starting replication | |
| 2017/03/28 10:34:21 [INFO] consul: cluster leadership acquired | |
| 2017/03/28 10:34:21 [INFO] consul: New leader elected: 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae | |
| 2017/03/28 10:34:21 [INFO] raft: pipelining replication to peer {Voter 10.2.2.1:8300 10.2.2.1:8300} | |
| 2017/03/28 10:34:21 [DEBUG] consul: reset tombstone GC to index 3 | |
| 2017/03/28 10:34:21 [INFO] consul: member '10.2.1.0/24,172.28.5.71,c2bd8b49b4ae' joined, marking health alive | |
| 2017/03/28 10:34:21 [INFO] consul: member '10.2.2.0/24,172.28.5.16,19883068ab3c' joined, marking health alive | |
| 2017/03/28 10:34:21 [INFO] raft: pipelining replication to peer {Voter 10.2.3.1:8300 10.2.3.1:8300} | |
| 2017/03/28 10:34:21 [INFO] consul: member '10.2.3.0/24,172.28.5.211,808195027ade' joined, marking health alive | |
| 2017/03/28 10:34:21 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 10:34:21 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 10:34:21 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 10:34:21 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 10:34:23 [INFO] agent: Synced service 'consul' | |
| 2017/03/28 10:34:23 [INFO] agent: Deregistered service 'f18e8d8ec6f20d24966d08fdc9f8f45f75146a9ae494e175a74e0849f3b0505c' | |
| 2017/03/28 10:34:23 [INFO] agent: Deregistered service '51cd03995eedba03e538fc6711b5d94cdf9ef1a459d9a24cef0cf936f8028834' | |
| 2017/03/28 10:34:23 [INFO] agent: Deregistered service 'c2fef4f24a918871e314fbb7854de71f77b003227f3d8e5ada26b2a8ffbaf4fb' | |
| 2017/03/28 10:34:23 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:34:28 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:34:28 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:34:39 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:34:57 [DEBUG] memberlist: TCP connection from=172.28.5.211:47600 | |
| 2017/03/28 10:35:09 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:35:27 [DEBUG] memberlist: TCP connection from=172.28.5.211:47656 | |
| 2017/03/28 10:35:39 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:35:42 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:35:42 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:36:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:45354 | |
| 2017/03/28 10:36:09 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:36:27 [DEBUG] memberlist: TCP connection from=172.28.5.211:47768 | |
| 2017/03/28 10:36:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:45406 | |
| 2017/03/28 10:36:39 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 10:37:09 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 10:37:27 [DEBUG] memberlist: TCP connection from=172.28.5.211:47880 | |
| 2017/03/28 10:37:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:45462 | |
| 2017/03/28 10:37:36 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:37:36 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:37:39 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 10:38:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:45514 | |
| 2017/03/28 10:38:09 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 10:38:39 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 10:38:54 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:38:54 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:38:57 [DEBUG] memberlist: TCP connection from=172.28.5.211:48048 | |
| 2017/03/28 10:39:09 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:39:21 [DEBUG] consul.state: Session e9598d79-90a2-7ccf-c7e7-cce56d0756d4 TTL expired | |
| 2017/03/28 10:39:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:45682 | |
| 2017/03/28 10:39:37 [INFO] agent.rpc: Accepted client: 127.0.0.1:52118 | |
| 2017/03/28 10:39:37 [INFO] agent.rpc: Graceful leave triggered | |
| 2017/03/28 10:39:37 [INFO] consul: server starting leave | |
| 2017/03/28 10:39:37 [INFO] raft: Updating configuration with RemoveServer (10.2.1.1:8300, ) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300}] | |
| 2017/03/28 10:39:37 [INFO] raft: Removed ourself, transitioning to follower | |
| 2017/03/28 10:39:37 [INFO] raft: Node at 10.2.1.1:8300 [Follower] entering Follower state (Leader: "") | |
| 2017/03/28 10:39:37 [INFO] consul: cluster leadership lost | |
| 2017/03/28 10:39:37 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.2.1:8300 10.2.2.1:8300} | |
| 2017/03/28 10:39:37 [INFO] serf: EventMemberLeave: 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae.dc1-3-santa_clara 10.2.1.1 | |
| 2017/03/28 10:39:37 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.3.1:8300 10.2.3.1:8300} | |
| 2017/03/28 10:39:37 [INFO] consul: Removing WAN server 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae.dc1-3-santa_clara (Addr: tcp/10.2.1.1:8300) (DC: dc1-3-santa_clara) | |
| 2017/03/28 10:39:37 [DEBUG] serf: messageLeaveType: 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae | |
| 2017/03/28 10:39:37 [DEBUG] serf: messageLeaveType: 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae | |
| 2017/03/28 10:39:37 [INFO] serf: EventMemberLeave: 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae 10.2.1.1 | |
| 2017/03/28 10:39:37 [INFO] consul: Removing LAN server 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae (Addr: tcp/10.2.1.1:8300) (DC: dc1-3-santa_clara) | |
| 2017/03/28 10:39:37 [DEBUG] serf: messageLeaveType: 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae | |
| 2017/03/28 10:39:37 [DEBUG] serf: messageLeaveType: 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae | |
| 2017/03/28 10:39:37 [INFO] agent: requesting shutdown | |
| 2017/03/28 10:39:37 [INFO] consul: shutting down server | |
| 2017/03/28 10:39:37 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:39:37 [ERR] agent: failed to sync changes: No cluster leader | |
| 2017/03/28 10:39:37 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:39:37 [ERR] agent: failed to sync changes: No cluster leader | |
| 2017/03/28 10:39:37 [INFO] agent: shutdown complete | |
| 2017/03/28 10:39:37 [DEBUG] http: Shutting down http server (0.0.0.0:8500) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ==> WARNING: the 'dc' flag has been deprecated. Use 'datacenter' instead | |
| ==> WARNING: Expect Mode enabled, expecting 3 servers | |
| ==> Starting Consul agent... | |
| ==> Starting Consul agent RPC... | |
| ==> Consul agent running! | |
| Version: 'v0.7.0' | |
| Node name: '10.2.3.0/24,172.28.5.211,808195027ade' | |
| Datacenter: 'dc1-3-santa_clara' | |
| Server: true (bootstrap: false) | |
| Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400) | |
| Cluster Addr: 10.2.3.1 (LAN: 8301, WAN: 8302) | |
| Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false | |
| Atlas: <disabled> | |
| ==> Log data will now stream in as it occurs: | |
| 2017/03/28 10:34:16 [INFO] raft: Initial configuration (index=0): [] | |
| 2017/03/28 10:34:16 [INFO] serf: EventMemberJoin: 10.2.3.0/24,172.28.5.211,808195027ade 10.2.3.1 | |
| 2017/03/28 10:34:16 [INFO] serf: EventMemberJoin: 10.2.3.0/24,172.28.5.211,808195027ade.dc1-3-santa_clara 10.2.3.1 | |
| 2017/03/28 10:34:16 [INFO] raft: Node at 10.2.3.1:8300 [Follower] entering Follower state (Leader: "") | |
| 2017/03/28 10:34:16 [INFO] consul: Adding LAN server 10.2.3.0/24,172.28.5.211,808195027ade (Addr: tcp/10.2.3.1:8300) (DC: dc1-3-santa_clara) | |
| 2017/03/28 10:34:16 [INFO] consul: Adding WAN server 10.2.3.0/24,172.28.5.211,808195027ade.dc1-3-santa_clara (Addr: tcp/10.2.3.1:8300) (DC: dc1-3-santa_clara) | |
| 2017/03/28 10:34:16 [DEBUG] agent: pausing 16.195040315s before first invocation of /bin/sh -c /usr/bin/agent healthcheck in container agent | |
| 2017/03/28 10:34:16 [DEBUG] Automatically reaping child processes | |
| 2017/03/28 10:34:16 [INFO] agent: Joining cluster... | |
| 2017/03/28 10:34:16 [INFO] agent: (LAN) joining: [10.2.1.1 10.2.2.1 10.2.3.1] | |
| 2017/03/28 10:34:16 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 10:34:16 [INFO] serf: EventMemberJoin: 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae 10.2.1.1 | |
| 2017/03/28 10:34:16 [INFO] serf: EventMemberJoin: 10.2.2.0/24,172.28.5.16,19883068ab3c 10.2.2.1 | |
| 2017/03/28 10:34:16 [INFO] consul: Adding LAN server 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae (Addr: tcp/10.2.1.1:8300) (DC: dc1-3-santa_clara) | |
| 2017/03/28 10:34:16 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:34:16 [DEBUG] memberlist: TCP connection from=172.28.5.211:44046 | |
| 2017/03/28 10:34:16 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.3.1:8301 | |
| 2017/03/28 10:34:16 [INFO] agent: (LAN) joined: 3 Err: <nil> | |
| 2017/03/28 10:34:16 [INFO] agent: Join completed. Synced with 3 initial agents | |
| 2017/03/28 10:34:16 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.2.3.1:8300,10.2.1.1:8300,10.2.2.1:8300 | |
| 2017/03/28 10:34:16 [INFO] consul: Adding LAN server 10.2.2.0/24,172.28.5.16,19883068ab3c (Addr: tcp/10.2.2.1:8300) (DC: dc1-3-santa_clara) | |
| 2017/03/28 10:34:16 [DEBUG] serf: messageJoinType: 10.2.3.0/24,172.28.5.211,808195027ade | |
| 2017/03/28 10:34:16 [DEBUG] serf: messageJoinType: 10.2.3.0/24,172.28.5.211,808195027ade | |
| 2017/03/28 10:34:16 [DEBUG] serf: messageJoinType: 10.2.3.0/24,172.28.5.211,808195027ade | |
| 2017/03/28 10:34:17 [DEBUG] serf: messageJoinType: 10.2.3.0/24,172.28.5.211,808195027ade | |
| 2017/03/28 10:34:21 [DEBUG] raft-net: 10.2.3.1:8300 accepted connection from: 172.28.5.71:40884 | |
| 2017/03/28 10:34:21 [DEBUG] raft-net: 10.2.3.1:8300 accepted connection from: 172.28.5.71:40886 | |
| 2017/03/28 10:34:21 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 10:34:21 [INFO] consul: New leader elected: 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae | |
| 2017/03/28 10:34:21 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 10:34:21 [INFO] agent: Synced service 'consul' | |
| 2017/03/28 10:34:21 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 10:34:21 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:34:21 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:34:21 [INFO] agent: Deregistered service '84380361714f560b1b6ae06729f2714f9b629614c117cb36c71ab5fcf6ab5202' | |
| 2017/03/28 10:34:21 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:34:21 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:34:21 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:34:21 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 10:34:34 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:34:34 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:34:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:38854 | |
| 2017/03/28 10:34:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 10:35:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:38910 | |
| 2017/03/28 10:35:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 10:35:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:38964 | |
| 2017/03/28 10:35:35 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:35:35 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:35:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:36:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 10:36:39 [DEBUG] memberlist: TCP connection from=172.28.5.71:52574 | |
| 2017/03/28 10:36:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:37:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:39126 | |
| 2017/03/28 10:37:09 [DEBUG] memberlist: TCP connection from=172.28.5.71:52628 | |
| 2017/03/28 10:37:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 10:37:29 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:37:29 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:37:39 [DEBUG] memberlist: TCP connection from=172.28.5.71:52682 | |
| 2017/03/28 10:37:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:38:09 [DEBUG] memberlist: TCP connection from=172.28.5.71:52684 | |
| 2017/03/28 10:38:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:38:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:39232 | |
| 2017/03/28 10:38:39 [DEBUG] memberlist: TCP connection from=172.28.5.71:52738 | |
| 2017/03/28 10:38:55 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:38:55 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:38:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 10:39:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:39288 | |
| 2017/03/28 10:39:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:39:37 [DEBUG] serf: messageLeaveType: 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae | |
| 2017/03/28 10:39:37 [DEBUG] serf: messageLeaveType: 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae | |
| 2017/03/28 10:39:37 [DEBUG] serf: messageLeaveType: 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae | |
| 2017/03/28 10:39:37 [DEBUG] serf: messageLeaveType: 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae | |
| 2017/03/28 10:39:37 [DEBUG] serf: messageLeaveType: 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae | |
| 2017/03/28 10:39:37 [INFO] serf: EventMemberLeave: 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae 10.2.1.1 | |
| 2017/03/28 10:39:37 [INFO] consul: Removing LAN server 10.2.1.0/24,172.28.5.71,c2bd8b49b4ae (Addr: tcp/10.2.1.1:8300) (DC: dc1-3-santa_clara) | |
| 2017/03/28 10:39:43 [WARN] raft: Heartbeat timeout from "10.2.1.1:8300" reached, starting election | |
| 2017/03/28 10:39:43 [INFO] raft: Node at 10.2.3.1:8300 [Candidate] entering Candidate state in term 3 | |
| 2017/03/28 10:39:43 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 10:39:43 [DEBUG] raft: Vote granted from 10.2.3.1:8300 in term 3. Tally: 1 | |
| 2017/03/28 10:39:44 [DEBUG] raft-net: 10.2.3.1:8300 accepted connection from: 172.28.5.16:40782 | |
| 2017/03/28 10:39:44 [INFO] raft: Duplicate RequestVote for same term: 3 | |
| 2017/03/28 10:39:48 [WARN] raft: Election timeout reached, restarting election | |
| 2017/03/28 10:39:48 [INFO] raft: Node at 10.2.3.1:8300 [Candidate] entering Candidate state in term 4 | |
| 2017/03/28 10:39:48 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 10:39:48 [DEBUG] raft: Vote granted from 10.2.3.1:8300 in term 4. Tally: 1 | |
| 2017/03/28 10:39:48 [DEBUG] raft: Vote granted from 10.2.2.1:8300 in term 4. Tally: 2 | |
| 2017/03/28 10:39:48 [INFO] raft: Election won. Tally: 2 | |
| 2017/03/28 10:39:48 [INFO] raft: Node at 10.2.3.1:8300 [Leader] entering Leader state | |
| 2017/03/28 10:39:48 [INFO] raft: Added peer 10.2.2.1:8300, starting replication | |
| 2017/03/28 10:39:48 [INFO] consul: cluster leadership acquired | |
| 2017/03/28 10:39:48 [INFO] consul: New leader elected: 10.2.3.0/24,172.28.5.211,808195027ade | |
| 2017/03/28 10:39:48 [INFO] raft: pipelining replication to peer {Voter 10.2.2.1:8300 10.2.2.1:8300} | |
| 2017/03/28 10:39:48 [DEBUG] consul: reset tombstone GC to index 68 | |
| 2017/03/28 10:39:48 [INFO] consul: member '10.2.1.0/24,172.28.5.71,c2bd8b49b4ae' left, deregistering | |
| 2017/03/28 10:39:49 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 10:39:49 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 10:39:49 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 10:39:49 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 10:39:55 [DEBUG] Check failed with exit code: 1 | |
| 2017/03/28 10:39:55 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:39:55 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:39:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:40:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:39406 | |
| 2017/03/28 10:40:08 [DEBUG] consul.state: Session be712a6e-64ee-560e-db65-6b09652e640a TTL expired | |
| 2017/03/28 10:40:25 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:40:25 [INFO] agent: Deregistered service 'a2a4f08f26fefd168acdd9ca74e28b4ca51dbbaa44207a78737428fc24a3c631' | |
| 2017/03/28 10:40:25 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:40:25 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:40:25 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:40:27 [DEBUG] Check failed with exit code: 1 | |
| 2017/03/28 10:40:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:40:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:39486 | |
| 2017/03/28 10:40:53 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:40:53 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:40:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:40:59 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:40:59 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:41:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:39540 | |
| 2017/03/28 10:41:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:41:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:39594 | |
| 2017/03/28 10:41:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:42:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:39648 | |
| 2017/03/28 10:42:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:42:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:39702 | |
| 2017/03/28 10:42:44 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:42:44 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:42:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:43:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:39758 | |
| 2017/03/28 10:43:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:43:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:39812 | |
| 2017/03/28 10:43:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:44:00 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:44:00 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:44:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:39814 | |
| 2017/03/28 10:44:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:44:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:39876 | |
| 2017/03/28 10:44:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:45:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:39932 | |
| 2017/03/28 10:45:19 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:45:19 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:45:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:45:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:39986 | |
| 2017/03/28 10:45:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:46:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:40040 | |
| 2017/03/28 10:46:20 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:46:20 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:46:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:46:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:40094 | |
| 2017/03/28 10:46:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:47:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:40148 | |
| 2017/03/28 10:47:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:47:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:40206 | |
| 2017/03/28 10:47:39 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:47:39 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:47:45 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:47:45 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:47:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:48:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:40264 | |
| 2017/03/28 10:48:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:48:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:40318 | |
| 2017/03/28 10:48:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:49:00 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:49:00 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:49:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:40372 | |
| 2017/03/28 10:49:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:49:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:40436 | |
| 2017/03/28 10:49:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:50:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:40492 | |
| 2017/03/28 10:50:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:50:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:40546 | |
| 2017/03/28 10:50:48 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:50:48 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:50:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:51:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:40600 | |
| 2017/03/28 10:51:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:51:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:40602 | |
| 2017/03/28 10:51:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:52:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:40656 | |
| 2017/03/28 10:52:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:52:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:40710 | |
| 2017/03/28 10:52:48 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:52:48 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:52:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:53:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:40764 | |
| 2017/03/28 10:53:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:53:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:40818 | |
| 2017/03/28 10:53:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:54:02 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:54:02 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:54:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:40872 | |
| 2017/03/28 10:54:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:54:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:40936 | |
| 2017/03/28 10:54:43 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:54:43 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:54:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:55:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:40992 | |
| 2017/03/28 10:55:20 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:55:20 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:55:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:55:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:41046 | |
| 2017/03/28 10:55:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:56:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:41102 | |
| 2017/03/28 10:56:22 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:56:22 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:56:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:56:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:41156 | |
| 2017/03/28 10:56:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:57:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:41212 | |
| 2017/03/28 10:57:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:57:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:41266 | |
| 2017/03/28 10:57:44 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:57:44 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:57:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:58:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:41324 | |
| 2017/03/28 10:58:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:58:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:41378 | |
| 2017/03/28 10:58:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:59:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:41432 | |
| 2017/03/28 10:59:14 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 10:59:14 [DEBUG] agent: Node info in sync | |
| 2017/03/28 10:59:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 10:59:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:41442 | |
| 2017/03/28 10:59:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:00:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:41502 | |
| 2017/03/28 11:00:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:00:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:41556 | |
| 2017/03/28 11:00:54 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:00:54 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:00:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:01:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:41610 | |
| 2017/03/28 11:01:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:01:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:41662 | |
| 2017/03/28 11:01:40 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:01:40 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:01:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:02:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:41716 | |
| 2017/03/28 11:02:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:02:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:41768 | |
| 2017/03/28 11:02:54 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:02:54 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:02:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:03:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:41824 | |
| 2017/03/28 11:03:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:03:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:41878 | |
| 2017/03/28 11:03:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:04:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:41932 | |
| 2017/03/28 11:04:27 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:04:27 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:04:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:04:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:41992 | |
| 2017/03/28 11:04:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:05:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:42050 | |
| 2017/03/28 11:05:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:05:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:42104 | |
| 2017/03/28 11:05:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:06:01 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:06:01 [INFO] agent: Deregistered service '31f504dc2020404b9cc3b84198d22062835ac6970210a8707c129647a6354440' | |
| 2017/03/28 11:06:01 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:06:01 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:06:01 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:06:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:42162 | |
| 2017/03/28 11:06:06 [DEBUG] Check failed with exit code: 1 | |
| 2017/03/28 11:06:06 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:06:06 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:06:06 [DEBUG] memberlist: TCP connection from=172.28.5.71:52912 | |
| 2017/03/28 11:06:06 [INFO] serf: EventMemberJoin: 10.2.1.0/24,172.28.5.71,3fc622a3e09b 10.2.1.1 | |
| 2017/03/28 11:06:06 [INFO] consul: Adding LAN server 10.2.1.0/24,172.28.5.71,3fc622a3e09b (Addr: tcp/10.2.1.1:8300) (DC: dc1-3-santa_clara) | |
| 2017/03/28 11:06:06 [INFO] raft: Updating configuration with AddStaging (10.2.1.1:8300, 10.2.1.1:8300) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300} {Suffrage:Voter ID:10.2.1.1:8300 Address:10.2.1.1:8300}] | |
| 2017/03/28 11:06:06 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:06:06 [WARN] raft: AppendEntries to {Voter 10.2.1.1:8300 10.2.1.1:8300} rejected, sending older logs (next: 1) | |
| 2017/03/28 11:06:06 [INFO] consul: member '10.2.1.0/24,172.28.5.71,3fc622a3e09b' joined, marking health alive | |
| 2017/03/28 11:06:07 [INFO] raft: pipelining replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:06:07 [DEBUG] serf: messageJoinType: 10.2.1.0/24,172.28.5.71,3fc622a3e09b | |
| 2017/03/28 11:06:07 [DEBUG] serf: messageJoinType: 10.2.1.0/24,172.28.5.71,3fc622a3e09b | |
| 2017/03/28 11:06:07 [DEBUG] serf: messageJoinType: 10.2.1.0/24,172.28.5.71,3fc622a3e09b | |
| 2017/03/28 11:06:07 [DEBUG] serf: messageJoinType: 10.2.1.0/24,172.28.5.71,3fc622a3e09b | |
| 2017/03/28 11:06:08 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:06:08 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:06:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:06:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:42226 | |
| 2017/03/28 11:06:38 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:06:38 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:06:40 [DEBUG] memberlist: TCP connection from=172.28.5.71:53032 | |
| 2017/03/28 11:06:48 [INFO] raft: Updating configuration with RemoveServer (10.2.1.1:8300, ) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300}] | |
| 2017/03/28 11:06:48 [INFO] raft: Removed peer 10.2.1.1:8300, stopping replication after 260 | |
| 2017/03/28 11:06:48 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:06:48 [INFO] raft: Updating configuration with AddStaging (10.2.1.1:8300, 10.2.1.1:8300) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300} {Suffrage:Voter ID:10.2.1.1:8300 Address:10.2.1.1:8300}] | |
| 2017/03/28 11:06:48 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:06:48 [WARN] raft: AppendEntries to {Voter 10.2.1.1:8300 10.2.1.1:8300} rejected, sending older logs (next: 261) | |
| 2017/03/28 11:06:49 [INFO] raft: pipelining replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:06:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:07:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:42280 | |
| 2017/03/28 11:07:13 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:07:13 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:07:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:07:48 [INFO] raft: Updating configuration with RemoveServer (10.2.1.1:8300, ) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300}] | |
| 2017/03/28 11:07:48 [INFO] raft: Removed peer 10.2.1.1:8300, stopping replication after 268 | |
| 2017/03/28 11:07:48 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:07:48 [INFO] raft: Updating configuration with AddStaging (10.2.1.1:8300, 10.2.1.1:8300) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300} {Suffrage:Voter ID:10.2.1.1:8300 Address:10.2.1.1:8300}] | |
| 2017/03/28 11:07:48 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:07:48 [WARN] raft: AppendEntries to {Voter 10.2.1.1:8300 10.2.1.1:8300} rejected, sending older logs (next: 269) | |
| 2017/03/28 11:07:49 [INFO] raft: pipelining replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:07:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:08:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:42336 | |
| 2017/03/28 11:08:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:08:48 [INFO] raft: Updating configuration with RemoveServer (10.2.1.1:8300, ) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300}] | |
| 2017/03/28 11:08:48 [INFO] raft: Removed peer 10.2.1.1:8300, stopping replication after 277 | |
| 2017/03/28 11:08:48 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:08:48 [INFO] raft: Updating configuration with AddStaging (10.2.1.1:8300, 10.2.1.1:8300) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300} {Suffrage:Voter ID:10.2.1.1:8300 Address:10.2.1.1:8300}] | |
| 2017/03/28 11:08:48 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:08:48 [WARN] raft: AppendEntries to {Voter 10.2.1.1:8300 10.2.1.1:8300} rejected, sending older logs (next: 278) | |
| 2017/03/28 11:08:49 [INFO] raft: pipelining replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:08:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:09:10 [DEBUG] memberlist: TCP connection from=172.28.5.71:53256 | |
| 2017/03/28 11:09:13 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:09:13 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:09:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:09:48 [INFO] raft: Updating configuration with RemoveServer (10.2.1.1:8300, ) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300}] | |
| 2017/03/28 11:09:48 [INFO] raft: Removed peer 10.2.1.1:8300, stopping replication after 287 | |
| 2017/03/28 11:09:48 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:09:48 [INFO] raft: Updating configuration with AddStaging (10.2.1.1:8300, 10.2.1.1:8300) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300} {Suffrage:Voter ID:10.2.1.1:8300 Address:10.2.1.1:8300}] | |
| 2017/03/28 11:09:48 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:09:49 [WARN] raft: AppendEntries to {Voter 10.2.1.1:8300 10.2.1.1:8300} rejected, sending older logs (next: 288) | |
| 2017/03/28 11:09:49 [INFO] raft: pipelining replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:09:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:10:21 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:10:21 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:10:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:10:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:42618 | |
| 2017/03/28 11:10:40 [DEBUG] memberlist: TCP connection from=172.28.5.71:53424 | |
| 2017/03/28 11:10:48 [INFO] raft: Updating configuration with RemoveServer (10.2.1.1:8300, ) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300}] | |
| 2017/03/28 11:10:48 [INFO] raft: Removed peer 10.2.1.1:8300, stopping replication after 297 | |
| 2017/03/28 11:10:48 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:10:48 [INFO] raft: Updating configuration with AddStaging (10.2.1.1:8300, 10.2.1.1:8300) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300} {Suffrage:Voter ID:10.2.1.1:8300 Address:10.2.1.1:8300}] | |
| 2017/03/28 11:10:48 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:10:49 [WARN] raft: AppendEntries to {Voter 10.2.1.1:8300 10.2.1.1:8300} rejected, sending older logs (next: 298) | |
| 2017/03/28 11:10:49 [INFO] raft: pipelining replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:10:53 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:10:53 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:10:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:11:10 [DEBUG] memberlist: TCP connection from=172.28.5.71:53480 | |
| 2017/03/28 11:11:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:11:48 [INFO] raft: Updating configuration with RemoveServer (10.2.1.1:8300, ) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300}] | |
| 2017/03/28 11:11:48 [INFO] raft: Removed peer 10.2.1.1:8300, stopping replication after 308 | |
| 2017/03/28 11:11:48 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:11:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:11:59 [DEBUG] raft-net: 10.2.3.1:8300 accepted connection from: 172.28.5.71:42194 | |
| 2017/03/28 11:11:59 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:12:03 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:12:03 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:12:06 [INFO] agent.rpc: Accepted client: 127.0.0.1:53792 | |
| 2017/03/28 11:12:07 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:12:16 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:12:26 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:12:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:12:34 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:12:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:42828 | |
| 2017/03/28 11:12:40 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:12:40 [DEBUG] memberlist: TCP connection from=172.28.5.71:53664 | |
| 2017/03/28 11:12:47 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:12:48 [INFO] raft: Updating configuration with AddStaging (10.2.1.1:8300, 10.2.1.1:8300) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300} {Suffrage:Voter ID:10.2.1.1:8300 Address:10.2.1.1:8300}] | |
| 2017/03/28 11:12:48 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:12:48 [ERR] raft: peer {Voter 10.2.1.1:8300 10.2.1.1:8300} has newer term, stopping replication | |
| 2017/03/28 11:12:48 [INFO] raft: Node at 10.2.3.1:8300 [Follower] entering Follower state (Leader: "") | |
| 2017/03/28 11:12:48 [INFO] consul: cluster leadership lost | |
| 2017/03/28 11:12:48 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.2.1:8300 10.2.2.1:8300} | |
| 2017/03/28 11:12:48 [ERR] consul: failed to add raft peer: leadership lost while committing log | |
| 2017/03/28 11:12:48 [ERR] consul: failed to reconcile member: {10.2.1.0/24,172.28.5.71,3fc622a3e09b 10.2.1.1 8301 map[port:8300 expect:3 role:consul dc:dc1-3-santa_clara vsn:2 vsn_min:2 vsn_max:3 build:0.7.0:'a189091] alive 1 4 2 2 4 4}: leadership lost while committing log | |
| 2017/03/28 11:12:48 [ERR] consul: failed to reconcile: leadership lost while committing log | |
| 2017/03/28 11:12:52 [WARN] consul.coordinate: Batch update failed: node is not the leader | |
| 2017/03/28 11:12:54 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since our last index is greater (315, 308) | |
| 2017/03/28 11:12:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:13:00 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:13:00 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:13:00 [INFO] consul: New leader elected: 10.2.2.0/24,172.28.5.16,19883068ab3c | |
| 2017/03/28 11:13:00 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:13:00 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:13:00 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:13:01 [DEBUG] raft-net: 10.2.3.1:8300 accepted connection from: 172.28.5.16:44268 | |
| 2017/03/28 11:13:08 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.2.1:8300 | |
| 2017/03/28 11:13:10 [DEBUG] memberlist: TCP connection from=172.28.5.71:53722 | |
| 2017/03/28 11:13:14 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.2.1:8300 | |
| 2017/03/28 11:13:18 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:13:18 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:13:20 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.2.1:8300 | |
| 2017/03/28 11:13:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:13:28 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.2.1:8300 | |
| 2017/03/28 11:13:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:42940 | |
| 2017/03/28 11:13:37 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.2.1:8300 | |
| 2017/03/28 11:13:43 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.2.1:8300 | |
| 2017/03/28 11:13:51 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.2.1:8300 | |
| 2017/03/28 11:13:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:13:57 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.2.1:8300 | |
| 2017/03/28 11:14:04 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.2.1:8300 | |
| 2017/03/28 11:14:05 [WARN] raft: Heartbeat timeout from "10.2.2.1:8300" reached, starting election | |
| 2017/03/28 11:14:05 [INFO] raft: Node at 10.2.3.1:8300 [Candidate] entering Candidate state in term 14 | |
| 2017/03/28 11:14:05 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:14:05 [DEBUG] raft: Newer term discovered, fallback to follower | |
| 2017/03/28 11:14:05 [INFO] raft: Node at 10.2.3.1:8300 [Follower] entering Follower state (Leader: "") | |
| 2017/03/28 11:14:07 [DEBUG] raft-net: 10.2.3.1:8300 accepted connection from: 172.28.5.16:44378 | |
| 2017/03/28 11:14:07 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:14:07 [INFO] consul: New leader elected: 10.2.2.0/24,172.28.5.16,19883068ab3c | |
| 2017/03/28 11:14:07 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:14:07 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:14:07 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:14:07 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:14:07 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:14:10 [DEBUG] memberlist: TCP connection from=172.28.5.71:53780 | |
| 2017/03/28 11:14:11 [DEBUG] memberlist: Potential blocking operation. Last command took 10.011445ms | |
| 2017/03/28 11:14:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:14:29 [INFO] agent.rpc: Accepted client: 127.0.0.1:54018 | |
| 2017/03/28 11:14:40 [DEBUG] memberlist: TCP connection from=172.28.5.71:53836 | |
| 2017/03/28 11:14:41 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:14:41 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:14:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:15:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:43066 | |
| 2017/03/28 11:15:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:15:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:43122 | |
| 2017/03/28 11:15:46 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:15:46 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:15:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:16:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:43176 | |
| 2017/03/28 11:16:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:16:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:43232 | |
| 2017/03/28 11:16:40 [DEBUG] memberlist: TCP connection from=172.28.5.71:54070 | |
| 2017/03/28 11:16:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:17:02 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:17:02 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:17:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:43284 | |
| 2017/03/28 11:17:10 [DEBUG] memberlist: TCP connection from=172.28.5.71:54126 | |
| 2017/03/28 11:17:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:17:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:18:10 [DEBUG] memberlist: TCP connection from=172.28.5.71:54238 | |
| 2017/03/28 11:18:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:18:40 [DEBUG] memberlist: TCP connection from=172.28.5.71:54294 | |
| 2017/03/28 11:18:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:18:59 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:18:59 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:19:10 [DEBUG] memberlist: TCP connection from=172.28.5.71:54350 | |
| 2017/03/28 11:19:15 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.2.1:8300 | |
| 2017/03/28 11:19:20 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.2.1:8300 | |
| 2017/03/28 11:19:27 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.2.1:8300 | |
| 2017/03/28 11:19:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:19:32 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.2.1:8300 | |
| 2017/03/28 11:19:40 [DEBUG] memberlist: TCP connection from=172.28.5.71:54406 | |
| 2017/03/28 11:19:40 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.2.1:8300 | |
| 2017/03/28 11:19:49 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.2.1:8300 | |
| 2017/03/28 11:19:57 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.2.1:8300 | |
| 2017/03/28 11:19:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:20:05 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.2.1:8300 | |
| 2017/03/28 11:20:10 [DEBUG] memberlist: TCP connection from=172.28.5.71:54458 | |
| 2017/03/28 11:20:14 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since our last index is greater (388, 381) | |
| 2017/03/28 11:20:14 [ERR] agent: coordinate update error: rpc error: No cluster leader | |
| 2017/03/28 11:20:21 [WARN] raft: Heartbeat timeout from "" reached, starting election | |
| 2017/03/28 11:20:21 [INFO] raft: Node at 10.2.3.1:8300 [Candidate] entering Candidate state in term 33 | |
| 2017/03/28 11:20:21 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:20:21 [DEBUG] raft: Vote granted from 10.2.3.1:8300 in term 33. Tally: 1 | |
| 2017/03/28 11:20:21 [DEBUG] raft: Vote granted from 10.2.2.1:8300 in term 33. Tally: 2 | |
| 2017/03/28 11:20:21 [INFO] raft: Election won. Tally: 2 | |
| 2017/03/28 11:20:21 [INFO] raft: Node at 10.2.3.1:8300 [Leader] entering Leader state | |
| 2017/03/28 11:20:21 [INFO] raft: Added peer 10.2.2.1:8300, starting replication | |
| 2017/03/28 11:20:21 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:20:21 [INFO] consul: cluster leadership acquired | |
| 2017/03/28 11:20:21 [INFO] consul: New leader elected: 10.2.3.0/24,172.28.5.211,808195027ade | |
| 2017/03/28 11:20:21 [INFO] raft: pipelining replication to peer {Voter 10.2.2.1:8300 10.2.2.1:8300} | |
| 2017/03/28 11:20:21 [WARN] raft: AppendEntries to {Voter 10.2.1.1:8300 10.2.1.1:8300} rejected, sending older logs (next: 382) | |
| 2017/03/28 11:20:21 [DEBUG] consul: reset tombstone GC to index 390 | |
| 2017/03/28 11:20:21 [INFO] raft: Updating configuration with RemoveServer (10.2.1.1:8300, ) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300}] | |
| 2017/03/28 11:20:21 [INFO] raft: pipelining replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:20:21 [INFO] raft: Removed peer 10.2.1.1:8300, stopping replication after 391 | |
| 2017/03/28 11:20:21 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:20:21 [INFO] raft: Updating configuration with AddStaging (10.2.1.1:8300, 10.2.1.1:8300) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300} {Suffrage:Voter ID:10.2.1.1:8300 Address:10.2.1.1:8300}] | |
| 2017/03/28 11:20:21 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:20:21 [WARN] raft: AppendEntries to {Voter 10.2.1.1:8300 10.2.1.1:8300} rejected, sending older logs (next: 392) | |
| 2017/03/28 11:20:21 [INFO] raft: pipelining replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:20:21 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:20:21 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:20:21 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:20:21 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:20:21 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:20:21 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:20:21 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:20:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:20:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:43684 | |
| 2017/03/28 11:20:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:21:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:43738 | |
| 2017/03/28 11:21:10 [DEBUG] memberlist: TCP connection from=172.28.5.71:54522 | |
| 2017/03/28 11:21:21 [INFO] raft: Updating configuration with RemoveServer (10.2.1.1:8300, ) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300}] | |
| 2017/03/28 11:21:21 [INFO] raft: Removed peer 10.2.1.1:8300, stopping replication after 402 | |
| 2017/03/28 11:21:21 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:21:27 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:21:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:21:30 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:21:30 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:21:36 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:21:40 [DEBUG] memberlist: TCP connection from=172.28.5.71:54588 | |
| 2017/03/28 11:21:42 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:21:49 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:21:55 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:21:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:22:05 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:22:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:43794 | |
| 2017/03/28 11:22:12 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:22:12 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:22:14 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:22:21 [INFO] raft: Updating configuration with AddStaging (10.2.1.1:8300, 10.2.1.1:8300) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300} {Suffrage:Voter ID:10.2.1.1:8300 Address:10.2.1.1:8300}] | |
| 2017/03/28 11:22:21 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:22:21 [ERR] raft: peer {Voter 10.2.1.1:8300 10.2.1.1:8300} has newer term, stopping replication | |
| 2017/03/28 11:22:21 [INFO] raft: Node at 10.2.3.1:8300 [Follower] entering Follower state (Leader: "") 2017/03/28 11:22:21 [ERR] consul: failed to add raft peer: leadership lost while committing log | |
| 2017/03/28 11:22:21 [ERR] consul: failed to reconcile member: {10.2.1.0/24,172.28.5.71,3fc622a3e09b 10.2.1.1 8301 map[vsn:2 vsn_min:2 vsn_max:3 build:0.7.0:'a189091 port:8300 expect:3 role:consul dc:dc1-3-santa_clara] alive 1 4 2 2 4 4}: leadership lost while committing log | |
| 2017/03/28 11:22:21 [ERR] consul: failed to reconcile: leadership lost while committing log | |
| 2017/03/28 11:22:21 [INFO] consul: cluster leadership lost | |
| 2017/03/28 11:22:21 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.2.1:8300 10.2.2.1:8300} | |
| 2017/03/28 11:22:22 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since our last index is greater (409, 402) | |
| 2017/03/28 11:22:27 [WARN] raft: Heartbeat timeout from "" reached, starting election | |
| 2017/03/28 11:22:27 [INFO] raft: Node at 10.2.3.1:8300 [Candidate] entering Candidate state in term 42 | |
| 2017/03/28 11:22:27 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:22:27 [DEBUG] raft: Vote granted from 10.2.2.1:8300 in term 42. Tally: 1 | |
| 2017/03/28 11:22:27 [DEBUG] raft: Vote granted from 10.2.3.1:8300 in term 42. Tally: 2 | |
| 2017/03/28 11:22:27 [INFO] raft: Election won. Tally: 2 | |
| 2017/03/28 11:22:27 [INFO] raft: Node at 10.2.3.1:8300 [Leader] entering Leader state | |
| 2017/03/28 11:22:27 [INFO] raft: Added peer 10.2.2.1:8300, starting replication | |
| 2017/03/28 11:22:27 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:22:27 [INFO] consul: cluster leadership acquired | |
| 2017/03/28 11:22:27 [INFO] consul: New leader elected: 10.2.3.0/24,172.28.5.211,808195027ade | |
| 2017/03/28 11:22:27 [INFO] raft: pipelining replication to peer {Voter 10.2.2.1:8300 10.2.2.1:8300} | |
| 2017/03/28 11:22:27 [DEBUG] consul: reset tombstone GC to index 411 | |
| 2017/03/28 11:22:27 [INFO] raft: Updating configuration with RemoveServer (10.2.1.1:8300, ) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300}] | |
| 2017/03/28 11:22:27 [INFO] raft: Removed peer 10.2.1.1:8300, stopping replication after 412 | |
| 2017/03/28 11:22:27 [WARN] raft: AppendEntries to {Voter 10.2.1.1:8300 10.2.1.1:8300} rejected, sending older logs (next: 403) | |
| 2017/03/28 11:22:27 [INFO] raft: Updating configuration with AddStaging (10.2.1.1:8300, 10.2.1.1:8300) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300} {Suffrage:Voter ID:10.2.1.1:8300 Address:10.2.1.1:8300}] | |
| 2017/03/28 11:22:27 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:22:27 [WARN] raft: AppendEntries to {Voter 10.2.1.1:8300 10.2.1.1:8300} rejected, sending older logs (next: 403) | |
| 2017/03/28 11:22:27 [INFO] raft: pipelining replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:22:27 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:22:27 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:22:27 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:22:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:22:27 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:22:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:43848 | |
| 2017/03/28 11:22:40 [DEBUG] memberlist: TCP connection from=172.28.5.71:54698 | |
| 2017/03/28 11:22:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:23:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:43902 | |
| 2017/03/28 11:23:10 [DEBUG] memberlist: TCP connection from=172.28.5.71:54754 | |
| 2017/03/28 11:23:27 [INFO] raft: Updating configuration with RemoveServer (10.2.1.1:8300, ) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300}] | |
| 2017/03/28 11:23:27 [INFO] raft: Removed peer 10.2.1.1:8300, stopping replication after 422 | |
| 2017/03/28 11:23:27 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:23:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:23:31 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:23:31 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:23:32 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:23:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:43956 | |
| 2017/03/28 11:23:40 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:23:49 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:23:56 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:23:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:24:02 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:24:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:44010 | |
| 2017/03/28 11:24:08 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:24:10 [DEBUG] memberlist: TCP connection from=172.28.5.71:54862 | |
| 2017/03/28 11:24:15 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:24:21 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:24:26 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:24:27 [INFO] raft: Updating configuration with AddStaging (10.2.1.1:8300, 10.2.1.1:8300) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300} {Suffrage:Voter ID:10.2.1.1:8300 Address:10.2.1.1:8300}] | |
| 2017/03/28 11:24:27 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:24:27 [ERR] raft: peer {Voter 10.2.1.1:8300 10.2.1.1:8300} has newer term, stopping replication | |
| 2017/03/28 11:24:27 [INFO] raft: Node at 10.2.3.1:8300 [Follower] entering Follower state (Leader: "") | |
| 2017/03/28 11:24:27 [INFO] consul: cluster leadership lost | |
| 2017/03/28 11:24:27 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.2.1:8300 10.2.2.1:8300} | |
| 2017/03/28 11:24:27 [ERR] consul: failed to add raft peer: leadership lost while committing log | |
| 2017/03/28 11:24:27 [ERR] consul: failed to reconcile member: {10.2.1.0/24,172.28.5.71,3fc622a3e09b 10.2.1.1 8301 map[build:0.7.0:'a189091 port:8300 expect:3 role:consul dc:dc1-3-santa_clara vsn:2 vsn_min:2 vsn_max:3] alive 1 4 2 2 4 4}: leadership lost while committing log | |
| 2017/03/28 11:24:27 [ERR] consul: failed to reconcile: leadership lost while committing log | |
| 2017/03/28 11:24:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:24:32 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since our last index is greater (430, 422) | |
| 2017/03/28 11:24:35 [WARN] raft: Heartbeat timeout from "" reached, starting election | |
| 2017/03/28 11:24:35 [INFO] raft: Node at 10.2.3.1:8300 [Candidate] entering Candidate state in term 53 | |
| 2017/03/28 11:24:35 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:24:35 [DEBUG] raft: Vote granted from 10.2.3.1:8300 in term 53. Tally: 1 | |
| 2017/03/28 11:24:35 [DEBUG] raft: Vote granted from 10.2.2.1:8300 in term 53. Tally: 2 | |
| 2017/03/28 11:24:35 [INFO] raft: Election won. Tally: 2 | |
| 2017/03/28 11:24:35 [INFO] raft: Node at 10.2.3.1:8300 [Leader] entering Leader state | |
| 2017/03/28 11:24:35 [INFO] raft: Added peer 10.2.2.1:8300, starting replication | |
| 2017/03/28 11:24:35 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:24:35 [INFO] consul: cluster leadership acquired | |
| 2017/03/28 11:24:35 [INFO] raft: pipelining replication to peer {Voter 10.2.2.1:8300 10.2.2.1:8300} | |
| 2017/03/28 11:24:35 [INFO] consul: New leader elected: 10.2.3.0/24,172.28.5.211,808195027ade | |
| 2017/03/28 11:24:35 [WARN] raft: AppendEntries to {Voter 10.2.1.1:8300 10.2.1.1:8300} rejected, sending older logs (next: 423) | |
| 2017/03/28 11:24:35 [DEBUG] consul: reset tombstone GC to index 432 | |
| 2017/03/28 11:24:35 [INFO] raft: Updating configuration with RemoveServer (10.2.1.1:8300, ) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300}] | |
| 2017/03/28 11:24:35 [INFO] raft: Removed peer 10.2.1.1:8300, stopping replication after 433 | |
| 2017/03/28 11:24:35 [INFO] raft: Updating configuration with AddStaging (10.2.1.1:8300, 10.2.1.1:8300) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300} {Suffrage:Voter ID:10.2.1.1:8300 Address:10.2.1.1:8300}] | |
| 2017/03/28 11:24:35 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:24:35 [WARN] raft: AppendEntries to {Voter 10.2.1.1:8300 10.2.1.1:8300} rejected, sending older logs (next: 431) | |
| 2017/03/28 11:24:35 [INFO] raft: pipelining replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:24:35 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:24:35 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:24:36 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:24:36 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:24:36 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:24:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:24:58 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:24:58 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:25:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:25:35 [INFO] raft: Updating configuration with RemoveServer (10.2.1.1:8300, ) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300}] | |
| 2017/03/28 11:25:35 [INFO] raft: Removed peer 10.2.1.1:8300, stopping replication after 444 | |
| 2017/03/28 11:25:35 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:25:35 [INFO] raft: Updating configuration with AddStaging (10.2.1.1:8300, 10.2.1.1:8300) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300} {Suffrage:Voter ID:10.2.1.1:8300 Address:10.2.1.1:8300}] | |
| 2017/03/28 11:25:35 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:25:35 [WARN] raft: AppendEntries to {Voter 10.2.1.1:8300 10.2.1.1:8300} rejected, sending older logs (next: 445) | |
| 2017/03/28 11:25:35 [INFO] raft: pipelining replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:25:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:26:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:44236 | |
| 2017/03/28 11:26:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:26:33 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:26:33 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:26:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:44290 | |
| 2017/03/28 11:26:35 [INFO] raft: Updating configuration with RemoveServer (10.2.1.1:8300, ) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300}] | |
| 2017/03/28 11:26:35 [INFO] raft: Removed peer 10.2.1.1:8300, stopping replication after 454 | |
| 2017/03/28 11:26:35 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:26:35 [INFO] raft: Updating configuration with AddStaging (10.2.1.1:8300, 10.2.1.1:8300) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300} {Suffrage:Voter ID:10.2.1.1:8300 Address:10.2.1.1:8300}] | |
| 2017/03/28 11:26:35 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:26:35 [WARN] raft: AppendEntries to {Voter 10.2.1.1:8300 10.2.1.1:8300} rejected, sending older logs (next: 455) | |
| 2017/03/28 11:26:35 [INFO] raft: pipelining replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:26:40 [DEBUG] memberlist: TCP connection from=172.28.5.71:55102 | |
| 2017/03/28 11:26:48 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:26:48 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:26:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:27:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:27:35 [INFO] raft: Updating configuration with RemoveServer (10.2.1.1:8300, ) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300}] | |
| 2017/03/28 11:27:35 [INFO] raft: Removed peer 10.2.1.1:8300, stopping replication after 465 | |
| 2017/03/28 11:27:35 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:27:41 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:27:46 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:27:55 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:27:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 | |
| 2017/03/28 11:28:00 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:28:05 [DEBUG] memberlist: TCP connection from=172.28.5.16:44456 | |
| 2017/03/28 11:28:07 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:28:15 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:28:15 [DEBUG] agent: Service 'consul' in sync | |
| 2017/03/28 11:28:15 [DEBUG] agent: Node info in sync | |
| 2017/03/28 11:28:22 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:28:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:28:31 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since we have a leader: 10.2.3.1:8300 | |
| 2017/03/28 11:28:35 [DEBUG] memberlist: TCP connection from=172.28.5.16:44510 | |
| 2017/03/28 11:28:35 [INFO] raft: Updating configuration with AddStaging (10.2.1.1:8300, 10.2.1.1:8300) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300} {Suffrage:Voter ID:10.2.1.1:8300 Address:10.2.1.1:8300}] | |
| 2017/03/28 11:28:35 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:28:35 [ERR] raft: peer {Voter 10.2.1.1:8300 10.2.1.1:8300} has newer term, stopping replication | |
| 2017/03/28 11:28:35 [INFO] raft: Node at 10.2.3.1:8300 [Follower] entering Follower state (Leader: "") 2017/03/28 11:28:35 [ERR] consul: failed to add raft peer: leadership lost while committing log | |
| 2017/03/28 11:28:35 [ERR] consul: failed to reconcile member: {10.2.1.0/24,172.28.5.71,3fc622a3e09b 10.2.1.1 8301 map[vsn:2 vsn_min:2 vsn_max:3 build:0.7.0:'a189091 port:8300 expect:3 role:consul dc:dc1-3-santa_clara] alive 1 4 2 2 4 4}: leadership lost while committing log | |
| 2017/03/28 11:28:35 [ERR] consul: failed to reconcile: leadership lost while committing log | |
| 2017/03/28 11:28:35 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.2.1:8300 10.2.2.1:8300} | |
| 2017/03/28 11:28:35 [INFO] consul: cluster leadership lost | |
| 2017/03/28 11:28:37 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since our last index is greater (472, 465) | |
| 2017/03/28 11:28:43 [WARN] raft: Rejecting vote request from 10.2.1.1:8300 since our last index is greater (472, 465) | |
| 2017/03/28 11:28:44 [WARN] raft: Heartbeat timeout from "" reached, starting election | |
| 2017/03/28 11:28:44 [INFO] raft: Node at 10.2.3.1:8300 [Candidate] entering Candidate state in term 64 | |
| 2017/03/28 11:28:44 [DEBUG] raft: Votes needed: 2 | |
| 2017/03/28 11:28:44 [DEBUG] raft: Vote granted from 10.2.3.1:8300 in term 64. Tally: 1 | |
| 2017/03/28 11:28:44 [DEBUG] raft: Vote granted from 10.2.2.1:8300 in term 64. Tally: 2 | |
| 2017/03/28 11:28:44 [INFO] raft: Election won. Tally: 2 | |
| 2017/03/28 11:28:44 [INFO] raft: Node at 10.2.3.1:8300 [Leader] entering Leader state | |
| 2017/03/28 11:28:44 [INFO] raft: Added peer 10.2.2.1:8300, starting replication | |
| 2017/03/28 11:28:44 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:28:44 [INFO] consul: cluster leadership acquired | |
| 2017/03/28 11:28:44 [INFO] raft: pipelining replication to peer {Voter 10.2.2.1:8300 10.2.2.1:8300} | |
| 2017/03/28 11:28:44 [INFO] consul: New leader elected: 10.2.3.0/24,172.28.5.211,808195027ade | |
| 2017/03/28 11:28:44 [WARN] raft: AppendEntries to {Voter 10.2.1.1:8300 10.2.1.1:8300} rejected, sending older logs (next: 466) | |
| 2017/03/28 11:28:44 [DEBUG] consul: reset tombstone GC to index 474 | |
| 2017/03/28 11:28:44 [INFO] raft: Updating configuration with RemoveServer (10.2.1.1:8300, ) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300}] | |
| 2017/03/28 11:28:44 [INFO] raft: pipelining replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:28:44 [INFO] raft: Removed peer 10.2.1.1:8300, stopping replication after 475 | |
| 2017/03/28 11:28:44 [INFO] raft: aborting pipeline replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:28:44 [INFO] raft: Updating configuration with AddStaging (10.2.1.1:8300, 10.2.1.1:8300) to [{Suffrage:Voter ID:10.2.2.1:8300 Address:10.2.2.1:8300} {Suffrage:Voter ID:10.2.3.1:8300 Address:10.2.3.1:8300} {Suffrage:Voter ID:10.2.1.1:8300 Address:10.2.1.1:8300}] | |
| 2017/03/28 11:28:44 [INFO] raft: Added peer 10.2.1.1:8300, starting replication | |
| 2017/03/28 11:28:44 [WARN] raft: AppendEntries to {Voter 10.2.1.1:8300 10.2.1.1:8300} rejected, sending older logs (next: 476) | |
| 2017/03/28 11:28:44 [INFO] raft: pipelining replication to peer {Voter 10.2.1.1:8300 10.2.1.1:8300} | |
| 2017/03/28 11:28:44 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:28:44 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:28:44 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:28:44 [DEBUG] serf: messageUserEventType: consul:new-leader | |
| 2017/03/28 11:28:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.2.1:8301 | |
| 2017/03/28 11:29:10 [DEBUG] memberlist: TCP connection from=172.28.5.71:55380 | |
| 2017/03/28 11:29:27 [DEBUG] memberlist: Initiating push/pull sync with: 10.2.1.1:8301 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment