Skip to content

Instantly share code, notes, and snippets.

@muradm

muradm/consul_failure.log

Created September 30, 2016 03:32
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save muradm/4b40611b64102a1dc0051d40b4e45d01 to your computer and use it in GitHub Desktop.
Save muradm/4b40611b64102a1dc0051d40b4e45d01 to your computer and use it in GitHub Desktop.
Download ZIP
Consul failure in swarm mode
Raw
consul_failure.log
LAB1:
2016-09-30T07:23:07.179341+04:00 lab1 consul[36768]: ==> Found address '10.100.0.2' for interface 'eth2', setting bind option...
2016-09-30T07:23:07.187363+04:00 lab1 consul[36768]: ==> Starting Consul agent...
2016-09-30T07:23:07.409924+04:00 lab1 consul[36768]: ==> Starting Consul agent RPC...
2016-09-30T07:23:07.410140+04:00 lab1 consul[36768]: ==> Consul agent running!
2016-09-30T07:23:07.410273+04:00 lab1 consul[36768]: Version: 'v0.7.0'
2016-09-30T07:23:07.410373+04:00 lab1 consul[36768]: Node name: '1ec48b592c2f'
2016-09-30T07:23:07.410445+04:00 lab1 consul[36768]: Datacenter: 'lab-dc'
2016-09-30T07:23:07.410512+04:00 lab1 consul[36768]: Server: true (bootstrap: false)
2016-09-30T07:23:07.410578+04:00 lab1 consul[36768]: Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400)
2016-09-30T07:23:07.410640+04:00 lab1 consul[36768]: Cluster Addr: 10.100.0.2 (LAN: 8301, WAN: 8302)
2016-09-30T07:23:07.410702+04:00 lab1 consul[36768]: Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
2016-09-30T07:23:07.410768+04:00 lab1 consul[36768]: Atlas: <disabled>
2016-09-30T07:23:07.410832+04:00 lab1 consul[36768]:
2016-09-30T07:23:07.410894+04:00 lab1 consul[36768]: ==> Log data will now stream in as it occurs:
2016-09-30T07:23:07.410956+04:00 lab1 consul[36768]:
2016-09-30T07:23:07.411017+04:00 lab1 consul[36768]: 2016/09/30 03:23:07 [INFO] raft: Initial configuration (index=0): []
2016-09-30T07:23:07.411085+04:00 lab1 consul[36768]: 2016/09/30 03:23:07 [INFO] serf: EventMemberJoin: 1ec48b592c2f 10.100.0.2
2016-09-30T07:23:07.411143+04:00 lab1 consul[36768]: 2016/09/30 03:23:07 [INFO] raft: Node at 10.100.0.2:8300 [Follower] entering Follower state (Leader: "")
2016-09-30T07:23:07.411203+04:00 lab1 consul[36768]: 2016/09/30 03:23:07 [INFO] serf: EventMemberJoin: 1ec48b592c2f.lab-dc 10.100.0.2
2016-09-30T07:23:07.411261+04:00 lab1 consul[36768]: 2016/09/30 03:23:07 [INFO] consul: Adding LAN server 1ec48b592c2f (Addr: tcp/10.100.0.2:8300) (DC: lab-dc)
2016-09-30T07:23:07.411319+04:00 lab1 consul[36768]: 2016/09/30 03:23:07 [INFO] consul: Adding WAN server 1ec48b592c2f.lab-dc (Addr: tcp/10.100.0.2:8300) (DC: lab-dc)
2016-09-30T07:23:07.411377+04:00 lab1 consul[36768]: 2016/09/30 03:23:07 [INFO] agent: Joining cluster...
2016-09-30T07:23:07.411435+04:00 lab1 consul[36768]: 2016/09/30 03:23:07 [INFO] agent: (LAN) joining: [tasks.consul]
2016-09-30T07:23:07.576726+04:00 lab1 consul[36768]: 2016/09/30 03:23:07 [INFO] serf: EventMemberJoin: 5559b12d7b78 10.100.0.5
2016-09-30T07:23:07.577130+04:00 lab1 consul[36768]: 2016/09/30 03:23:07 [INFO] serf: EventMemberJoin: 667edff3624f 10.100.0.3
2016-09-30T07:23:07.577275+04:00 lab1 consul[36768]: 2016/09/30 03:23:07 [INFO] consul: Adding LAN server 5559b12d7b78 (Addr: tcp/10.100.0.5:8300) (DC: lab-dc)
2016-09-30T07:23:07.577407+04:00 lab1 consul[36768]: 2016/09/30 03:23:07 [INFO] consul: Adding LAN server 667edff3624f (Addr: tcp/10.100.0.3:8300) (DC: lab-dc)
2016-09-30T07:23:07.578754+04:00 lab1 consul[36768]: 2016/09/30 03:23:07 [INFO] agent: (LAN) joined: 3 Err: <nil>
2016-09-30T07:23:07.578947+04:00 lab1 consul[36768]: 2016/09/30 03:23:07 [INFO] agent: Join completed. Synced with 3 initial agents
2016-09-30T07:23:07.617762+04:00 lab1 consul[36768]: 2016/09/30 03:23:07 [WARN] raft: Failed to get previous log: 6070 log not found (last: 0)
2016-09-30T07:23:07.779373+04:00 lab1 consul[36768]: 2016/09/30 03:23:07 [INFO] agent: Synced service 'consul'
2016-09-30T07:24:01.740885+04:00 lab1 consul[36768]: 2016/09/30 03:24:01 [WARN] raft: Failed to get previous log: 6081 log not found (last: 6080)
2016-09-30T07:25:01.722878+04:00 lab1 consul[36768]: 2016/09/30 03:25:01 [WARN] raft: Failed to get previous log: 6090 log not found (last: 6089)
2016-09-30T07:26:01.721331+04:00 lab1 consul[36768]: 2016/09/30 03:26:01 [WARN] raft: Failed to get previous log: 6100 log not found (last: 6099)
2016-09-30T07:27:01.719827+04:00 lab1 consul[36768]: 2016/09/30 03:27:01 [WARN] raft: Failed to get previous log: 6110 log not found (last: 6109)
2016-09-30T07:27:07.730900+04:00 lab1 consul[36768]: 2016/09/30 03:27:07 [WARN] raft: Heartbeat timeout from "10.100.0.3:8300" reached, starting election
2016-09-30T07:27:07.731216+04:00 lab1 consul[36768]: 2016/09/30 03:27:07 [INFO] raft: Node at 10.100.0.2:8300 [Candidate] entering Candidate state in term 135
2016-09-30T07:27:13.033611+04:00 lab1 consul[36768]: 2016/09/30 03:27:13 [WARN] raft: Election timeout reached, restarting election
2016-09-30T07:27:13.033936+04:00 lab1 consul[36768]: 2016/09/30 03:27:13 [INFO] raft: Node at 10.100.0.2:8300 [Candidate] entering Candidate state in term 136
2016-09-30T07:27:19.579238+04:00 lab1 consul[36768]: 2016/09/30 03:27:19 [ERR] agent: failed to sync remote state: No cluster leader
2016-09-30T07:27:22.772331+04:00 lab1 consul[36768]: 2016/09/30 03:27:22 [WARN] raft: Election timeout reached, restarting election
2016-09-30T07:27:22.772661+04:00 lab1 consul[36768]: 2016/09/30 03:27:22 [INFO] raft: Node at 10.100.0.2:8300 [Candidate] entering Candidate state in term 137
2016-09-30T07:27:28.269429+04:00 lab1 consul[36768]: 2016/09/30 03:27:28 [WARN] raft: Election timeout reached, restarting election
2016-09-30T07:27:28.269745+04:00 lab1 consul[36768]: 2016/09/30 03:27:28 [INFO] raft: Node at 10.100.0.2:8300 [Candidate] entering Candidate state in term 138
2016-09-30T07:27:33.166207+04:00 lab1 consul[36768]: 2016/09/30 03:27:33 [ERR] agent: coordinate update error: No cluster leader
2016-09-30T07:27:37.761780+04:00 lab1 consul[36768]: 2016/09/30 03:27:37 [WARN] raft: Election timeout reached, restarting election
2016-09-30T07:27:37.762082+04:00 lab1 consul[36768]: 2016/09/30 03:27:37 [INFO] raft: Node at 10.100.0.2:8300 [Candidate] entering Candidate state in term 139
2016-09-30T07:27:45.421772+04:00 lab1 consul[36768]: 2016/09/30 03:27:45 [WARN] raft: Election timeout reached, restarting election
2016-09-30T07:27:45.422095+04:00 lab1 consul[36768]: 2016/09/30 03:27:45 [INFO] raft: Node at 10.100.0.2:8300 [Candidate] entering Candidate state in term 140
2016-09-30T07:27:48.806230+04:00 lab1 consul[36768]: 2016/09/30 03:27:48 [ERR] agent: failed to sync remote state: No cluster leader
2016-09-30T07:27:52.902096+04:00 lab1 consul[36768]: 2016/09/30 03:27:52 [WARN] raft: Election timeout reached, restarting election
2016-09-30T07:27:52.902451+04:00 lab1 consul[36768]: 2016/09/30 03:27:52 [INFO] raft: Node at 10.100.0.2:8300 [Candidate] entering Candidate state in term 141
2016-09-30T07:27:57.963963+04:00 lab1 consul[36768]: 2016/09/30 03:27:57 [WARN] raft: Election timeout reached, restarting election
2016-09-30T07:27:57.964269+04:00 lab1 consul[36768]: 2016/09/30 03:27:57 [INFO] raft: Node at 10.100.0.2:8300 [Candidate] entering Candidate state in term 142
2016-09-30T07:28:05.315450+04:00 lab1 consul[36768]: 2016/09/30 03:28:05 [WARN] raft: Election timeout reached, restarting election
2016-09-30T07:28:05.315798+04:00 lab1 consul[36768]: 2016/09/30 03:28:05 [INFO] raft: Node at 10.100.0.2:8300 [Candidate] entering Candidate state in term 143
2016-09-30T07:28:08.969528+04:00 lab1 consul[36768]: 2016/09/30 03:28:08 [ERR] agent: coordinate update error: No cluster leader
2016-09-30T07:28:10.924416+04:00 lab1 consul[36768]: 2016/09/30 03:28:10 [INFO] raft: Node at 10.100.0.2:8300 [Follower] entering Follower state (Leader: "")
2016-09-30T07:28:10.925064+04:00 lab1 consul[36768]: 2016/09/30 03:28:10 [WARN] raft: Failed to get previous log: 6119 log not found (last: 6111)
2016-09-30T07:28:11.042603+04:00 lab1 consul[36768]: 2016/09/30 03:28:11 [INFO] consul: New leader elected: 667edff3624f
2016-09-30T07:28:11.143508+04:00 lab1 consul[36768]: 2016/09/30 03:28:11 [WARN] raft: Failed to get previous log: 6123 log not found (last: 6122)
2016-09-30T07:28:17.961262+04:00 lab1 consul[36768]: 2016/09/30 03:28:17 [WARN] raft: Heartbeat timeout from "10.100.0.3:8300" reached, starting election
2016-09-30T07:28:17.961638+04:00 lab1 consul[36768]: 2016/09/30 03:28:17 [INFO] raft: Node at 10.100.0.2:8300 [Candidate] entering Candidate state in term 145
2016-09-30T07:28:24.977746+04:00 lab1 consul[36768]: 2016/09/30 03:28:24 [WARN] raft: Election timeout reached, restarting election
2016-09-30T07:28:24.978070+04:00 lab1 consul[36768]: 2016/09/30 03:28:24 [INFO] raft: Node at 10.100.0.2:8300 [Candidate] entering Candidate state in term 146
2016-09-30T07:28:32.823440+04:00 lab1 consul[36768]: 2016/09/30 03:28:32 [WARN] raft: Election timeout reached, restarting election
2016-09-30T07:28:32.823817+04:00 lab1 consul[36768]: 2016/09/30 03:28:32 [INFO] raft: Node at 10.100.0.2:8300 [Candidate] entering Candidate state in term 147
2016-09-30T07:28:37.996054+04:00 lab1 consul[36768]: 2016/09/30 03:28:37 [ERR] agent: coordinate update error: No cluster leader
2016-09-30T07:28:40.534070+04:00 lab1 consul[36768]: 2016/09/30 03:28:40 [WARN] raft: Election timeout reached, restarting election
2016-09-30T07:28:40.534358+04:00 lab1 consul[36768]: 2016/09/30 03:28:40 [INFO] raft: Node at 10.100.0.2:8300 [Candidate] entering Candidate state in term 148
2016-09-30T07:28:46.176035+04:00 lab1 consul[36768]: 2016/09/30 03:28:46 [WARN] raft: Election timeout reached, restarting election
2016-09-30T07:28:46.176203+04:00 lab1 consul[36768]: 2016/09/30 03:28:46 [INFO] raft: Node at 10.100.0.2:8300 [Candidate] entering Candidate state in term 149
2016-09-30T07:28:53.520091+04:00 lab1 consul[36768]: 2016/09/30 03:28:53 [WARN] raft: Election timeout reached, restarting election
2016-09-30T07:28:53.520320+04:00 lab1 consul[36768]: 2016/09/30 03:28:53 [INFO] raft: Node at 10.100.0.2:8300 [Candidate] entering Candidate state in term 150
LAB2:
2016-09-30T07:27:07.783116+04:00 lab2 consul[2620]: 2016/09/30 03:27:07 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:27:13.049205+04:00 lab2 consul[2620]: 2016/09/30 03:27:13 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:27:22.790532+04:00 lab2 consul[2620]: 2016/09/30 03:27:22 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:27:28.290367+04:00 lab2 consul[2620]: 2016/09/30 03:27:28 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:27:37.781775+04:00 lab2 consul[2620]: 2016/09/30 03:27:37 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:27:45.439711+04:00 lab2 consul[2620]: 2016/09/30 03:27:45 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:27:52.947866+04:00 lab2 consul[2620]: 2016/09/30 03:27:52 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:27:58.022699+04:00 lab2 consul[2620]: 2016/09/30 03:27:58 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:28:05.330920+04:00 lab2 consul[2620]: 2016/09/30 03:28:05 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:28:11.040364+04:00 lab2 consul[2620]: 2016/09/30 03:28:11 [INFO] consul: New leader elected: 667edff3624f
2016-09-30T07:28:17.980317+04:00 lab2 consul[2620]: 2016/09/30 03:28:17 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:28:25.005076+04:00 lab2 consul[2620]: 2016/09/30 03:28:25 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:28:32.854912+04:00 lab2 consul[2620]: 2016/09/30 03:28:32 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:28:40.546242+04:00 lab2 consul[2620]: 2016/09/30 03:28:40 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:28:46.196128+04:00 lab2 consul[2620]: 2016/09/30 03:28:46 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:28:53.537406+04:00 lab2 consul[2620]: 2016/09/30 03:28:53 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:29:01.087394+04:00 lab2 consul[2620]: 2016/09/30 03:29:01 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:29:08.303714+04:00 lab2 consul[2620]: 2016/09/30 03:29:08 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:29:16.095235+04:00 lab2 consul[2620]: 2016/09/30 03:29:16 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:29:20.040292+04:00 lab2 consul[2620]: 2016/09/30 03:29:20 [INFO] consul: New leader elected: 667edff3624f
LAB3:
2016-09-30T07:22:53.120066+04:00 lab3 consul[9295]: 2016/09/30 03:22:53 [INFO] serf: EventMemberLeave: 2f74fbbffdd2 10.100.0.2
2016-09-30T07:22:53.120377+04:00 lab3 consul[9295]: 2016/09/30 03:22:53 [INFO] consul: Removing LAN server 2f74fbbffdd2 (Addr: tcp/10.100.0.2:8300) (DC: lab-dc)
2016-09-30T07:22:53.120554+04:00 lab3 consul[9295]: 2016/09/30 03:22:53 [INFO] consul: member '2f74fbbffdd2' left, deregistering
2016-09-30T07:23:07.578830+04:00 lab3 consul[9295]: 2016/09/30 03:23:07 [INFO] serf: EventMemberJoin: 1ec48b592c2f 10.100.0.2
2016-09-30T07:23:07.579001+04:00 lab3 consul[9295]: 2016/09/30 03:23:07 [INFO] consul: Adding LAN server 1ec48b592c2f (Addr: tcp/10.100.0.2:8300) (DC: lab-dc)
2016-09-30T07:23:07.579080+04:00 lab3 consul[9295]: 2016/09/30 03:23:07 [INFO] raft: Updating configuration with AddStaging (10.100.0.2:8300, 10.100.0.2:8300) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300} {Suffrage:Voter ID:10.100.0.2:8300 Address:10.100.0.2:8300}]
2016-09-30T07:23:07.602687+04:00 lab3 consul[9295]: 2016/09/30 03:23:07 [INFO] raft: Added peer 10.100.0.2:8300, starting replication
2016-09-30T07:23:07.620550+04:00 lab3 consul[9295]: 2016/09/30 03:23:07 [WARN] raft: AppendEntries to {Voter 10.100.0.2:8300 10.100.0.2:8300} rejected, sending older logs (next: 1)
2016-09-30T07:23:07.622914+04:00 lab3 consul[9295]: 2016/09/30 03:23:07 [INFO] consul: member '1ec48b592c2f' joined, marking health alive
2016-09-30T07:23:13.095596+04:00 lab3 consul[9295]: 2016/09/30 03:23:13 [INFO] raft: pipelining replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:24:01.602239+04:00 lab3 consul[9295]: 2016/09/30 03:24:01 [INFO] raft: Updating configuration with RemoveServer (10.100.0.2:8300, ) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300}]
2016-09-30T07:24:01.643138+04:00 lab3 consul[9295]: 2016/09/30 03:24:01 [INFO] raft: Removed peer 10.100.0.2:8300, stopping replication after 6080
2016-09-30T07:24:01.643388+04:00 lab3 consul[9295]: 2016/09/30 03:24:01 [INFO] raft: aborting pipeline replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:24:01.674950+04:00 lab3 consul[9295]: 2016/09/30 03:24:01 [INFO] raft: Updating configuration with AddStaging (10.100.0.2:8300, 10.100.0.2:8300) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300} {Suffrage:Voter ID:10.100.0.2:8300 Address:10.100.0.2:8300}]
2016-09-30T07:24:01.743081+04:00 lab3 consul[9295]: 2016/09/30 03:24:01 [INFO] raft: Added peer 10.100.0.2:8300, starting replication
2016-09-30T07:24:01.743611+04:00 lab3 consul[9295]: 2016/09/30 03:24:01 [WARN] raft: AppendEntries to {Voter 10.100.0.2:8300 10.100.0.2:8300} rejected, sending older logs (next: 6081)
2016-09-30T07:24:01.793858+04:00 lab3 consul[9295]: 2016/09/30 03:24:01 [INFO] raft: pipelining replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:25:01.617560+04:00 lab3 consul[9295]: 2016/09/30 03:25:01 [INFO] raft: Updating configuration with RemoveServer (10.100.0.2:8300, ) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300}]
2016-09-30T07:25:01.658363+04:00 lab3 consul[9295]: 2016/09/30 03:25:01 [INFO] raft: Removed peer 10.100.0.2:8300, stopping replication after 6089
2016-09-30T07:25:01.658633+04:00 lab3 consul[9295]: 2016/09/30 03:25:01 [INFO] raft: aborting pipeline replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:25:01.682281+04:00 lab3 consul[9295]: 2016/09/30 03:25:01 [INFO] raft: Updating configuration with AddStaging (10.100.0.2:8300, 10.100.0.2:8300) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300} {Suffrage:Voter ID:10.100.0.2:8300 Address:10.100.0.2:8300}]
2016-09-30T07:25:01.725018+04:00 lab3 consul[9295]: 2016/09/30 03:25:01 [INFO] raft: Added peer 10.100.0.2:8300, starting replication
2016-09-30T07:25:01.725635+04:00 lab3 consul[9295]: 2016/09/30 03:25:01 [WARN] raft: AppendEntries to {Voter 10.100.0.2:8300 10.100.0.2:8300} rejected, sending older logs (next: 6090)
2016-09-30T07:25:01.741934+04:00 lab3 consul[9295]: 2016/09/30 03:25:01 [INFO] raft: pipelining replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:26:01.614883+04:00 lab3 consul[9295]: 2016/09/30 03:26:01 [INFO] raft: Updating configuration with RemoveServer (10.100.0.2:8300, ) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300}]
2016-09-30T07:26:01.656898+04:00 lab3 consul[9295]: 2016/09/30 03:26:01 [INFO] raft: Removed peer 10.100.0.2:8300, stopping replication after 6099
2016-09-30T07:26:01.657141+04:00 lab3 consul[9295]: 2016/09/30 03:26:01 [INFO] raft: aborting pipeline replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:26:01.681081+04:00 lab3 consul[9295]: 2016/09/30 03:26:01 [INFO] raft: Updating configuration with AddStaging (10.100.0.2:8300, 10.100.0.2:8300) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300} {Suffrage:Voter ID:10.100.0.2:8300 Address:10.100.0.2:8300}]
2016-09-30T07:26:01.723625+04:00 lab3 consul[9295]: 2016/09/30 03:26:01 [INFO] raft: Added peer 10.100.0.2:8300, starting replication
2016-09-30T07:26:01.724226+04:00 lab3 consul[9295]: 2016/09/30 03:26:01 [WARN] raft: AppendEntries to {Voter 10.100.0.2:8300 10.100.0.2:8300} rejected, sending older logs (next: 6100)
2016-09-30T07:26:01.740222+04:00 lab3 consul[9295]: 2016/09/30 03:26:01 [INFO] raft: pipelining replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:27:01.604692+04:00 lab3 consul[9295]: 2016/09/30 03:27:01 [INFO] raft: Updating configuration with RemoveServer (10.100.0.2:8300, ) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300}]
2016-09-30T07:27:01.647116+04:00 lab3 consul[9295]: 2016/09/30 03:27:01 [INFO] raft: Removed peer 10.100.0.2:8300, stopping replication after 6109
2016-09-30T07:27:01.647273+04:00 lab3 consul[9295]: 2016/09/30 03:27:01 [INFO] raft: aborting pipeline replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:27:01.679878+04:00 lab3 consul[9295]: 2016/09/30 03:27:01 [INFO] raft: Updating configuration with AddStaging (10.100.0.2:8300, 10.100.0.2:8300) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300} {Suffrage:Voter ID:10.100.0.2:8300 Address:10.100.0.2:8300}]
2016-09-30T07:27:01.722120+04:00 lab3 consul[9295]: 2016/09/30 03:27:01 [INFO] raft: Added peer 10.100.0.2:8300, starting replication
2016-09-30T07:27:01.722618+04:00 lab3 consul[9295]: 2016/09/30 03:27:01 [WARN] raft: AppendEntries to {Voter 10.100.0.2:8300 10.100.0.2:8300} rejected, sending older logs (next: 6110)
2016-09-30T07:27:01.746416+04:00 lab3 consul[9295]: 2016/09/30 03:27:01 [INFO] raft: pipelining replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:27:01.746666+04:00 lab3 consul[9295]: 2016/09/30 03:27:01 [INFO] raft: Updating configuration with RemoveServer (10.100.0.2:8300, ) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300}]
2016-09-30T07:27:01.788726+04:00 lab3 consul[9295]: 2016/09/30 03:27:01 [INFO] raft: Removed peer 10.100.0.2:8300, stopping replication after 6111
2016-09-30T07:27:01.788826+04:00 lab3 consul[9295]: 2016/09/30 03:27:01 [INFO] raft: aborting pipeline replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:27:07.788381+04:00 lab3 consul[9295]: 2016/09/30 03:27:07 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:27:13.054346+04:00 lab3 consul[9295]: 2016/09/30 03:27:13 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:27:22.795664+04:00 lab3 consul[9295]: 2016/09/30 03:27:22 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:27:28.295424+04:00 lab3 consul[9295]: 2016/09/30 03:27:28 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:27:37.788747+04:00 lab3 consul[9295]: 2016/09/30 03:27:37 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:27:45.444830+04:00 lab3 consul[9295]: 2016/09/30 03:27:45 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:27:52.953140+04:00 lab3 consul[9295]: 2016/09/30 03:27:52 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:27:58.027851+04:00 lab3 consul[9295]: 2016/09/30 03:27:58 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:28:01.620482+04:00 lab3 consul[9295]: 2016/09/30 03:28:01 [INFO] raft: Updating configuration with AddStaging (10.100.0.2:8300, 10.100.0.2:8300) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300} {Suffrage:Voter ID:10.100.0.2:8300 Address:10.100.0.2:8300}]
2016-09-30T07:28:01.662381+04:00 lab3 consul[9295]: 2016/09/30 03:28:01 [INFO] raft: Added peer 10.100.0.2:8300, starting replication
2016-09-30T07:28:01.663414+04:00 lab3 consul[9295]: 2016/09/30 03:28:01 [ERR] raft: peer {Voter 10.100.0.2:8300 10.100.0.2:8300} has newer term, stopping replication
2016-09-30T07:28:01.663678+04:00 lab3 consul[9295]: 2016/09/30 03:28:01 [INFO] raft: Node at 10.100.0.3:8300 [Follower] entering Follower state (Leader: "")
2016-09-30T07:28:01.663842+04:00 lab3 consul[9295]: 2016/09/30 03:28:01 [INFO] raft: aborting pipeline replication to peer {Voter 10.100.0.5:8300 10.100.0.5:8300} 2016/09/30 03:28:01 [INFO] consul: cluster leadership lost
2016-09-30T07:28:01.663971+04:00 lab3 consul[9295]:
2016-09-30T07:28:01.664092+04:00 lab3 consul[9295]: 2016/09/30 03:28:01 [ERR] consul: failed to add raft peer: leadership lost while committing log
2016-09-30T07:28:01.664211+04:00 lab3 consul[9295]: 2016/09/30 03:28:01 [ERR] consul: failed to reconcile member: {1ec48b592c2f 10.100.0.2 8301 map[vsn:2 vsn_min:2 vsn_max:3 build:0.7.0:'a189091 port:8300 role:consul dc:lab-dc] alive 1 4 2 2 4 4}: leadership lost while committing log
2016-09-30T07:28:01.664333+04:00 lab3 consul[9295]: 2016/09/30 03:28:01 [ERR] consul: failed to reconcile: leadership lost while committing log
2016-09-30T07:28:05.379163+04:00 lab3 consul[9295]: 2016/09/30 03:28:05 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since our last index is greater (6119, 6111)
2016-09-30T07:28:10.728567+04:00 lab3 consul[9295]: 2016/09/30 03:28:10 [WARN] raft: Heartbeat timeout from "" reached, starting election
2016-09-30T07:28:10.728730+04:00 lab3 consul[9295]: 2016/09/30 03:28:10 [INFO] raft: Node at 10.100.0.3:8300 [Candidate] entering Candidate state in term 144
2016-09-30T07:28:10.927296+04:00 lab3 consul[9295]: 2016/09/30 03:28:10 [INFO] raft: Election won. Tally: 2
2016-09-30T07:28:10.927394+04:00 lab3 consul[9295]: 2016/09/30 03:28:10 [INFO] raft: Node at 10.100.0.3:8300 [Leader] entering Leader state
2016-09-30T07:28:10.927466+04:00 lab3 consul[9295]: 2016/09/30 03:28:10 [INFO] raft: Added peer 10.100.0.5:8300, starting replication
2016-09-30T07:28:10.927529+04:00 lab3 consul[9295]: 2016/09/30 03:28:10 [INFO] raft: Added peer 10.100.0.2:8300, starting replication
2016-09-30T07:28:10.927590+04:00 lab3 consul[9295]: 2016/09/30 03:28:10 [INFO] consul: cluster leadership acquired
2016-09-30T07:28:10.927668+04:00 lab3 consul[9295]: 2016/09/30 03:28:10 [INFO] consul: New leader elected: 667edff3624f
2016-09-30T07:28:10.927890+04:00 lab3 consul[9295]: 2016/09/30 03:28:10 [WARN] raft: AppendEntries to {Voter 10.100.0.2:8300 10.100.0.2:8300} rejected, sending older logs (next: 6112)
2016-09-30T07:28:10.974600+04:00 lab3 consul[9295]: 2016/09/30 03:28:10 [INFO] raft: pipelining replication to peer {Voter 10.100.0.5:8300 10.100.0.5:8300}
2016-09-30T07:28:10.977141+04:00 lab3 consul[9295]: 2016/09/30 03:28:10 [INFO] raft: pipelining replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:28:11.027251+04:00 lab3 consul[9295]: 2016/09/30 03:28:11 [INFO] raft: Updating configuration with RemoveServer (10.100.0.2:8300, ) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300}]
2016-09-30T07:28:11.070537+04:00 lab3 consul[9295]: 2016/09/30 03:28:11 [INFO] raft: Removed peer 10.100.0.2:8300, stopping replication after 6122
2016-09-30T07:28:11.070751+04:00 lab3 consul[9295]: 2016/09/30 03:28:11 [INFO] raft: aborting pipeline replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:28:11.108199+04:00 lab3 consul[9295]: 2016/09/30 03:28:11 [INFO] raft: Updating configuration with AddStaging (10.100.0.2:8300, 10.100.0.2:8300) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300} {Suffrage:Voter ID:10.100.0.2:8300 Address:10.100.0.2:8300}]
2016-09-30T07:28:11.145438+04:00 lab3 consul[9295]: 2016/09/30 03:28:11 [INFO] raft: Added peer 10.100.0.2:8300, starting replication
2016-09-30T07:28:11.146335+04:00 lab3 consul[9295]: 2016/09/30 03:28:11 [WARN] raft: AppendEntries to {Voter 10.100.0.2:8300 10.100.0.2:8300} rejected, sending older logs (next: 6123)
2016-09-30T07:28:11.166612+04:00 lab3 consul[9295]: 2016/09/30 03:28:11 [INFO] raft: Updating configuration with RemoveServer (10.100.0.2:8300, ) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300}]
2016-09-30T07:28:11.168905+04:00 lab3 consul[9295]: 2016/09/30 03:28:11 [INFO] raft: pipelining replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:28:11.220625+04:00 lab3 consul[9295]: 2016/09/30 03:28:11 [INFO] raft: Removed peer 10.100.0.2:8300, stopping replication after 6124
2016-09-30T07:28:11.220867+04:00 lab3 consul[9295]: 2016/09/30 03:28:11 [INFO] raft: aborting pipeline replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:28:17.985551+04:00 lab3 consul[9295]: 2016/09/30 03:28:17 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:28:25.010348+04:00 lab3 consul[9295]: 2016/09/30 03:28:25 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:28:32.860039+04:00 lab3 consul[9295]: 2016/09/30 03:28:32 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:28:40.551476+04:00 lab3 consul[9295]: 2016/09/30 03:28:40 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:28:46.201386+04:00 lab3 consul[9295]: 2016/09/30 03:28:46 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:28:53.542605+04:00 lab3 consul[9295]: 2016/09/30 03:28:53 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:29:01.092588+04:00 lab3 consul[9295]: 2016/09/30 03:29:01 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:29:08.308824+04:00 lab3 consul[9295]: 2016/09/30 03:29:08 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:29:10.999121+04:00 lab3 consul[9295]: 2016/09/30 03:29:10 [INFO] raft: Updating configuration with AddStaging (10.100.0.2:8300, 10.100.0.2:8300) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300} {Suffrage:Voter ID:10.100.0.2:8300 Address:10.100.0.2:8300}]
2016-09-30T07:29:11.035763+04:00 lab3 consul[9295]: 2016/09/30 03:29:11 [INFO] raft: Added peer 10.100.0.2:8300, starting replication
2016-09-30T07:29:11.036880+04:00 lab3 consul[9295]: 2016/09/30 03:29:11 [ERR] raft: peer {Voter 10.100.0.2:8300 10.100.0.2:8300} has newer term, stopping replication
2016-09-30T07:29:11.037147+04:00 lab3 consul[9295]: 2016/09/30 03:29:11 [INFO] raft: Node at 10.100.0.3:8300 [Follower] entering Follower state (Leader: "")
2016-09-30T07:29:11.037572+04:00 lab3 consul[9295]: 2016/09/30 03:29:11 [INFO] consul: cluster leadership lost
2016-09-30T07:29:11.037868+04:00 lab3 consul[9295]: 2016/09/30 03:29:11 [INFO] raft: aborting pipeline replication to peer {Voter 10.100.0.5:8300 10.100.0.5:8300}
2016-09-30T07:29:11.038140+04:00 lab3 consul[9295]: 2016/09/30 03:29:11 [ERR] consul: failed to add raft peer: leadership lost while committing log
2016-09-30T07:29:11.038328+04:00 lab3 consul[9295]: 2016/09/30 03:29:11 [ERR] consul: failed to reconcile member: {1ec48b592c2f 10.100.0.2 8301 map[build:0.7.0:'a189091 port:8300 role:consul dc:lab-dc vsn:2 vsn_min:2 vsn_max:3] alive 1 4 2 2 4 4}: leadership lost while committing log
2016-09-30T07:29:11.038494+04:00 lab3 consul[9295]: 2016/09/30 03:29:11 [ERR] consul: failed to reconcile: leadership lost while committing log
2016-09-30T07:29:16.144031+04:00 lab3 consul[9295]: 2016/09/30 03:29:16 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since our last index is greater (6131, 6124)
2016-09-30T07:29:19.708014+04:00 lab3 consul[9295]: 2016/09/30 03:29:19 [WARN] raft: Heartbeat timeout from "" reached, starting election
2016-09-30T07:29:19.708322+04:00 lab3 consul[9295]: 2016/09/30 03:29:19 [INFO] raft: Node at 10.100.0.3:8300 [Candidate] entering Candidate state in term 154
2016-09-30T07:29:19.899955+04:00 lab3 consul[9295]: 2016/09/30 03:29:19 [INFO] raft: Election won. Tally: 2
2016-09-30T07:29:19.900053+04:00 lab3 consul[9295]: 2016/09/30 03:29:19 [INFO] raft: Node at 10.100.0.3:8300 [Leader] entering Leader state
2016-09-30T07:29:19.900128+04:00 lab3 consul[9295]: 2016/09/30 03:29:19 [INFO] raft: Added peer 10.100.0.5:8300, starting replication
2016-09-30T07:29:19.900192+04:00 lab3 consul[9295]: 2016/09/30 03:29:19 [INFO] raft: Added peer 10.100.0.2:8300, starting replication
2016-09-30T07:29:19.900293+04:00 lab3 consul[9295]: 2016/09/30 03:29:19 [INFO] consul: cluster leadership acquired
2016-09-30T07:29:19.900397+04:00 lab3 consul[9295]: 2016/09/30 03:29:19 [INFO] consul: New leader elected: 667edff3624f
2016-09-30T07:29:19.900526+04:00 lab3 consul[9295]: 2016/09/30 03:29:19 [WARN] raft: AppendEntries to {Voter 10.100.0.2:8300 10.100.0.2:8300} rejected, sending older logs (next: 6125)
2016-09-30T07:29:19.925227+04:00 lab3 consul[9295]: 2016/09/30 03:29:19 [INFO] raft: pipelining replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:29:19.928266+04:00 lab3 consul[9295]: 2016/09/30 03:29:19 [INFO] raft: pipelining replication to peer {Voter 10.100.0.5:8300 10.100.0.5:8300}
2016-09-30T07:29:19.991687+04:00 lab3 consul[9295]: 2016/09/30 03:29:19 [INFO] raft: Updating configuration with RemoveServer (10.100.0.2:8300, ) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300}]
2016-09-30T07:29:20.027147+04:00 lab3 consul[9295]: 2016/09/30 03:29:20 [INFO] raft: Removed peer 10.100.0.2:8300, stopping replication after 6134
2016-09-30T07:29:20.027277+04:00 lab3 consul[9295]: 2016/09/30 03:29:20 [INFO] raft: aborting pipeline replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:29:20.061687+04:00 lab3 consul[9295]: 2016/09/30 03:29:20 [INFO] raft: Updating configuration with AddStaging (10.100.0.2:8300, 10.100.0.2:8300) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300} {Suffrage:Voter ID:10.100.0.2:8300 Address:10.100.0.2:8300}]
2016-09-30T07:29:20.102204+04:00 lab3 consul[9295]: 2016/09/30 03:29:20 [INFO] raft: Added peer 10.100.0.2:8300, starting replication
2016-09-30T07:29:20.103254+04:00 lab3 consul[9295]: 2016/09/30 03:29:20 [WARN] raft: AppendEntries to {Voter 10.100.0.2:8300 10.100.0.2:8300} rejected, sending older logs (next: 6135)
2016-09-30T07:29:20.125207+04:00 lab3 consul[9295]: 2016/09/30 03:29:20 [INFO] raft: pipelining replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:29:20.125477+04:00 lab3 consul[9295]: 2016/09/30 03:29:20 [INFO] raft: Updating configuration with RemoveServer (10.100.0.2:8300, ) to [{Suffrage:Voter ID:10.100.0.3:8300 Address:10.100.0.3:8300} {Suffrage:Voter ID:10.100.0.5:8300 Address:10.100.0.5:8300}]
2016-09-30T07:29:20.168941+04:00 lab3 consul[9295]: 2016/09/30 03:29:20 [INFO] raft: Removed peer 10.100.0.2:8300, stopping replication after 6136
2016-09-30T07:29:20.169330+04:00 lab3 consul[9295]: 2016/09/30 03:29:20 [INFO] raft: aborting pipeline replication to peer {Voter 10.100.0.2:8300 10.100.0.2:8300}
2016-09-30T07:29:32.308206+04:00 lab3 consul[9295]: 2016/09/30 03:29:32 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:29:38.732849+04:00 lab3 consul[9295]: 2016/09/30 03:29:38 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
2016-09-30T07:29:44.932742+04:00 lab3 consul[9295]: 2016/09/30 03:29:44 [WARN] raft: Rejecting vote request from 10.100.0.2:8300 since we have a leader: 10.100.0.3:8300
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment