Skip to content

Instantly share code, notes, and snippets.

@slobo

slobo/nomad log

Created February 14, 2018 03:34
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save slobo/8b74e70fd6013d988f2d1dc8446ee50c to your computer and use it in GitHub Desktop.
Save slobo/8b74e70fd6013d988f2d1dc8446ee50c to your computer and use it in GitHub Desktop.
Download ZIP
Raw
nomad log
Started nomad server.
Loaded configuration from /etc/nomad.d/server/server.hcl
==> Starting Nomad agent...
2018/02/13 06:42:22 [WARN] raft: Rejecting vote request from 10.10.1.119:8300 since we have a leader: 10.10.11.34:8300
raft: Rejecting vote request from 10.10.1.119:8300 since we have a leader: 10.10.11.34:8300
2018/02/13 06:42:24 [ERR] raft: peer {Voter e53b47d1-5560-54bc-d221-b609472f2ab3 10.10.1.119:8300} has newer term, stopping replication
raft: peer {Voter e53b47d1-5560-54bc-d221-b609472f2ab3 10.10.1.119:8300} has newer term, stopping replication
2018/02/13 06:42:25 [INFO] raft: aborting pipeline replication to peer {Voter e53b47d1-5560-54bc-d221-b609472f2ab3 10.10.1.119:8300}
2018/02/13 06:42:25 [INFO] raft: Node at 10.10.11.34:8300 [Follower] entering Follower state (Leader: "")
2018/02/13 06:42:25 [INFO] consul: cluster leadership lost
2018/02/13 06:42:25 [WARN] memberlist: Refuting a suspect message (from: cluster-spot-worker-03369fa4893ad47ea)
2018/02/13 06:42:25 [WARN] memberlist: Refuting a suspect message (from: live-cluster-admin-06ddc0342365da23b.us-east-1)
2018/02/13 06:42:25 [INFO] raft: aborting pipeline replication to peer {Voter 876f6571-6a39-8ecd-d10c-ae37b37734da 10.10.3.249:8300}
2018/02/13 06:42:25 [ERR] memberlist: Failed fallback ping: write tcp 10.10.11.34:45200->10.10.6.69:8301: i/o timeout
2018/02/13 06:42:25 [INFO] memberlist: Suspect cluster-spot-worker-03369fa4893ad47ea has failed, no acks received
raft: aborting pipeline replication to peer {Voter e53b47d1-5560-54bc-d221-b609472f2ab3 10.10.1.119:8300}
raft: Node at 10.10.11.34:8300 [Follower] entering Follower state (Leader: "")
consul: cluster leadership lost
memberlist: Refuting a suspect message (from: cluster-spot-worker-03369fa4893ad47ea)
memberlist: Refuting a suspect message (from: live-cluster-admin-06ddc0342365da23b.us-east-1)
raft: aborting pipeline replication to peer {Voter 876f6571-6a39-8ecd-d10c-ae37b37734da 10.10.3.249:8300}
memberlist: Failed fallback ping: write tcp 10.10.11.34:45200->10.10.6.69:8301: i/o timeout
memberlist: Suspect cluster-spot-worker-03369fa4893ad47ea has failed, no acks received
2018/02/13 06:42:30 [WARN] consul: error getting server health from "live-cluster-admin-06ddc0342365da23b": context deadline exceeded
consul: error getting server health from "live-cluster-admin-06ddc0342365da23b": context deadline exceeded
2018/02/13 06:42:31 [WARN] consul: error getting server health from "live-cluster-admin-094858a83da874c6a": context deadline exceeded
consul: error getting server health from "live-cluster-admin-094858a83da874c6a": context deadline exceeded
2018/02/13 06:42:32 [ERR] consul.acl: Failed to get policy from ACL datacenter: No cluster leader
consul.acl: Failed to get policy from ACL datacenter: No cluster leader
2018/02/13 06:42:33 [WARN] consul.coordinate: Batch update failed: node is not the leader
consul.coordinate: Batch update failed: node is not the leader
2018/02/13 06:42:33 [WARN] consul: error getting server health from "live-cluster-admin-06ddc0342365da23b": last request still outstanding
consul: error getting server health from "live-cluster-admin-06ddc0342365da23b": last request still outstanding
2018/02/13 06:42:34 [DEBUG] raft-net: 10.10.11.34:8300 accepted connection from: 10.10.1.119:53248
2018/02/13 06:42:34 [WARN] consul: error getting server health from "live-cluster-admin-094858a83da874c6a": last request still outstanding
2018/02/13 06:42:34 [WARN] consul: error getting server health from "live-cluster-admin-06bb5a1bedeb9b2a2": context deadline exceeded
consul: error getting server health from "live-cluster-admin-094858a83da874c6a": last request still outstanding
consul: error getting server health from "live-cluster-admin-06bb5a1bedeb9b2a2": context deadline exceeded
2018/02/13 06:42:34 [ERR] memberlist: Failed fallback ping: write tcp 10.10.11.34:50910->10.10.3.249:8301: i/o timeout
2018/02/13 06:42:35 [WARN] memberlist: Refuting a suspect message (from: live-cluster-admin-094858a83da874c6a)
2018/02/13 06:42:35 [INFO] consul: New leader elected: live-cluster-admin-06bb5a1bedeb9b2a2
memberlist: Refuting a suspect message (from: live-cluster-admin-094858a83da874c6a)
consul: New leader elected: live-cluster-admin-06bb5a1bedeb9b2a2
2018/02/13 06:42:35 [WARN] consul: error getting server health from "live-cluster-admin-06ddc0342365da23b": last request still outstanding
consul: error getting server health from "live-cluster-admin-06ddc0342365da23b": last request still outstanding
2018/02/13 06:42:35 [INFO] memberlist: Suspect live-cluster-admin-06ddc0342365da23b has failed, no acks received
memberlist: Suspect live-cluster-admin-06ddc0342365da23b has failed, no acks received
2018/02/13 06:42:36 [WARN] consul: error getting server health from "live-cluster-admin-094858a83da874c6a": context deadline exceeded
consul: error getting server health from "live-cluster-admin-094858a83da874c6a": context deadline exceeded
2018/02/13 06:42:39 [ERR] memberlist: Failed fallback ping: write tcp 10.10.11.34:38696->10.10.3.249:8302: i/o timeout
memberlist: Failed fallback ping: write tcp 10.10.11.34:38696->10.10.3.249:8302: i/o timeout
2018/02/13 06:42:43 [INFO] memberlist: Suspect live-cluster-admin-06ddc0342365da23b.us-east-1 has failed, no acks received
2018/02/13 06:42:40 [WARN] consul: error getting server health from "live-cluster-admin-06ddc0342365da23b": last request still outstanding
2018/02/13 06:42:46 [ERR] raft-net: Failed to flush response: write tcp 10.10.11.34:8300->10.10.1.119:59380: write: broken pipe
2018/02/13 06:42:46 [WARN] consul: error getting server health from "live-cluster-admin-094858a83da874c6a": context deadline exceeded
consul: error getting server health from "live-cluster-admin-094858a83da874c6a": context deadline exceeded
2018/02/13 06:42:47 [WARN] consul: error getting server health from "live-cluster-admin-06bb5a1bedeb9b2a2": context deadline exceeded
2018/02/13 06:42:49 [ERR] memberlist: Failed fallback ping: write tcp 10.10.11.34:55990->10.10.5.157:8301: i/o timeout
2018/02/13 06:42:51 [INFO] memberlist: Suspect cluster-spot-worker-001782337f6a961d4 has failed, no acks received
2018/02/13 06:42:55 [INFO] memberlist: Marking live-cluster-admin-06ddc0342365da23b as failed, suspect timeout reached (1 peer confirmations)
2018/02/13 06:43:00 [DEBUG] raft-net: 10.10.11.34:8300 accepted connection from: 10.10.1.119:44746
2018/02/13 06:43:01 [ERR] consul.acl: Failed to get policy from ACL datacenter: No cluster leader
consul.acl: Failed to get policy from ACL datacenter: No cluster leader
2018/02/13 06:42:59 [ERR] agent: failed to sync remote state: No cluster leader
agent: failed to sync remote state: No cluster leader
2018/02/13 06:43:02 [INFO] raft: Duplicate RequestVote for same term: 63
raft: Duplicate RequestVote for same term: 63
2018/02/13 06:43:00 [ERR] consul.acl: Failed to get policy from ACL datacenter: No cluster leader
2018/02/13 06:43:01 [INFO] serf: EventMemberFailed: live-cluster-admin-06ddc0342365da23b 10.10.3.249
2018/02/13 06:43:01 [ERR] consul.acl: Failed to get policy from ACL datacenter: No cluster leader
2018/02/13 06:43:02 [ERR] consul.acl: Failed to get policy from ACL datacenter: No cluster leader
2018/02/13 06:43:04 [DEBUG] raft-net: 10.10.11.34:8300 accepted connection from: 10.10.1.119:48998
2018/02/13 06:43:03 [WARN] consul: error getting server health from "live-cluster-admin-06ddc0342365da23b": last request still outstanding
2018/02/13 06:43:03 [INFO] consul: Removing LAN server live-cluster-admin-06ddc0342365da23b (Addr: tcp/10.10.3.249:8300) (DC: us-east-1)
2018/02/13 06:43:03 [WARN] memberlist: Refuting a suspect message (from: live-cluster-admin-06ddc0342365da23b.us-east-1)
memberlist: Refuting a suspect message (from: live-cluster-admin-06ddc0342365da23b.us-east-1)
2018/02/13 06:43:05 [WARN] raft: Failed to get previous log: 5376652 log not found (last: 5376650)
raft: Failed to get previous log: 5376652 log not found (last: 5376650)
2018/02/13 06:43:05 [WARN] consul: error getting server health from "live-cluster-admin-094858a83da874c6a": context deadline exceeded
consul: error getting server health from "live-cluster-admin-094858a83da874c6a": context deadline exceeded
2018/02/13 06:43:08 [ERR] memberlist: Failed fallback ping: write tcp 10.10.11.34:44680->10.10.1.119:8302: i/o timeout
memberlist: Failed fallback ping: write tcp 10.10.11.34:44680->10.10.1.119:8302: i/o timeout
....
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment