Skip to content

Instantly share code, notes, and snippets.

@sprutner

sprutner/nomad-log

Created July 24, 2017 17:47
Show Gist options
  • Save sprutner/270583dc97a50d53493a3b6ab5722e3c to your computer and use it in GitHub Desktop.
Save sprutner/270583dc97a50d53493a3b6ab5722e3c to your computer and use it in GitHub Desktop.
Download ZIP
nomad error log
Raw
nomad-log
Jul 24 17:44:25 ip-10-104-20-31 systemd[1]: nomad.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 24 17:44:25 ip-10-104-20-31 systemd[1]: nomad.service: Unit entered failed state.
Jul 24 17:44:25 ip-10-104-20-31 systemd[1]: nomad.service: Failed with result 'exit-code'.
Jul 24 17:44:35 ip-10-104-20-31 systemd[1]: nomad.service: Service hold-off time over, scheduling restart.
Jul 24 17:44:35 ip-10-104-20-31 systemd[1]: Stopped nomad server.
Jul 24 17:44:35 ip-10-104-20-31 systemd[1]: Started nomad server.
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: Loaded configuration from /etc/nomad.d/server.hcl
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: ==> Starting Nomad agent...
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: ==> Nomad agent configuration:
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: Atlas: <disabled>
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: Client: false
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: Log Level: INFO
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: Region: global (DC: dc1)
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: Server: true
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: Version: 0.6.0rc1
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: ==> Nomad agent started! Log data will stream in below:
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] raft: Restored from snapshot 69-21032-1500793356023
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] raft: Initial configuration (index=12867): [{Suffrage:Voter ID:10.104.21.74:4647 Address:10.104.21.74:4647} {Suffrage:Voter ID:10.104.21.92:4647 Address:10.104.21.92:4647} {Suffrage:Voter ID:10.104.20.31:4647 Address:10.104.20.31:4647} {Suffrage:Voter ID:10.104.20.214:4647 Address:10.104.20.214:4647} {Suffrage:Voter ID:10.104.20.32:4647 Address:10.104.20.32:4647}]
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] serf: EventMemberJoin: ip-10-104-20-31.global 10.104.20.31
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35.490241 [INFO] nomad: starting 1 scheduling worker(s) for [service batch system _core]
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] raft: Node at 10.104.20.31:4647 [Follower] entering Follower state (Leader: "")
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] serf: Attempting re-join to previously known node: ip-10-104-20-214.global: 10.104.20.214:4648
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35.496115 [INFO] nomad: adding server ip-10-104-20-31.global (Addr: 10.104.20.31:4647) (DC: dc1)
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] serf: EventMemberJoin: ip-10-104-21-74.global 10.104.21.74
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] serf: EventMemberJoin: ip-10-104-20-214.global 10.104.20.214
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] serf: EventMemberJoin: ip-10-104-20-32.global 10.104.20.32
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [WARN] memberlist: Refuting an alive message
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] serf: EventMemberJoin: ip-10-104-21-92.global 10.104.21.92
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35.503595 [INFO] nomad: adding server ip-10-104-21-74.global (Addr: 10.104.21.74:4647) (DC: dc1)
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35.503855 [INFO] nomad: adding server ip-10-104-20-214.global (Addr: 10.104.20.214:4647) (DC: dc1)
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35.504090 [INFO] nomad: adding server ip-10-104-20-32.global (Addr: 10.104.20.32:4647) (DC: dc1)
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35.504317 [INFO] nomad: adding server ip-10-104-21-92.global (Addr: 10.104.21.92:4647) (DC: dc1)
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] serf: Attempting re-join to previously known node: ip-10-104-20-32.global: 10.104.20.32:4648
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [INFO] serf: Re-joined to previously known node: ip-10-104-20-32.global: 10.104.20.32:4648
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35.510453 [INFO] server.nomad: successfully contacted 2 Nomad Servers
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35.597080 [INFO] vault: renewing token in 35h59m59.999995944s
Jul 24 17:44:35 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:35 [WARN] raft: Failed to get previous log: 24360 log not found (last: 24356)
Jul 24 17:44:37 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:37 [WARN] raft: Rejecting vote request from 10.104.21.92:4647 since we have a leader: 10.104.20.32:4647
Jul 24 17:44:37 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:37 [WARN] raft: Heartbeat timeout from "10.104.20.32:4647" reached, starting election
Jul 24 17:44:37 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:37 [INFO] raft: Node at 10.104.20.31:4647 [Candidate] entering Candidate state in term 1422
Jul 24 17:44:37 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:37 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.104.21.74:4647 10.104.21.74:4647}: dial tcp 10.104.21.74:4647: getsockopt: connection refused
Jul 24 17:44:37 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:37 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.104.20.32:4647 10.104.20.32:4647}: dial tcp 10.104.20.32:4647: getsockopt: connection refused
Jul 24 17:44:37 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:37 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.104.20.214:4647 10.104.20.214:4647}: dial tcp 10.104.20.214:4647: getsockopt: connection refused
Jul 24 17:44:38 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:38 [INFO] raft: Node at 10.104.20.31:4647 [Follower] entering Follower state (Leader: "")
Jul 24 17:44:39 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:39 [INFO] serf: EventMemberUpdate: ip-10-104-20-214.global
Jul 24 17:44:40 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:40.239753 [ERR] http: Request /v1/status/peers, error: rpc error: failed to get conn: dial tcp 10.104.21.92:4647: getsockopt: connection refused
Jul 24 17:44:41 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:41 [WARN] raft: Heartbeat timeout from "10.104.21.92:4647" reached, starting election
Jul 24 17:44:41 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:41 [INFO] raft: Node at 10.104.20.31:4647 [Candidate] entering Candidate state in term 1425
Jul 24 17:44:41 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:41 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.104.21.92:4647 10.104.21.92:4647}: EOF
Jul 24 17:44:41 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:41 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.104.21.74:4647 10.104.21.74:4647}: dial tcp 10.104.21.74:4647: getsockopt: connection refused
Jul 24 17:44:41 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:41 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.104.20.32:4647 10.104.20.32:4647}: dial tcp 10.104.20.32:4647: getsockopt: connection refused
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: Duplicate RequestVote for same term: 1425
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] serf: EventMemberUpdate: ip-10-104-21-74.global
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [WARN] raft: Election timeout reached, restarting election
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: Node at 10.104.20.31:4647 [Candidate] entering Candidate state in term 1426
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.104.21.92:4647 10.104.21.92:4647}: dial tcp 10.104.21.92:4647: getsockopt: connection refused
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.104.20.32:4647 10.104.20.32:4647}: dial tcp 10.104.20.32:4647: getsockopt: connection refused
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: Election won. Tally: 3
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: Node at 10.104.20.31:4647 [Leader] entering Leader state
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: Added peer 10.104.21.74:4647, starting replication
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: Added peer 10.104.21.92:4647, starting replication
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: Added peer 10.104.20.214:4647, starting replication
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: Added peer 10.104.20.32:4647, starting replication
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42.738016 [INFO] nomad: cluster leadership acquired
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [ERR] raft: Failed to AppendEntries to {Voter 10.104.20.32:4647 10.104.20.32:4647}: dial tcp 10.104.20.32:4647: getsockopt: connection refused
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: pipelining replication to peer {Voter 10.104.20.214:4647 10.104.20.214:4647}
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [ERR] raft: Failed to AppendEntries to {Voter 10.104.21.92:4647 10.104.21.92:4647}: dial tcp 10.104.21.92:4647: getsockopt: connection refused
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [WARN] raft: AppendEntries to {Voter 10.104.21.74:4647 10.104.21.74:4647} rejected, sending older logs (next: 24361)
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [ERR] raft: Failed to AppendEntries to {Voter 10.104.20.32:4647 10.104.20.32:4647}: dial tcp 10.104.20.32:4647: getsockopt: connection refused
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [ERR] raft: Failed to AppendEntries to {Voter 10.104.21.92:4647 10.104.21.92:4647}: dial tcp 10.104.21.92:4647: getsockopt: connection refused
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [ERR] raft: Failed to AppendEntries to {Voter 10.104.20.32:4647 10.104.20.32:4647}: dial tcp 10.104.20.32:4647: getsockopt: connection refused
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [ERR] raft: Failed to AppendEntries to {Voter 10.104.21.92:4647 10.104.21.92:4647}: dial tcp 10.104.21.92:4647: getsockopt: connection refused
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: 2017/07/24 17:44:42 [INFO] raft: pipelining replication to peer {Voter 10.104.21.74:4647 10.104.21.74:4647}
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: panic: runtime error: slice bounds out of range
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: goroutine 42 [running]:
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/scheduler.(*allocReconciler).computeGroup(0xc4202087e0, 0xc420260ea0, 0x11, 0xc4202dede0, 0xc4202ded80)
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/scheduler/reconcile.go:393 +0x1592
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/scheduler.(*allocReconciler).Compute(0xc4202087e0, 0xc420179640)
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/scheduler/reconcile.go:182 +0x1f1
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/scheduler.(*GenericScheduler).computeJobAllocs(0xc42059ccf0, 0xc420422780, 0xc420447580)
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/scheduler/generic_sched.go:391 +0x3cc
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/scheduler.(*GenericScheduler).process(0xc42059ccf0, 0x24, 0x24, 0xc4207a57b0)
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/scheduler/generic_sched.go:230 +0x53e
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/scheduler.(*GenericScheduler).(github.com/hashicorp/nomad/scheduler.process)-fm(0xc4203b6400, 0xc4207a57b0, 0x4c41ae)
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/scheduler/generic_sched.go:131 +0x2a
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/scheduler.retryMax(0x5, 0xc42078f908, 0xc42078f918, 0xc, 0xffffffffffffff01)
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/scheduler/util.go:268 +0x42
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/scheduler.(*GenericScheduler).Process(0xc42059ccf0, 0xc420662c60, 0xc420433360, 0x1b3ec80)
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/scheduler/generic_sched.go:131 +0x2a0
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/nomad.(*nomadFSM).reconcileQueuedAllocations(0xc420176a80, 0x5af3, 0x0, 0x0)
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/nomad/fsm.go:919 +0x3e5
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/nomad.(*nomadFSM).applyReconcileSummaries(0xc420176a80, 0xc4203e85a1, 0x8, 0x8, 0x5af3, 0xc420726ea0, 0xc4204e5ee0)
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/nomad/fsm.go:526 +0x7e
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/nomad.(*nomadFSM).Apply(0xc420176a80, 0xc42073f9e0, 0x1b666c0, 0xed1082903)
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/nomad/fsm.go:153 +0x6b3
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(*Raft).runFSM.func1(0xc4203e85d0)
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/fsm.go:57 +0x189
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(*Raft).runFSM(0xc4200bf080)
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/fsm.go:120 +0x2d6
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(*Raft).(github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.runFSM)-fm()
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/api.go:505 +0x2a
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc4200bf080, 0xc420190c20)
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/state.go:142 +0x53
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft.(*raftState).goFunc
Jul 24 17:44:42 ip-10-104-20-31 nomad[700]: #011/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/raft/state.go:143 +0x66
Jul 24 17:44:42 ip-10-104-20-31 systemd[1]: nomad.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 24 17:44:42 ip-10-104-20-31 systemd[1]: nomad.service: Unit entered failed state.
Jul 24 17:44:42 ip-10-104-20-31 systemd[1]: nomad.service: Failed with result 'exit-code'.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment