Skip to content

Instantly share code, notes, and snippets.

@rsvalerio

rsvalerio/consul crash logs

Created September 21, 2020 12:31
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save rsvalerio/9f9726cd7cc43224564a1a3141fb7e8b to your computer and use it in GitHub Desktop.
Save rsvalerio/9f9726cd7cc43224564a1a3141fb7e8b to your computer and use it in GitHub Desktop.
Download ZIP
reference to consul issue
Raw
consul crash logs
2020-09-18T13:41:07.918Z 2020-09-18T13:41:07.655Z [INFO] agent.server.fsm: snapshot created: duration=73.995µs
2020-09-18T13:41:07.918Z 2020-09-18T13:41:07.655Z [INFO] snapshot: creating new snapshot: path=/consul/data/raft/snapshots/7808-882522967-1600436467655.tmp
2020-09-18T13:41:07.918Z 2020-09-18T13:41:07.655Z [INFO] agent.server.raft: starting snapshot up to: index=882522967
2020-09-18T13:41:07.919Z 2020-09-18T13:41:07.726Z [INFO] snapshot: reaping snapshot: path=/consul/data/raft/snapshots/7808-882488512-1600435239963
2020-09-18T13:41:07.919Z 2020-09-18T13:41:07.727Z [INFO] agent.server.raft: compacting logs: from=882496245 to=882512727
2020-09-18T13:41:07.919Z 2020-09-18T13:41:07.782Z [INFO] agent.server.raft: snapshot complete up to: index=882522967
2020-09-18T13:43:07.669Z github.com/hashicorp/go-immutable-radix.(*Iterator).Next(0xc0077183a0, 0xc006c7f260, 0x0, 0x0, 0xc0033fae40, 0x0, 0xffffffffffffffff)
2020-09-18T13:43:07.669Z 2020-09-18T13:43:07.365Z [INFO] agent.server: deregistering member: member=[nodeid_redacted] reason=left
2020-09-18T13:43:07.669Z 2020-09-18T13:43:07.365Z [INFO] agent.server.serf.lan: serf: EventMemberLeave: [nodeid_redacted]-ca5e72a6-0417 [ip_redacted]
2020-09-18T13:43:07.669Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x788578]
2020-09-18T13:43:07.669Z goroutine 61 [running]:
2020-09-18T13:43:07.669Z panic: runtime error: invalid memory address or nil pointer dereference
2020-09-18T13:43:07.670Z github.com/hashicorp/consul/agent/consul/state.(*Store).deleteServiceTxn(0xc00065b0e0, 0xc004b4f240, 0x349a48cc, 0xc0032af880, 0x33, 0xc000678180, 0x80, 0xc0005e28f0, 0x0, 0x0)
2020-09-18T13:43:07.670Z github.com/hashicorp/go-memdb.(*radixIterator).Next(0xc0033fae00, 0x349a48cc, 0x3192dcf)
2020-09-18T13:43:07.670Z /go/pkg/mod/github.com/hashicorp/go-memdb@v1.0.3/txn.go:669 +0x2e
2020-09-18T13:43:07.670Z github.com/hashicorp/consul/agent/consul/state.(*Store).deleteNodeTxn(0xc00065b0e0, 0xc004b4f240, 0x349a48cc, 0xc0032af880, 0x33, 0xb12e4c, 0xc005146c60)
2020-09-18T13:43:07.670Z /go/pkg/mod/github.com/hashicorp/go-immutable-radix@v1.2.0/iter.go:178 +0xa8
2020-09-18T13:43:07.670Z /home/circleci/project/consul/agent/consul/state/catalog.go:1511 +0xa50
2020-09-18T13:43:07.671Z github.com/hashicorp/consul/agent/consul/fsm.(*FSM).applyDeregister(0xc000185680, 0xc00246e901, 0x6c, 0xb2, 0x349a48cc, 0x0, 0x0)
2020-09-18T13:43:07.671Z github.com/hashicorp/consul/agent/consul/state.(*Store).DeleteNode(0xc00065b0e0, 0x349a48cc, 0xc0032af880, 0x33, 0x0, 0x0)
2020-09-18T13:43:07.671Z github.com/hashicorp/consul/agent/consul/fsm.New.func1(0xc00246e901, 0x6c, 0xb2, 0x349a48cc, 0x4, 0xc004671be8)
2020-09-18T13:43:07.671Z /home/circleci/project/consul/agent/consul/state/catalog.go:666 +0x5f6
2020-09-18T13:43:07.671Z /home/circleci/project/consul/agent/consul/fsm/commands_oss.go:78 +0x41c
2020-09-18T13:43:07.671Z /home/circleci/project/consul/agent/consul/state/catalog.go:598 +0xa4
2020-09-18T13:43:07.672Z /home/circleci/project/consul/agent/consul/fsm/fsm.go:120 +0x24b
2020-09-18T13:43:07.672Z github.com/hashicorp/raft.(*Raft).runFSM.func2(0xc00713ce00, 0x1, 0x40)
2020-09-18T13:43:07.672Z /go/pkg/mod/github.com/hashicorp/raft@v1.1.2/fsm.go:113 +0x75
2020-09-18T13:43:07.672Z github.com/hashicorp/go-raftchunking.(*ChunkingFSM).Apply(0xc00084d170, 0xc003824c80, 0x539cfa0, 0xbfd14c3aa8e9e44f)
2020-09-18T13:43:07.672Z /go/pkg/mod/github.com/hashicorp/raft@v1.1.2/fsm.go:90 +0x2c1
2020-09-18T13:43:07.672Z github.com/hashicorp/raft.(*Raft).runFSM.func1(0xc00319d820)
2020-09-18T13:43:07.672Z github.com/hashicorp/consul/agent/consul/fsm.(*FSM).Apply(0xc000185680, 0xc003824c80, 0x0, 0x0)
2020-09-18T13:43:07.672Z /home/circleci/project/consul/agent/consul/fsm/fsm.go:85 +0x56
2020-09-18T13:43:07.672Z /go/pkg/mod/github.com/hashicorp/go-raftchunking@v0.6.1/fsm.go:66 +0x5b
2020-09-18T13:43:07.673Z created by github.com/hashicorp/raft.(*raftState).goFunc
2020-09-18T13:43:07.673Z /go/pkg/mod/github.com/hashicorp/raft@v1.1.2/fsm.go:219 +0x42f
2020-09-18T13:43:07.673Z /go/pkg/mod/github.com/hashicorp/raft@v1.1.2/state.go:146 +0x55
2020-09-18T13:43:07.673Z github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc0005e6300, 0xc0014fb3d0)
2020-09-18T13:43:07.673Z github.com/hashicorp/raft.(*Raft).runFSM(0xc0005e6300)
2020-09-18T13:43:07.673Z /go/pkg/mod/github.com/hashicorp/raft@v1.1.2/state.go:144 +0x66
2020-09-18T13:43:08.618Z Server: true (Bootstrap: false)
2020-09-18T13:43:08.618Z Err:510
2020-09-18T13:43:08.618Z Node name: '[nodeid_redacted]'
2020-09-18T13:43:08.618Z Node ID: 'c486b0c3-87f6-7880-8401-cd5e26aeb3de'
2020-09-18T13:43:08.618Z Datacenter: '[dc_redacted]' (Segment: '<all>')
2020-09-18T13:43:08.618Z Client Addr: [0.0.0.0] (HTTP: 28500, HTTPS: -1, gRPC: 28400, DNS: 28600)
2020-09-18T13:43:08.618Z bootstrap_expect > 0: expecting 3 servers
2020-09-18T13:43:08.618Z Version: '1.8.1'
2020-09-18T13:43:08.618Z Cluster Addr: 10.5.128.46 (LAN: 28301, WAN: 28302)
2020-09-18T13:43:08.619Z 2020-09-18T13:43:08.613Z [INFO] agent.server.raft: restored from snapshot: id=7808-882522967-1600436467655
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment