Skip to content

Instantly share code, notes, and snippets.

@evanphx

evanphx/a_setup.txt

Last active September 4, 2015 15:45
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save evanphx/58c3a88c647a7f229de6 to your computer and use it in GitHub Desktop.
Save evanphx/58c3a88c647a7f229de6 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
a_setup.txt
52.28.77.117$ kill -SUSPEND $consul
52.88.80.220$ curl localhost:8500/v1/kv/hello
Raw
output.txt
2015/09/04 15:42:37 [WARN] raft: Heartbeat timeout reached, starting election
2015/09/04 15:42:37 [INFO] raft: Node at 52.88.80.220:8300 [Candidate] entering Candidate state
2015/09/04 15:42:38 [INFO] raft: Duplicate RequestVote for same term: 3
2015/09/04 15:42:46 [WARN] raft: Election timeout reached, restarting election
2015/09/04 15:42:46 [INFO] raft: Node at 52.88.80.220:8300 [Candidate] entering Candidate state
2015/09/04 15:42:46 [INFO] raft: Election won. Tally: 2
2015/09/04 15:42:46 [INFO] raft: Node at 52.88.80.220:8300 [Leader] entering Leader state
2015/09/04 15:42:46 [INFO] consul: cluster leadership acquired
2015/09/04 15:42:46 [INFO] consul: New leader elected: ip-172-31-33-205
2015/09/04 15:42:47 [INFO] raft: pipelining replication to peer 54.153.213.7:8300
2015/09/04 15:42:47 [ERR] raft: Failed to make RequestVote RPC to 52.28.77.117:8300: read tcp 52.28.77.117:8300: i/o timeout
2015/09/04 15:42:49 [WARN] raft: Failed to contact 52.28.77.117:8300 in 2.505105692s
2015/09/04 15:42:49 [ERR] memberlist: Failed TCP fallback ping: read tcp 52.28.77.117:8301: i/o timeout
2015/09/04 15:42:49 [INFO] memberlist: Suspect ip-172-31-10-54 has failed, no acks received
2015/09/04 15:42:51 [WARN] raft: Failed to contact 52.28.77.117:8300 in 4.90351528s
2015/09/04 15:42:54 [WARN] raft: Failed to contact 52.28.77.117:8300 in 7.305664598s
2015/09/04 15:42:56 [ERR] raft: Failed to make RequestVote RPC to 52.28.77.117:8300: read tcp 52.28.77.117:8300: i/o timeout
2015/09/04 15:42:57 [ERR] raft: Failed to AppendEntries to 52.28.77.117:8300: read tcp 52.28.77.117:8300: i/o timeout
2015/09/04 15:42:57 [ERR] raft: Failed to heartbeat to 52.28.77.117:8300: read tcp 52.28.77.117:8300: i/o timeout
2015/09/04 15:42:59 [ERR] memberlist: Failed TCP fallback ping: read tcp 52.28.77.117:8301: i/o timeout
2015/09/04 15:42:59 [INFO] memberlist: Suspect ip-172-31-10-54 has failed, no acks received
2015/09/04 15:43:04 [ERR] memberlist: Failed TCP fallback ping: read tcp 52.28.77.117:8301: i/o timeout
2015/09/04 15:43:04 [INFO] memberlist: Suspect ip-172-31-10-54 has failed, no acks received
2015/09/04 15:43:07 [ERR] raft: Failed to AppendEntries to 52.28.77.117:8300: read tcp 52.28.77.117:8300: i/o timeout
2015/09/04 15:43:08 [ERR] raft: Failed to heartbeat to 52.28.77.117:8300: read tcp 52.28.77.117:8300: i/o timeout
2015/09/04 15:43:13 [INFO] memberlist: Marking ip-172-31-10-54 as failed, suspect timeout reached
2015/09/04 15:43:13 [INFO] serf: EventMemberFailed: ip-172-31-10-54 52.28.77.117
2015/09/04 15:43:13 [INFO] consul: removing server ip-172-31-10-54 (Addr: 52.28.77.117:8300) (DC: dc1)
2015/09/04 15:43:13 [INFO] consul: member 'ip-172-31-10-54' failed, marking health critical
2015/09/04 15:43:17 [ERR] raft: Failed to AppendEntries to 52.28.77.117:8300: read tcp 52.28.77.117:8300: i/o timeout
2015/09/04 15:43:19 [ERR] raft: Failed to heartbeat to 52.28.77.117:8300: read tcp 52.28.77.117:8300: i/o timeout
SIGABRT: abort
PC=0x4507d1
goroutine 0 [idle]:
runtime.futex(0x11851d8, 0x0, 0x0, 0x0, 0x0, 0x1184480, 0x1, 0x41dd7f, 0x41df9e, 0x11851d8, ...)
/usr/local/go/src/runtime/sys_linux_amd64.s:277 +0x21
runtime.futexsleep(0x11851d8, 0x0, 0xffffffffffffffff)
/usr/local/go/src/runtime/os_linux.c:49 +0x47
runtime.notesleep(0x11851d8)
/usr/local/go/src/runtime/lock_futex.go:145 +0xae
stopm()
/usr/local/go/src/runtime/proc.c:1178 +0x119
findrunnable(0xc208012000)
/usr/local/go/src/runtime/proc.c:1487 +0x562
schedule()
/usr/local/go/src/runtime/proc.c:1575 +0x151
runtime.park_m(0xc208237560)
/usr/local/go/src/runtime/proc.c:1654 +0x113
runtime.mcall(0x446c14)
/usr/local/go/src/runtime/asm_amd64.s:186 +0x5a
goroutine 1 [select, 602 minutes]:
github.com/hashicorp/consul/command/agent.(*Command).handleSignals(0xc2080ca0c0, 0xc208047080, 0xc208162c60, 0xc208162cc0, 0x1)
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/command.go:778 +0xa54
github.com/hashicorp/consul/command/agent.(*Command).Run(0xc2080ca0c0, 0xc20800a020, 0x4, 0x4, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/command.go:767 +0x252a
github.com/mitchellh/cli.(*CLI).Run(0xc208045130, 0xc20802b8f0, 0x0, 0x0)
/home/ubuntu/go/src/github.com/mitchellh/cli/cli.go:112 +0x5be
main.realMain(0x4120ff)
/home/ubuntu/go/src/github.com/hashicorp/consul/main.go:40 +0x23b
main.main()
/home/ubuntu/go/src/github.com/hashicorp/consul/main.go:12 +0x1f
goroutine 5 [syscall, 602 minutes]:
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
/usr/local/go/src/os/signal/signal_unix.go:27 +0x35
goroutine 14 [select]:
github.com/hashicorp/raft.(*Raft).leaderLoop(0xc20810a000)
/home/ubuntu/go/src/github.com/hashicorp/raft/raft.go:826 +0xc7c
github.com/hashicorp/raft.(*Raft).runLeader(0xc20810a000)
/home/ubuntu/go/src/github.com/hashicorp/raft/raft.go:798 +0x74e
github.com/hashicorp/raft.(*Raft).run(0xc20810a000)
/home/ubuntu/go/src/github.com/hashicorp/raft/raft.go:595 +0xc9
github.com/hashicorp/raft.*Raft.(github.com/hashicorp/raft.run)·fm()
/home/ubuntu/go/src/github.com/hashicorp/raft/raft.go:245 +0x27
github.com/hashicorp/raft.func·011()
/home/ubuntu/go/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
/home/ubuntu/go/src/github.com/hashicorp/raft/state.go:153 +0xe3
goroutine 8 [select, 602 minutes]:
github.com/armon/go-metrics.(*InmemSignal).run(0xc2080da000)
/home/ubuntu/go/src/github.com/armon/go-metrics/inmem_signal.go:63 +0xf1
created by github.com/armon/go-metrics.NewInmemSignal
/home/ubuntu/go/src/github.com/armon/go-metrics/inmem_signal.go:37 +0x258
goroutine 9 [sleep]:
github.com/armon/go-metrics.(*Metrics).collectStats(0xc20803e190)
/home/ubuntu/go/src/github.com/armon/go-metrics/metrics.go:67 +0x2c
created by github.com/armon/go-metrics.New
/home/ubuntu/go/src/github.com/armon/go-metrics/start.go:61 +0xe8
goroutine 13 [select, 602 minutes]:
github.com/hashicorp/consul/consul.(*RaftLayer).Accept(0xc2080da540, 0x0, 0x0, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/raft_rpc.go:57 +0x1b8
github.com/hashicorp/raft.(*NetworkTransport).listen(0xc2080103f0)
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:346 +0x50
created by github.com/hashicorp/raft.NewNetworkTransport
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:138 +0x2c4
goroutine 11 [select]:
github.com/hashicorp/consul/consul.(*ConnPool).reap(0xc20803ef50)
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/pool.go:410 +0x4bb
created by github.com/hashicorp/consul/consul.NewPool
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/pool.go:173 +0x1b0
goroutine 17 [syscall, 602 minutes, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1
goroutine 15 [select]:
github.com/hashicorp/raft.(*Raft).runFSM(0xc20810a000)
/home/ubuntu/go/src/github.com/hashicorp/raft/raft.go:502 +0x1043
github.com/hashicorp/raft.*Raft.(github.com/hashicorp/raft.runFSM)·fm()
/home/ubuntu/go/src/github.com/hashicorp/raft/raft.go:246 +0x27
github.com/hashicorp/raft.func·011()
/home/ubuntu/go/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
/home/ubuntu/go/src/github.com/hashicorp/raft/state.go:153 +0xe3
goroutine 16 [select, 1 minutes]:
github.com/hashicorp/raft.(*Raft).runSnapshots(0xc20810a000)
/home/ubuntu/go/src/github.com/hashicorp/raft/raft.go:1613 +0x466
github.com/hashicorp/raft.*Raft.(github.com/hashicorp/raft.runSnapshots)·fm()
/home/ubuntu/go/src/github.com/hashicorp/raft/raft.go:247 +0x27
github.com/hashicorp/raft.func·011()
/home/ubuntu/go/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
/home/ubuntu/go/src/github.com/hashicorp/raft/state.go:153 +0xe3
goroutine 18 [select]:
github.com/hashicorp/consul/consul.(*Server).monitorLeadership(0xc20800c2c0)
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/leader.go:33 +0x1fd
created by github.com/hashicorp/consul/consul.(*Server).setupRaft
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/server.go:394 +0xc8b
goroutine 19 [select]:
github.com/hashicorp/serf/serf.(*serfQueries).stream(0xc2080ce000)
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/internal_query.go:80 +0x288
created by github.com/hashicorp/serf/serf.newSerfQueries
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/internal_query.go:73 +0x137
goroutine 20 [select]:
github.com/hashicorp/serf/serf.(*Snapshotter).stream(0xc208044000)
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/snapshot.go:175 +0xa2b
created by github.com/hashicorp/serf/serf.NewSnapshotter
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/snapshot.go:123 +0x6ec
goroutine 21 [IO wait]:
net.(*pollDesc).Wait(0xc208010220, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010220, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc2080101c0, 0x0, 0x7f9e973fae18, 0xc208343f30)
/usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc20803a050, 0xc20803a650, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
github.com/hashicorp/memberlist.(*Memberlist).tcpListen(0xc2080a61c0)
/home/ubuntu/go/src/github.com/hashicorp/memberlist/net.go:188 +0x2f
created by github.com/hashicorp/memberlist.newMemberlist
/home/ubuntu/go/src/github.com/hashicorp/memberlist/memberlist.go:126 +0xeec
goroutine 22 [IO wait]:
net.(*pollDesc).Wait(0xc208010290, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010290, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).readFrom(0xc208010230, 0xc20840c000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x7f9e973fae18, 0xc20836a400)
/usr/local/go/src/net/fd_unix.go:269 +0x4a1
net.(*UDPConn).ReadFromUDP(0xc20803a058, 0xc20840c000, 0x10000, 0x10000, 0x24, 0x0, 0x0, 0x0)
/usr/local/go/src/net/udpsock_posix.go:67 +0x124
net.(*UDPConn).ReadFrom(0xc20803a058, 0xc20840c000, 0x10000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/udpsock_posix.go:82 +0x12e
github.com/hashicorp/memberlist.(*Memberlist).udpListen(0xc2080a61c0)
/home/ubuntu/go/src/github.com/hashicorp/memberlist/net.go:283 +0x2f2
created by github.com/hashicorp/memberlist.newMemberlist
/home/ubuntu/go/src/github.com/hashicorp/memberlist/memberlist.go:127 +0xf0c
goroutine 23 [select]:
github.com/hashicorp/memberlist.(*Memberlist).udpHandler(0xc2080a61c0)
/home/ubuntu/go/src/github.com/hashicorp/memberlist/net.go:369 +0x36d
created by github.com/hashicorp/memberlist.newMemberlist
/home/ubuntu/go/src/github.com/hashicorp/memberlist/memberlist.go:128 +0xf2c
goroutine 24 [select]:
github.com/hashicorp/memberlist.(*Memberlist).triggerFunc(0xc2080a61c0, 0x12a05f200, 0xc20800b2c0, 0xc20800b260, 0xc20802bc70)
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:105 +0x151
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:71 +0x18d
goroutine 25 [select]:
github.com/hashicorp/memberlist.(*Memberlist).pushPullTrigger(0xc2080a61c0, 0xc20800b260)
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:132 +0x1a8
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:77 +0x2d3
goroutine 26 [select]:
github.com/hashicorp/memberlist.(*Memberlist).triggerFunc(0xc2080a61c0, 0x1dcd6500, 0xc20800b320, 0xc20800b260, 0xc20802bc80)
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:105 +0x151
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:83 +0x3bb
goroutine 27 [select]:
github.com/hashicorp/serf/serf.(*Serf).handleReap(0xc208036680)
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:1341 +0x281
created by github.com/hashicorp/serf/serf.Create
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:372 +0x1bfd
goroutine 28 [select]:
github.com/hashicorp/serf/serf.(*Serf).handleReconnect(0xc208036680)
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:1357 +0x106
created by github.com/hashicorp/serf/serf.Create
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:373 +0x1c1d
goroutine 29 [select]:
github.com/hashicorp/serf/serf.(*Serf).checkQueueDepth(0xc208036680, 0xbc22b0, 0x6, 0xc2080ce1e0)
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:1447 +0x52e
created by github.com/hashicorp/serf/serf.Create
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:374 +0x1c5c
goroutine 30 [select]:
github.com/hashicorp/serf/serf.(*Serf).checkQueueDepth(0xc208036680, 0xbc0610, 0x5, 0xc2080ce210)
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:1447 +0x52e
created by github.com/hashicorp/serf/serf.Create
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:375 +0x1c9e
goroutine 31 [select]:
github.com/hashicorp/serf/serf.(*Serf).checkQueueDepth(0xc208036680, 0xbc6110, 0x5, 0xc2080ce240)
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:1447 +0x52e
created by github.com/hashicorp/serf/serf.Create
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:376 +0x1ce0
goroutine 63 [IO wait]:
net.(*pollDesc).Wait(0xc2081c8b50, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2081c8b50, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc2081c8af0, 0xc2081e5000, 0x1000, 0x1000, 0x0, 0x7f9e973fae18, 0xc208318f40)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20803a5f0, 0xc2081e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
bufio.(*Reader).fill(0xc208163ec0)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadByte(0xc208163ec0, 0x300000002, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:196 +0x86
github.com/hashicorp/raft.(*NetworkTransport).handleCommand(0xc2080103f0, 0xc208163ec0, 0xc208163f20, 0xc2081c8c40, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:386 +0x4e
github.com/hashicorp/raft.(*NetworkTransport).handleConn(0xc2080103f0, 0x7f9e973fd720, 0xc20803a5f0)
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:370 +0x278
created by github.com/hashicorp/raft.(*NetworkTransport).listen
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:357 +0x34b
goroutine 33 [select]:
github.com/hashicorp/consul/consul.(*Server).lanEventHandler(0xc20800c2c0)
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/serf.go:37 +0x433
created by github.com/hashicorp/consul/consul.NewServer
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/server.go:249 +0xf83
goroutine 34 [select, 602 minutes]:
github.com/hashicorp/serf/serf.(*serfQueries).stream(0xc2080cee70)
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/internal_query.go:80 +0x288
created by github.com/hashicorp/serf/serf.newSerfQueries
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/internal_query.go:73 +0x137
goroutine 35 [select]:
github.com/hashicorp/serf/serf.(*Snapshotter).stream(0xc208044160)
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/snapshot.go:175 +0xa2b
created by github.com/hashicorp/serf/serf.NewSnapshotter
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/snapshot.go:123 +0x6ec
goroutine 36 [IO wait, 602 minutes]:
net.(*pollDesc).Wait(0xc208010d80, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010d80, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc208010d20, 0x0, 0x7f9e973fae18, 0xc2081528c0)
/usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc20803a1a0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
github.com/hashicorp/memberlist.(*Memberlist).tcpListen(0xc2080a62a0)
/home/ubuntu/go/src/github.com/hashicorp/memberlist/net.go:188 +0x2f
created by github.com/hashicorp/memberlist.newMemberlist
/home/ubuntu/go/src/github.com/hashicorp/memberlist/memberlist.go:126 +0xeec
goroutine 37 [IO wait, 602 minutes]:
net.(*pollDesc).Wait(0xc208010df0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010df0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).readFrom(0xc208010d90, 0xc208174000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x7f9e973fae18, 0xc2081528f0)
/usr/local/go/src/net/fd_unix.go:269 +0x4a1
net.(*UDPConn).ReadFromUDP(0xc20803a1a8, 0xc208174000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/udpsock_posix.go:67 +0x124
net.(*UDPConn).ReadFrom(0xc20803a1a8, 0xc208174000, 0x10000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/udpsock_posix.go:82 +0x12e
github.com/hashicorp/memberlist.(*Memberlist).udpListen(0xc2080a62a0)
/home/ubuntu/go/src/github.com/hashicorp/memberlist/net.go:283 +0x2f2
created by github.com/hashicorp/memberlist.newMemberlist
/home/ubuntu/go/src/github.com/hashicorp/memberlist/memberlist.go:127 +0xf0c
goroutine 38 [select, 602 minutes]:
github.com/hashicorp/memberlist.(*Memberlist).udpHandler(0xc2080a62a0)
/home/ubuntu/go/src/github.com/hashicorp/memberlist/net.go:369 +0x36d
created by github.com/hashicorp/memberlist.newMemberlist
/home/ubuntu/go/src/github.com/hashicorp/memberlist/memberlist.go:128 +0xf2c
goroutine 39 [select]:
github.com/hashicorp/memberlist.(*Memberlist).triggerFunc(0xc2080a62a0, 0x12a05f200, 0xc20800ba40, 0xc20800b9e0, 0xc208152220)
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:105 +0x151
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:71 +0x18d
goroutine 40 [select]:
github.com/hashicorp/memberlist.(*Memberlist).pushPullTrigger(0xc2080a62a0, 0xc20800b9e0)
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:132 +0x1a8
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:77 +0x2d3
goroutine 41 [select]:
github.com/hashicorp/memberlist.(*Memberlist).triggerFunc(0xc2080a62a0, 0x1dcd6500, 0xc20800baa0, 0xc20800b9e0, 0xc208152230)
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:105 +0x151
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:83 +0x3bb
goroutine 42 [select]:
github.com/hashicorp/serf/serf.(*Serf).handleReap(0xc208036820)
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:1341 +0x281
created by github.com/hashicorp/serf/serf.Create
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:372 +0x1bfd
goroutine 43 [select]:
github.com/hashicorp/serf/serf.(*Serf).handleReconnect(0xc208036820)
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:1357 +0x106
created by github.com/hashicorp/serf/serf.Create
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:373 +0x1c1d
goroutine 44 [select]:
github.com/hashicorp/serf/serf.(*Serf).checkQueueDepth(0xc208036820, 0xbc22b0, 0x6, 0xc2080cef90)
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:1447 +0x52e
created by github.com/hashicorp/serf/serf.Create
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:374 +0x1c5c
goroutine 45 [select]:
github.com/hashicorp/serf/serf.(*Serf).checkQueueDepth(0xc208036820, 0xbc0610, 0x5, 0xc2080cefc0)
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:1447 +0x52e
created by github.com/hashicorp/serf/serf.Create
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:375 +0x1c9e
goroutine 46 [select]:
github.com/hashicorp/serf/serf.(*Serf).checkQueueDepth(0xc208036820, 0xbc6110, 0x5, 0xc2080ceff0)
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:1447 +0x52e
created by github.com/hashicorp/serf/serf.Create
/home/ubuntu/go/src/github.com/hashicorp/serf/serf/serf.go:376 +0x1ce0
goroutine 1729 [chan send, 55 minutes]:
github.com/hashicorp/memberlist.func·003()
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:285 +0x23e
created by github.com/hashicorp/memberlist.(*Memberlist).probeNode
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:287 +0xe88
goroutine 48 [select, 602 minutes]:
github.com/hashicorp/consul/consul.(*Server).wanEventHandler(0xc20800c2c0)
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/serf.go:67 +0x300
created by github.com/hashicorp/consul/consul.NewServer
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/server.go:258 +0x114f
goroutine 49 [IO wait, 602 minutes]:
net.(*pollDesc).Wait(0xc2080103e0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2080103e0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc208010380, 0x0, 0x7f9e973fae18, 0xc20821e780)
/usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc20803a150, 0x696474, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net.(*TCPListener).Accept(0xc20803a150, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:244 +0x4c
github.com/hashicorp/consul/consul.(*Server).listen(0xc20800c2c0)
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/rpc.go:57 +0x56
created by github.com/hashicorp/consul/consul.NewServer
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/server.go:261 +0x1169
goroutine 50 [select]:
github.com/hashicorp/consul/consul.(*Server).sessionStats(0xc20800c2c0)
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/session_ttl.go:152 +0x1e3
created by github.com/hashicorp/consul/consul.NewServer
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/server.go:264 +0x1183
goroutine 51 [select, 602 minutes]:
github.com/hashicorp/consul/command/agent.(*Agent).handleEvents(0xc20800c160)
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/user_event.go:107 +0x2cc
created by github.com/hashicorp/consul/command/agent.Create
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/agent.go:191 +0xbf6
goroutine 52 [IO wait, 602 minutes]:
net.(*pollDesc).Wait(0xc208011170, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208011170, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc208011110, 0x0, 0x7f9e973fae18, 0xc208152938)
/usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc20803a1e8, 0x0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net.(*TCPListener).Accept(0xc20803a1e8, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:244 +0x4c
github.com/hashicorp/consul/command/agent.(*AgentRPC).listen(0xc20803f720)
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/rpc.go:289 +0x59
created by github.com/hashicorp/consul/command/agent.NewAgentRPC
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/rpc.go:257 +0x2fb
goroutine 53 [IO wait]:
net.(*pollDesc).Wait(0xc2080111e0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2080111e0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc208011180, 0x0, 0x7f9e973fae18, 0xc208319130)
/usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc20803a1f0, 0x64750e, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
github.com/hashicorp/consul/command/agent.tcpKeepAliveListener.Accept(0xc20803a1f0, 0x0, 0x0, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/http.go:177 +0x4c
net/http.(*Server).Serve(0xc208162a20, 0x7f9e973fd508, 0xc20803a1f0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:1728 +0x92
net/http.Serve(0x7f9e973fd508, 0xc20803a1f0, 0x7f9e973fd540, 0xc2080cf710, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:1606 +0xa1
created by github.com/hashicorp/consul/command/agent.NewHTTPServers
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/http.go:141 +0x18da
goroutine 54 [syscall, 602 minutes]:
syscall.Syscall(0x2f, 0x13, 0xc2080fd9c8, 0x0, 0x0, 0x7f9e973e9000, 0x42c6d8)
/usr/local/go/src/syscall/asm_linux_amd64.s:21 +0x5
syscall.recvmsg(0x13, 0xc2080fd9c8, 0x0, 0x42ba9c, 0x0, 0x0)
/usr/local/go/src/syscall/zsyscall_linux_amd64.go:1933 +0x57
syscall.Recvmsg(0x13, 0xc208184000, 0xffff, 0xffff, 0xc2080ce030, 0x28, 0x28, 0x0, 0x7f9e973e9000, 0x0, ...)
/usr/local/go/src/syscall/syscall_linux.go:519 +0x19b
net.(*netFD).readMsg(0xc2080112d0, 0xc208184000, 0xffff, 0xffff, 0xc2080ce030, 0x28, 0x28, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:292 +0x42b
net.(*UDPConn).ReadMsgUDP(0xc20803a5a8, 0xc208184000, 0xffff, 0xffff, 0xc2080ce030, 0x28, 0x28, 0x18b40c9a, 0x7f9e973e9000, 0x41ee31, ...)
/usr/local/go/src/net/udpsock_posix.go:96 +0x17f
github.com/miekg/dns.ReadFromSessionUDP(0xc20803a5a8, 0xc208184000, 0xffff, 0xffff, 0xffff, 0xffff, 0x0, 0x0)
/home/ubuntu/go/src/github.com/miekg/dns/udp.go:47 +0xca
github.com/miekg/dns.(*Server).readUDP(0xc2080301e0, 0xc20803a5a8, 0x77359400, 0x0, 0x0, 0x0, 0xb7af00, 0x0, 0x0)
/home/ubuntu/go/src/github.com/miekg/dns/server.go:615 +0x14e
github.com/miekg/dns.(*defaultReader).ReadUDP(0xc20803a5c0, 0xc20803a5a8, 0x77359400, 0x0, 0x0, 0x0, 0x118fd80, 0x0, 0x0)
/home/ubuntu/go/src/github.com/miekg/dns/server.go:230 +0x72
github.com/miekg/dns.(*Server).serveUDP(0xc2080301e0, 0xc20803a5a8, 0x0, 0x0)
/home/ubuntu/go/src/github.com/miekg/dns/server.go:485 +0x1ea
github.com/miekg/dns.(*Server).ListenAndServe(0xc2080301e0, 0x0, 0x0)
/home/ubuntu/go/src/github.com/miekg/dns/server.go:328 +0x81d
github.com/hashicorp/consul/command/agent.func·014()
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/dns.go:107 +0x44
created by github.com/hashicorp/consul/command/agent.NewDNSServer
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/dns.go:111 +0xcdf
goroutine 55 [IO wait, 602 minutes]:
net.(*pollDesc).Wait(0xc208010300, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010300, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc2080102a0, 0x0, 0x7f9e973fae18, 0xc208152010)
/usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc20803a0c8, 0xc2080302d0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
github.com/miekg/dns.(*Server).serveTCP(0xc2080302d0, 0xc20803a0c8, 0x0, 0x0)
/home/ubuntu/go/src/github.com/miekg/dns/server.go:446 +0x1bb
github.com/miekg/dns.(*Server).ListenAndServe(0xc2080302d0, 0x0, 0x0)
/home/ubuntu/go/src/github.com/miekg/dns/server.go:313 +0x462
github.com/hashicorp/consul/command/agent.func·015()
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/dns.go:115 +0x44
created by github.com/hashicorp/consul/command/agent.NewDNSServer
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/dns.go:119 +0xddf
goroutine 57 [select, 602 minutes]:
github.com/hashicorp/go-checkpoint.func·001()
/home/ubuntu/go/src/github.com/hashicorp/go-checkpoint/checkpoint.go:188 +0x154
created by github.com/hashicorp/go-checkpoint.CheckInterval
/home/ubuntu/go/src/github.com/hashicorp/go-checkpoint/checkpoint.go:196 +0x1d5
goroutine 59 [select, 1 minutes]:
github.com/hashicorp/consul/command/agent.(*localState).antiEntropy(0xc20800c190, 0xc20800a480)
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/local.go:323 +0x33d
created by github.com/hashicorp/consul/command/agent.(*Agent).StartSync
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/agent.go:545 +0x4a
goroutine 67 [IO wait]:
net.(*pollDesc).Wait(0xc2082825a0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2082825a0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208282540, 0xc208145000, 0x1000, 0x1000, 0x0, 0x7f9e973fae18, 0xc208343868)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20803a338, 0xc208145000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
bufio.(*Reader).fill(0xc208284a20)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadByte(0xc208284a20, 0x300000002, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:196 +0x86
github.com/hashicorp/raft.(*NetworkTransport).handleCommand(0xc2080103f0, 0xc208284a20, 0xc208284a80, 0xc208282620, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:386 +0x4e
github.com/hashicorp/raft.(*NetworkTransport).handleConn(0xc2080103f0, 0x7f9e973fd720, 0xc20803a338)
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:370 +0x278
created by github.com/hashicorp/raft.(*NetworkTransport).listen
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:357 +0x34b
goroutine 65 [IO wait]:
net.(*pollDesc).Wait(0xc208219560, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208219560, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208219500, 0xc208224000, 0x1000, 0x1000, 0x0, 0x7f9e973fae18, 0xc208319bc8)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20803a7f0, 0xc208224000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
bufio.(*Reader).fill(0xc208214f60)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadByte(0xc208214f60, 0x300000002, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:196 +0x86
github.com/hashicorp/raft.(*NetworkTransport).handleCommand(0xc2080103f0, 0xc208214f60, 0xc208214fc0, 0xc2082195e0, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:386 +0x4e
github.com/hashicorp/raft.(*NetworkTransport).handleConn(0xc2080103f0, 0x7f9e973fd720, 0xc20803a7f0)
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:370 +0x278
created by github.com/hashicorp/raft.(*NetworkTransport).listen
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:357 +0x34b
goroutine 68 [IO wait, 1 minutes]:
net.(*pollDesc).Wait(0xc208282760, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208282760, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208282700, 0xc208068000, 0x1000, 0x1000, 0x0, 0x7f9e973fae18, 0xc208353540)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20803a578, 0xc208068000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
bufio.(*Reader).fill(0xc2082858c0)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Read(0xc2082858c0, 0xc20821efc0, 0xc, 0xc, 0xc208353510, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:174 +0x26c
io.ReadAtLeast(0x7f9e973fb350, 0xc2082858c0, 0xc20821efc0, 0xc, 0xc, 0xc, 0x0, 0x0, 0x0)
/usr/local/go/src/io/io.go:298 +0xf1
io.ReadFull(0x7f9e973fb350, 0xc2082858c0, 0xc20821efc0, 0xc, 0xc, 0xc, 0x0, 0x0)
/usr/local/go/src/io/io.go:316 +0x6d
github.com/hashicorp/yamux.(*Session).recvLoop(0xc2080451e0, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/yamux/session.go:398 +0x10d
github.com/hashicorp/yamux.(*Session).recv(0xc2080451e0)
/home/ubuntu/go/src/github.com/hashicorp/yamux/session.go:386 +0x28
created by github.com/hashicorp/yamux.newSession
/home/ubuntu/go/src/github.com/hashicorp/yamux/session.go:104 +0x499
goroutine 69 [select]:
github.com/hashicorp/yamux.(*Session).send(0xc2080451e0)
/home/ubuntu/go/src/github.com/hashicorp/yamux/session.go:348 +0x554
created by github.com/hashicorp/yamux.newSession
/home/ubuntu/go/src/github.com/hashicorp/yamux/session.go:105 +0x4b3
goroutine 70 [select]:
github.com/hashicorp/yamux.(*Session).Ping(0xc2080451e0, 0xc208285aa0, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/yamux/session.go:273 +0x388
github.com/hashicorp/yamux.(*Session).keepalive(0xc2080451e0)
/home/ubuntu/go/src/github.com/hashicorp/yamux/session.go:289 +0xc0
created by github.com/hashicorp/yamux.newSession
/home/ubuntu/go/src/github.com/hashicorp/yamux/session.go:107 +0x4de
goroutine 71 [select, 1 minutes]:
github.com/hashicorp/yamux.(*Stream).Read(0xc208070000, 0xc208281000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/yamux/stream.go:125 +0x3ab
bufio.(*Reader).fill(0xc208285da0)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadByte(0xc208285da0, 0x7a7b45, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:196 +0x86
github.com/hashicorp/go-msgpack/codec.(*ioDecReader).readn1(0xc20826db00, 0x979020)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/decode.go:90 +0x45
github.com/hashicorp/go-msgpack/codec.(*msgpackDecDriver).initReadNext(0xc20821c900)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/msgpack.go:540 +0x48
github.com/hashicorp/go-msgpack/codec.(*Decoder).decode(0xc208285e00, 0x9860a0, 0xc208220090)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/decode.go:635 +0x59
github.com/hashicorp/go-msgpack/codec.(*Decoder).Decode(0xc208285e00, 0x9860a0, 0xc208220090, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/decode.go:630 +0x7f
github.com/hashicorp/go-msgpack/codec.(*rpcCodec).read(0xc2082577c0, 0x9860a0, 0xc208220090, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/rpc.go:88 +0xd9
github.com/hashicorp/go-msgpack/codec.(*goRpcCodec).ReadResponseHeader(0xc2082577c0, 0xc208220090, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/rpc.go:123 +0x61
net/rpc.(*Client).input(0xc208285e60)
/usr/local/go/src/net/rpc/client.go:109 +0xb6
created by net/rpc.NewClientWithCodec
/usr/local/go/src/net/rpc/client.go:201 +0xd0
goroutine 1894 [chan receive]:
net/rpc.(*Client).Call(0xc208285e60, 0xbc2a50, 0x7, 0xb2d200, 0xc20819e320, 0x979020, 0xc20826de90, 0x0, 0x0)
/usr/local/go/src/net/rpc/client.go:315 +0xd6
github.com/hashicorp/consul/consul.(*ConnPool).RPC(0xc20803ef50, 0xbcf070, 0x3, 0x7f9e973fca58, 0xc2081cc240, 0x2, 0xbc2a50, 0x7, 0xb2d200, 0xc20819e320, ...)
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/pool.go:393 +0x20f
github.com/hashicorp/consul/consul.(*Server).forwardLeader(0xc20800c2c0, 0xbc2a50, 0x7, 0xb2d200, 0xc20819e320, 0x979020, 0xc20826de90, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/rpc.go:220 +0x20b
github.com/hashicorp/consul/consul.(*Server).forward(0xc20800c2c0, 0xbc2a50, 0x7, 0x7f9e97408b08, 0xc20819e320, 0xb2d200, 0xc20819e320, 0x979020, 0xc20826de90, 0x9ad2c0, ...)
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/rpc.go:197 +0x1af
github.com/hashicorp/consul/consul.(*KVS).Get(0xc20803a130, 0xc20819e320, 0xc20826de90, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/kvs_endpoint.go:82 +0x13e
reflect.Value.call(0xa7f620, 0xb2c3d0, 0x13, 0xbbea50, 0x4, 0xc2083816e0, 0x3, 0x3, 0x0, 0x0, ...)
/usr/local/go/src/reflect/value.go:419 +0x10e5
reflect.Value.Call(0xa7f620, 0xb2c3d0, 0x13, 0xc2083816e0, 0x3, 0x3, 0x0, 0x0, 0x0)
/usr/local/go/src/reflect/value.go:296 +0xbc
net/rpc.(*service).call(0xc2080da380, 0xc2080da140, 0xc2083191d0, 0xc2080a8a00, 0xc208154240, 0xb2d200, 0xc20819e320, 0x16, 0x979020, 0xc20826de90, ...)
/usr/local/go/src/net/rpc/server.go:382 +0x1f7
net/rpc.(*Server).ServeRequest(0xc2080da140, 0x7f9e973fe9c8, 0xc20832a840, 0x0, 0x0)
/usr/local/go/src/net/rpc/server.go:497 +0x2a0
github.com/hashicorp/consul/consul.(*Server).RPC(0xc20800c2c0, 0xbc2a50, 0x7, 0x96a8c0, 0xc20803a690, 0x979020, 0xc20826dd40, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/server.go:681 +0x13a
github.com/hashicorp/consul/command/agent.(*Agent).RPC(0xc20800c160, 0xbc2a50, 0x7, 0x96a8c0, 0xc20803a690, 0x979020, 0xc20826dd40, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/agent.go:409 +0x8e
github.com/hashicorp/consul/command/agent.(*HTTPServer).KVSGet(0xc20803f7c0, 0x7f9e973ff520, 0xc20834a820, 0xc2080d9450, 0xc20819e2d0, 0x0, 0x0, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/kvs_endpoint.go:69 +0x199
github.com/hashicorp/consul/command/agent.(*HTTPServer).KVSEndpoint(0xc20803f7c0, 0x7f9e973ff520, 0xc20834a820, 0xc2080d9450, 0x0, 0x0, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/kvs_endpoint.go:44 +0x39c
github.com/hashicorp/consul/command/agent.*HTTPServer.KVSEndpoint·fm(0x7f9e973ff520, 0xc20834a820, 0xc2080d9450, 0x0, 0x0, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/http.go:235 +0x69
github.com/hashicorp/consul/command/agent.func·021(0x7f9e973ff520, 0xc20834a820, 0xc2080d9450)
/home/ubuntu/go/src/github.com/hashicorp/consul/command/agent/http.go:307 +0x489
net/http.HandlerFunc.ServeHTTP(0xc208154c00, 0x7f9e973ff520, 0xc20834a820, 0xc2080d9450)
/usr/local/go/src/net/http/server.go:1265 +0x41
net/http.(*ServeMux).ServeHTTP(0xc2080cf710, 0x7f9e973ff520, 0xc20834a820, 0xc2080d9450)
/usr/local/go/src/net/http/server.go:1541 +0x17d
net/http.serverHandler.ServeHTTP(0xc208162a20, 0x7f9e973ff520, 0xc20834a820, 0xc2080d9450)
/usr/local/go/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc20834a780)
/usr/local/go/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:1751 +0x35e
goroutine 1588 [chan send, 99 minutes]:
github.com/hashicorp/memberlist.func·003()
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:285 +0x23e
created by github.com/hashicorp/memberlist.(*Memberlist).probeNode
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:287 +0xe88
goroutine 1344 [chan send, 181 minutes]:
github.com/hashicorp/memberlist.func·003()
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:285 +0x23e
created by github.com/hashicorp/memberlist.(*Memberlist).probeNode
/home/ubuntu/go/src/github.com/hashicorp/memberlist/state.go:287 +0xe88
goroutine 1900 [IO wait]:
net.(*pollDesc).Wait(0xc2082832c0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2082832c0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208283260, 0xc2084d4000, 0x1000, 0x1000, 0x0, 0x7f9e973fae18, 0xc208343688)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20803a0e0, 0xc2084d4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
bufio.(*Reader).fill(0xc208492a20)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadByte(0xc208492a20, 0x200000003, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:196 +0x86
github.com/hashicorp/go-msgpack/codec.(*ioDecReader).readn1(0xc2081b37a0, 0xc208237440)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/decode.go:90 +0x45
github.com/hashicorp/go-msgpack/codec.(*msgpackDecDriver).initReadNext(0xc2083f3e00)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/msgpack.go:540 +0x48
github.com/hashicorp/go-msgpack/codec.(*Decoder).decode(0xc208492a80, 0x988f80, 0xc2083436a0)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/decode.go:635 +0x59
github.com/hashicorp/go-msgpack/codec.(*Decoder).Decode(0xc208492a80, 0x988f80, 0xc2083436a0, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/decode.go:630 +0x7f
github.com/hashicorp/raft.decodeResponse(0xc2084daf80, 0x97b480, 0xc2083f3c60, 0xc20836c320, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:479 +0x79
github.com/hashicorp/raft.(*NetworkTransport).genericRPC(0xc2080103f0, 0xc208241760, 0x11, 0x0, 0x97b420, 0xc20836c320, 0x97b480, 0xc2083f3c60, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:287 +0x1e9
github.com/hashicorp/raft.(*NetworkTransport).AppendEntries(0xc2080103f0, 0xc208241760, 0x11, 0xc20836c320, 0xc2083f3c60, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:260 +0x95
github.com/hashicorp/raft.(*Raft).replicateTo(0xc20810a000, 0xc208044370, 0x26d, 0xc208241700)
/home/ubuntu/go/src/github.com/hashicorp/raft/replication.go:157 +0x469
github.com/hashicorp/raft.(*Raft).replicate(0xc20810a000, 0xc208044370)
/home/ubuntu/go/src/github.com/hashicorp/raft/replication.go:105 +0x2fa
github.com/hashicorp/raft.func·003()
/home/ubuntu/go/src/github.com/hashicorp/raft/raft.go:817 +0x36
github.com/hashicorp/raft.func·011()
/home/ubuntu/go/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
/home/ubuntu/go/src/github.com/hashicorp/raft/state.go:153 +0xe3
goroutine 1901 [select]:
github.com/hashicorp/raft.(*Raft).pipelineReplicate(0xc20810a000, 0xc208044420, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/raft/replication.go:348 +0x90b
github.com/hashicorp/raft.(*Raft).replicate(0xc20810a000, 0xc208044420)
/home/ubuntu/go/src/github.com/hashicorp/raft/replication.go:124 +0x348
github.com/hashicorp/raft.func·003()
/home/ubuntu/go/src/github.com/hashicorp/raft/raft.go:817 +0x36
github.com/hashicorp/raft.func·011()
/home/ubuntu/go/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
/home/ubuntu/go/src/github.com/hashicorp/raft/state.go:153 +0xe3
goroutine 1902 [select]:
github.com/hashicorp/consul/consul.(*Server).leaderLoop(0xc20800c2c0, 0xc2084df0e0)
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/leader.go:105 +0x587
created by github.com/hashicorp/consul/consul.(*Server).monitorLeadership
/home/ubuntu/go/src/github.com/hashicorp/consul/consul/leader.go:37 +0x10b
goroutine 1903 [IO wait]:
net.(*pollDesc).Wait(0xc2082183e0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2082183e0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208218380, 0xc2084d9000, 0x1000, 0x1000, 0x0, 0x7f9e973fae18, 0xc20836a8c0)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20803a1e0, 0xc2084d9000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
bufio.(*Reader).fill(0xc2083b91a0)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadByte(0xc2083b91a0, 0x200000003, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:196 +0x86
github.com/hashicorp/go-msgpack/codec.(*ioDecReader).readn1(0xc2083b6c30, 0xc2082379e0)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/decode.go:90 +0x45
github.com/hashicorp/go-msgpack/codec.(*msgpackDecDriver).initReadNext(0xc208368ac0)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/msgpack.go:540 +0x48
github.com/hashicorp/go-msgpack/codec.(*Decoder).decode(0xc2083b9200, 0x988f80, 0xc20836a8b0)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/decode.go:635 +0x59
github.com/hashicorp/go-msgpack/codec.(*Decoder).Decode(0xc2083b9200, 0x988f80, 0xc20836a8b0, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/decode.go:630 +0x7f
github.com/hashicorp/raft.decodeResponse(0xc2082ecdc0, 0x97b480, 0xc208368ec0, 0xc20836d130, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:479 +0x79
github.com/hashicorp/raft.(*NetworkTransport).genericRPC(0xc2080103f0, 0xc208241760, 0x11, 0x0, 0x97b420, 0xc20836d130, 0x97b480, 0xc208368ec0, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:287 +0x1e9
github.com/hashicorp/raft.(*NetworkTransport).AppendEntries(0xc2080103f0, 0xc208241760, 0x11, 0xc20836d130, 0xc208368ec0, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:260 +0x95
github.com/hashicorp/raft.(*Raft).heartbeat(0xc20810a000, 0xc208044370, 0xc2084df1a0)
/home/ubuntu/go/src/github.com/hashicorp/raft/replication.go:303 +0x288
github.com/hashicorp/raft.func·009()
/home/ubuntu/go/src/github.com/hashicorp/raft/replication.go:92 +0x43
github.com/hashicorp/raft.func·011()
/home/ubuntu/go/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
/home/ubuntu/go/src/github.com/hashicorp/raft/state.go:153 +0xe3
goroutine 1904 [select]:
github.com/hashicorp/raft.(*Raft).heartbeat(0xc20810a000, 0xc208044420, 0xc2084df260)
/home/ubuntu/go/src/github.com/hashicorp/raft/replication.go:295 +0x691
github.com/hashicorp/raft.func·009()
/home/ubuntu/go/src/github.com/hashicorp/raft/replication.go:92 +0x43
github.com/hashicorp/raft.func·011()
/home/ubuntu/go/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
/home/ubuntu/go/src/github.com/hashicorp/raft/state.go:153 +0xe3
goroutine 1905 [IO wait]:
net.(*pollDesc).Wait(0xc208218f40, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208218f40, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208218ee0, 0xc2084ec000, 0x1000, 0x1000, 0x0, 0x7f9e973fae18, 0xc20836a9b8)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20803a708, 0xc2084ec000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
bufio.(*Reader).fill(0xc208304fc0)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadByte(0xc208304fc0, 0x83, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:196 +0x86
github.com/hashicorp/go-msgpack/codec.(*ioDecReader).readn1(0xc208221560, 0x16)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/decode.go:90 +0x45
github.com/hashicorp/go-msgpack/codec.(*msgpackDecDriver).initReadNext(0xc20825e3e0)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/msgpack.go:540 +0x48
github.com/hashicorp/go-msgpack/codec.(*Decoder).decode(0xc208305020, 0x988f80, 0xc20836a9c0)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/decode.go:635 +0x59
github.com/hashicorp/go-msgpack/codec.(*Decoder).Decode(0xc208305020, 0x988f80, 0xc20836a9c0, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/go-msgpack/codec/decode.go:630 +0x7f
github.com/hashicorp/raft.decodeResponse(0xc2082aa540, 0x97b480, 0xc208368ba0, 0x1184200, 0x0, 0x0)
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:479 +0x79
github.com/hashicorp/raft.(*netPipeline).decodeResponses(0xc2084bba40)
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:544 +0x1bf
created by github.com/hashicorp/raft.newNetPipeline
/home/ubuntu/go/src/github.com/hashicorp/raft/net_transport.go:529 +0x184
goroutine 1906 [select]:
github.com/hashicorp/raft.(*Raft).pipelineDecode(0xc20810a000, 0xc208044420, 0x7f9e97419d68, 0xc2084bba40, 0xc2084dff80, 0xc2083b8060)
/home/ubuntu/go/src/github.com/hashicorp/raft/replication.go:399 +0x290
github.com/hashicorp/raft.func·010()
/home/ubuntu/go/src/github.com/hashicorp/raft/replication.go:340 +0x5c
github.com/hashicorp/raft.func·011()
/home/ubuntu/go/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
/home/ubuntu/go/src/github.com/hashicorp/raft/state.go:153 +0xe3
rax 0xca
rbx 0x11851d8
rcx 0xffffffffffffffff
rdx 0x0
rdi 0x11851d8
rsi 0x0
rbp 0x1
rsp 0x7fff7045da98
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x21
r13 0x0
r14 0x0
r15 0x0
rip 0x4507d1
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment