Created
April 7, 2018 05:16
-
-
Save bclary/f8cf2f2414cfabcf3a7aa7e4c7dfb930 to your computer and use it in GitHub Desktop.
taskcluster-worker-0.1.13 stack
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ERRO[0000] Cannot set up syslog output error="Unix syslog delivery error" incidentId=bfee12cb-e7d9-4d7a-868f-c3d4c50eccc2 | |
| ERRO[0000] Cannot set up syslog output, syslog name configured: 'samsung-gs3-3' error="Unix syslog delivery error" incidentId=dc1b3698-3712-429a-a5f8-72dcce897a6f | |
| INFO[0001] starting up prefix=worker | |
| INFO[0002] Prefixing task logs: 'host' = 'tc-worker-samsung-gs3-3' plugin=logprefix prefix=plugin.logprefix | |
| INFO[0002] Prefixing task logs: 'provisionerId' = 'proj-autophone' plugin=logprefix prefix=plugin.logprefix | |
| INFO[0002] Prefixing task logs: 'workerType' = 'gecko-t-ap-unit-gs3' plugin=logprefix prefix=plugin.logprefix | |
| INFO[0002] Prefixing task logs: 'workerGroup' = 'sophie' plugin=logprefix prefix=plugin.logprefix | |
| INFO[0002] Prefixing task logs: 'workerId' = 'samsung-gs3-3' plugin=logprefix prefix=plugin.logprefix | |
| INFO[0004] starting to process task prefix=worker runId=0 taskId=Xhl8567cSO-Q_ORqC07l0A | |
| DEBU[0004] running stage: prepare prefix=worker.taskrun runId=0 stage=prepare taskId=Xhl8567cSO-Q_ORqC07l0A | |
| DEBU[0004] watchdog touched plugin=watchdog prefix=plugin.watchdog | |
| INFO[0004] Stopping worker after 1 tasks as configured plugin=reboot prefix=plugin.reboot | |
| DEBU[0004] running stage: build prefix=worker.taskrun runId=0 stage=build taskId=Xhl8567cSO-Q_ORqC07l0A | |
| DEBU[0004] watchdog touched plugin=watchdog prefix=plugin.watchdog | |
| DEBU[0004] running stage: start prefix=worker.taskrun runId=0 stage=start taskId=Xhl8567cSO-Q_ORqC07l0A | |
| DEBU[0004] running stage: started prefix=worker.taskrun runId=0 stage=started taskId=Xhl8567cSO-Q_ORqC07l0A | |
| DEBU[0004] watchdog running task incremented to: 1 plugin=watchdog prefix=plugin.watchdog | |
| DEBU[0004] watchdog touched plugin=watchdog prefix=plugin.watchdog | |
| DEBU[0004] watchdog timer paused, currently running: 1 plugin=watchdog prefix=plugin.watchdog | |
| DEBU[0004] running stage: waiting prefix=worker.taskrun runId=0 stage=waiting taskId=Xhl8567cSO-Q_ORqC07l0A | |
| DEBU[2898] running stage: stopped prefix=worker.taskrun runId=0 stage=stopped taskId=Xhl8567cSO-Q_ORqC07l0A | |
| DEBU[2898] watchdog running task decremented to: 0 plugin=watchdog prefix=plugin.watchdog | |
| DEBU[2898] watchdog touched plugin=watchdog prefix=plugin.watchdog | |
| DEBU[2898] running stage: finished prefix=worker.taskrun runId=0 stage=finished taskId=Xhl8567cSO-Q_ORqC07l0A | |
| DEBU[2898] watchdog touched plugin=watchdog prefix=plugin.watchdog | |
| DEBU[2901] running stage: dispose prefix=worker.taskrun runId=0 stage=dispose taskId=Xhl8567cSO-Q_ORqC07l0A | |
| DEBU[2901] watchdog touched plugin=watchdog prefix=plugin.watchdog | |
| ERRO[5001] watchdog-stackdump: goroutine 290 [running]: | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/plugins/watchdog.allStacks(0xc420684f98, 0x5a6e69) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/plugins/watchdog/watchdog.go:78 +0xe2 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/plugins/watchdog.(*plugin).waitForTimeout.func1() | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/plugins/watchdog/watchdog.go:93 +0x40 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/runtime/atomics.(*Once).Do(0xc4203da3c0, 0xc4204b4000, 0xc42043a900) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/runtime/atomics/once.go:47 +0x9c | |
| watchdog-stackdump: created by github.com/taskcluster/taskcluster-worker/plugins/watchdog.(*plugin).waitForTimeout | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/plugins/watchdog/watchdog.go:88 +0x197 | |
| watchdog-stackdump: | |
| watchdog-stackdump: goroutine 1 [select, 83 minutes]: | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/commands/work.cmd.Execute(0xc42046b1a0, 0x1) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/commands/work/work.go:53 +0x359 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/commands.Run(0x0, 0x0, 0x0, 0xc4200aa058) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/commands/run.go:59 +0x7bb | |
| watchdog-stackdump: main.main() | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/main.go:65 +0x32 | |
| watchdog-stackdump: | |
| watchdog-stackdump: goroutine 20 [syscall, 83 minutes]: | |
| watchdog-stackdump: os/signal.signal_recv(0x0) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/sigqueue.go:139 +0xa6 | |
| watchdog-stackdump: os/signal.loop() | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/os/signal/signal_unix.go:22 +0x22 | |
| watchdog-stackdump: created by os/signal.init.0 | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/os/signal/signal_unix.go:28 +0x41 | |
| watchdog-stackdump: | |
| watchdog-stackdump: goroutine 5 [chan receive, 83 minutes]: | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/vendor/github.com/getsentry/raven-go.(*Client).worker(0xc4200d8000) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/github.com/getsentry/raven-go/client.go:449 +0x170 | |
| watchdog-stackdump: created by github.com/taskcluster/taskcluster-worker/vendor/github.com/getsentry/raven-go.newClient | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/github.com/getsentry/raven-go/client.go:327 +0x101 | |
| watchdog-stackdump: | |
| watchdog-stackdump: goroutine 16 [chan receive, 83 minutes]: | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/vendor/github.com/getsentry/raven-go.(*Client).worker(0xc4200d8180) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/github.com/getsentry/raven-go/client.go:449 +0x170 | |
| watchdog-stackdump: created by github.com/taskcluster/taskcluster-worker/vendor/github.com/getsentry/raven-go.newClient | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/github.com/getsentry/raven-go/client.go:327 +0x101 | |
| watchdog-stackdump: | |
| watchdog-stackdump: goroutine 75 [select, 83 minutes, locked to thread]: | |
| watchdog-stackdump: runtime.gopark(0xc85a78, 0x0, 0xbfdffe, 0x6, 0x18, 0x1) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/proc.go:291 +0x11a | |
| watchdog-stackdump: runtime.selectgo(0xc420062750, 0xc4206f2de0) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/select.go:392 +0xe50 | |
| watchdog-stackdump: runtime.ensureSigM.func1() | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/signal_unix.go:549 +0x1f4 | |
| watchdog-stackdump: runtime.goexit() | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/asm_amd64.s:2361 +0x1 | |
| watchdog-stackdump: | |
| watchdog-stackdump: goroutine 26 [IO wait]: | |
| watchdog-stackdump: internal/poll.runtime_pollWait(0x7f51cb948f00, 0x72, 0xc420070928) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/netpoll.go:173 +0x57 | |
| watchdog-stackdump: internal/poll.(*pollDesc).wait(0xc42022a318, 0x72, 0xffffffffffffff00, 0xcecc40, 0x128c330) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9b | |
| watchdog-stackdump: internal/poll.(*pollDesc).waitRead(0xc42022a318, 0xc420534400, 0x400, 0x400) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3d | |
| watchdog-stackdump: internal/poll.(*FD).Read(0xc42022a300, 0xc420534400, 0x400, 0x400, 0x0, 0x0, 0x0) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17d | |
| watchdog-stackdump: net.(*netFD).Read(0xc42022a300, 0xc420534400, 0x400, 0x400, 0x0, 0x8, 0x3fb) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/net/fd_unix.go:202 +0x4f | |
| watchdog-stackdump: net.(*conn).Read(0xc42000e010, 0xc420534400, 0x400, 0x400, 0x0, 0x0, 0x0) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/net/net.go:176 +0x6a | |
| watchdog-stackdump: crypto/tls.(*block).readFromUntil(0xc4204c8540, 0x7f51cb824040, 0xc42000e010, 0x5, 0xc42000e010, 0x0) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/crypto/tls/conn.go:493 +0x96 | |
| watchdog-stackdump: crypto/tls.(*Conn).readRecord(0xc420530000, 0xc85b17, 0xc420530120, 0x0) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/crypto/tls/conn.go:595 +0xe0 | |
| watchdog-stackdump: crypto/tls.(*Conn).Read(0xc420530000, 0xc420537000, 0x1000, 0x1000, 0x0, 0x0, 0x0) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/crypto/tls/conn.go:1156 +0x100 | |
| watchdog-stackdump: bufio.(*Reader).fill(0xc4203b2300) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/bufio/bufio.go:100 +0x11e | |
| watchdog-stackdump: bufio.(*Reader).Peek(0xc4203b2300, 0x2, 0xc4206bacd0, 0xdf8475800, 0x1, 0xc420070d58, 0x79670e) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/bufio/bufio.go:132 +0x3a | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/vendor/github.com/gorilla/websocket.(*Conn).read(0xc420532500, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/github.com/gorilla/websocket/conn_read.go:12 +0x40 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/vendor/github.com/gorilla/websocket.(*Conn).advanceFrame(0xc420532500, 0xa, 0x0, 0x0) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/github.com/gorilla/websocket/conn.go:778 +0x5c | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/vendor/github.com/gorilla/websocket.(*Conn).NextReader(0xc420532500, 0xc4200c2038, 0xc420070f28, 0x4068fa, 0xc42022c180, 0x0) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/github.com/gorilla/websocket/conn.go:938 +0xa3 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/vendor/github.com/gorilla/websocket.(*Conn).ReadMessage(0xc420532500, 0xc42022c180, 0xc4206a2c00, 0x0, 0x5fb, 0x0, 0x0) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/github.com/gorilla/websocket/conn.go:1019 +0x2f | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/wsmux.(*Session).recvLoop(0xc4207a4090) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/wsmux/session.go:286 +0x6f | |
| watchdog-stackdump: created by github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/wsmux.newSession | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/wsmux/session.go:145 +0x2e4 | |
| watchdog-stackdump: | |
| watchdog-stackdump: goroutine 28 [select]: | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/wsmux.(*Session).sendKeepAlives(0xc4207a4090) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/wsmux/session.go:70 +0x188 | |
| watchdog-stackdump: created by github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/wsmux.newSession | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/wsmux/session.go:147 +0x328 | |
| watchdog-stackdump: | |
| watchdog-stackdump: goroutine 27 [select]: | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/wsmux.(*Session).removeDeadStreams(0xc4207a4090) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/wsmux/session.go:387 +0x1b3 | |
| watchdog-stackdump: created by github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/wsmux.newSession | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/wsmux/session.go:146 +0x306 | |
| watchdog-stackdump: | |
| watchdog-stackdump: goroutine 29 [select, 35 minutes]: | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/wsmux.(*Session).Accept(0xc4207a4090, 0x0, 0x0, 0x0, 0x0) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/wsmux/session.go:164 +0xf5 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/whclient.(*Client).Accept(0xc42010c4d0, 0x0, 0x0, 0x0, 0x0) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/whclient/client.go:93 +0x10d | |
| watchdog-stackdump: net/http.(*Server).Serve(0xc4200b8ea0, 0xcf1a80, 0xc42010c4d0, 0x0, 0x0) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/net/http/server.go:2770 +0x1a5 | |
| watchdog-stackdump: net/http.Serve(0xcf1a80, 0xc42010c4d0, 0xcec680, 0xc4203dc930, 0xab9860, 0xc4202dbbf0) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/net/http/server.go:2389 +0x73 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/runtime/webhookserver.NewWebhookTunnel.func2(0xc4204c8d50) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/runtime/webhookserver/webhooktunnel.go:53 +0x81 | |
| watchdog-stackdump: created by github.com/taskcluster/taskcluster-worker/runtime/webhookserver.NewWebhookTunnel | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/runtime/webhookserver/webhooktunnel.go:52 +0x122 | |
| watchdog-stackdump: | |
| watchdog-stackdump: goroutine 74 [semacquire, 83 minutes]: | |
| watchdog-stackdump: sync.runtime_notifyListWait(0xc420332408, 0xc400000000) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/sema.go:510 +0x10b | |
| watchdog-stackdump: sync.(*Cond).Wait(0xc4203323f8) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/sync/cond.go:56 +0x80 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/worker.(*taskCounter).WaitForLessThan(0xc4203323f0, 0x1) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/worker/taskcounter.go:55 +0x66 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/worker.(*Worker).Start(0xc420332240, 0x0, 0x0) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/worker/worker.go:251 +0x462 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/commands/work.cmd.Execute.func1(0xc420332240, 0xc4206f2d20) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/commands/work/work.go:48 +0x2b | |
| watchdog-stackdump: created by github.com/taskcluster/taskcluster-worker/commands/work.cmd.Execute | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/commands/work/work.go:47 +0x228 | |
| watchdog-stackdump: | |
| watchdog-stackdump: goroutine 101 [select]: | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/plugins/watchdog.(*plugin).waitForTimeout(0xc4205421c0) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/plugins/watchdog/watchdog.go:99 +0x263 | |
| watchdog-stackdump: created by github.com/taskcluster/taskcluster-worker/plugins/watchdog.provider.NewPlugin | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/plugins/watchdog/watchdog.go:67 +0x170 | |
| watchdog-stackdump: | |
| watchdog-stackdump: goroutine 40 [chan receive, 83 minutes]: | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/runtime/atomics.(*Once).Wait(0xc4203322e8) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/runtime/atomics/once.go:57 +0x42 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/worker.(*Worker).Start.func1(0xc420332240, 0xc4204bd080) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/worker/worker.go:193 +0x49 | |
| watchdog-stackdump: created by github.com/taskcluster/taskcluster-worker/worker.(*Worker).Start | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/worker/worker.go:192 +0x110 | |
| watchdog-stackdump: | |
| watchdog-stackdump: goroutine 157 [semacquire, 69 minutes]: | |
| watchdog-stackdump: sync.runtime_notifyListWait(0xc4203cc2d0, 0x86) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/sema.go:510 +0x10b | |
| watchdog-stackdump: sync.(*Cond).Wait(0xc4203cc2c0) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/sync/cond.go:56 +0x80 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/wsmux.(*stream).Write(0xc4201ec070, 0xc420409000, 0x1000, 0x1000, 0x0, 0x0, 0x0) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/github.com/taskcluster/webhooktunnel/wsmux/stream.go:299 +0x141 | |
| watchdog-stackdump: net/http.checkConnErrorWriter.Write(0xc4202ff5e0, 0xc420409000, 0x1000, 0x1000, 0xc420065c80, 0xc420065ca8, 0xc420807838) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/net/http/server.go:3313 +0x56 | |
| watchdog-stackdump: bufio.(*Writer).Flush(0xc4203cc600, 0xc4206d4000, 0xffa) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/bufio/bufio.go:573 +0x7e | |
| watchdog-stackdump: bufio.(*Writer).Write(0xc4203cc600, 0xc4206d4000, 0x1000, 0x8000, 0xc420807920, 0x1, 0x1) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/bufio/bufio.go:609 +0x107 | |
| watchdog-stackdump: net/http.(*chunkWriter).Write(0xc4206ee038, 0xc4206d4000, 0x1000, 0x8000, 0xc420807978, 0xc420807998, 0x6e48be) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/net/http/server.go:372 +0xab | |
| watchdog-stackdump: bufio.(*Writer).Write(0xc4203cc6c0, 0xc4206d4000, 0x1000, 0x8000, 0xc420001980, 0xc4204be000, 0xc4200c2058) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/bufio/bufio.go:605 +0x14d | |
| watchdog-stackdump: net/http.(*response).write(0xc4206ee000, 0x1000, 0xc4206d4000, 0x1000, 0x8000, 0x0, 0x0, 0xc4204be000, 0xc420807a48, 0xc420807a58) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/net/http/server.go:1529 +0x142 | |
| watchdog-stackdump: net/http.(*response).Write(0xc4206ee000, 0xc4206d4000, 0x1000, 0x8000, 0x0, 0x0, 0xc4200301f8) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/net/http/server.go:1499 +0x56 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/runtime/ioext.(*threadSafeWriteFlusher).Write(0xc4203da1c0, 0xc4206d4000, 0x1000, 0x8000, 0x0, 0x0, 0x0) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/runtime/ioext/copyflush.go:37 +0x9d | |
| watchdog-stackdump: io.copyBuffer(0xcea900, 0xc4203da1c0, 0x7f51cb95b368, 0xc4200301e0, 0xc4206d4000, 0x8000, 0x8000, 0xc420807b78, 0x20, 0x7667f4) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/io/io.go:402 +0x240 | |
| watchdog-stackdump: io.Copy(0xcea900, 0xc4203da1c0, 0x7f51cb95b368, 0xc4200301e0, 0xc42022c0c0, 0xc4203ae110, 0xc420807bd0) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/io/io.go:362 +0x5a | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/runtime/ioext.CopyAndFlush(0x7f51cb95b340, 0xc4206ee000, 0x7f51cb95b368, 0xc4200301e0, 0x5f5e100, 0x801, 0xc4202dc000, 0x40c680) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/runtime/ioext/copyflush.go:68 +0x128 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/plugins/livelog.(*taskPlugin).setup.func1(0xcf2d40, 0xc4206ee000, 0xc4201d8000) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/plugins/livelog/livelog.go:107 +0x375 | |
| watchdog-stackdump: net/http.HandlerFunc.ServeHTTP(0xc42038c0b0, 0xcf2d40, 0xc4206ee000, 0xc4201d8000) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/net/http/server.go:1947 +0x44 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/runtime/webhookserver.(*WebhookTunnel).handle(0xc4204c8d50, 0xcf2d40, 0xc4206ee000, 0xc4201d8000) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/runtime/webhookserver/webhooktunnel.go:99 +0x15b | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/runtime/webhookserver.(*WebhookTunnel).(github.com/taskcluster/taskcluster-worker/runtime/webhookserver.handle)-fm(0xcf2d40, 0xc4206ee000, 0xc4201d8000) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/runtime/webhookserver/webhooktunnel.go:53 +0x48 | |
| watchdog-stackdump: net/http.HandlerFunc.ServeHTTP(0xc4203dc930, 0xcf2d40, 0xc4206ee000, 0xc4201d8000) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/net/http/server.go:1947 +0x44 | |
| watchdog-stackdump: net/http.serverHandler.ServeHTTP(0xc4200b8ea0, 0xcf2d40, 0xc4206ee000, 0xc4201d8000) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/net/http/server.go:2694 +0xbc | |
| watchdog-stackdump: net/http.(*conn).serve(0xc4202ff5e0, 0xcf3c80, 0xc4203cc400) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/net/http/server.go:1830 +0x651 | |
| watchdog-stackdump: created by net/http.(*Server).Serve | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/net/http/server.go:2795 +0x27b | |
| watchdog-stackdump: | |
| watchdog-stackdump: goroutine 76 [semacquire, 35 minutes]: | |
| watchdog-stackdump: sync.runtime_Semacquire(0xc42037c244) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/sema.go:56 +0x39 | |
| watchdog-stackdump: sync.(*WaitGroup).Wait(0xc42037c238) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/sync/waitgroup.go:129 +0x72 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/vendor/gopkg.in/djherbis/stream%2ev1.(*Stream).Remove(0xc42037c230, 0x15, 0x0) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/vendor/gopkg.in/djherbis/stream.v1/stream.go:68 +0x41 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/runtime.(*TaskContextController).Dispose(0xc42000e0a0, 0x0, 0xc42080d208) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/runtime/taskcontext.go:125 +0x66 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/runtime.(*TaskContextController).Dispose-fm(0xc42080d228, 0x10) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/engines/enginetest/attachvolume.go:29 +0x2a | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/worker/taskrun.(*TaskRun).capturePanicAndError.func1() | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/worker/taskrun/taskrun.go:236 +0x32 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/runtime/monitoring.(*monitor).CapturePanic(0xc4205a23f0, 0xc4203ac080, 0x0, 0x0) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/runtime/monitoring/monitor.go:140 +0x5f | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/worker/taskrun.(*TaskRun).capturePanicAndError(0xc4200fdd40, 0xbfe6fb, 0x7, 0xc4204b42f0) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/worker/taskrun/taskrun.go:235 +0x112 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/worker/taskrun.(*TaskRun).Dispose(0xc4200fdd40, 0xc4206b60a0, 0x16) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/worker/taskrun/taskrun.go:321 +0x297 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/worker.(*Worker).processClaim(0xc420332240, 0xc4205bb230, 0x2b, 0xc420336780, 0x178, 0xc42044e240, 0x51, 0x0, 0x2e6e98c0, 0xed25b4d74, ...) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/worker/worker.go:461 +0xacf | |
| watchdog-stackdump: created by github.com/taskcluster/taskcluster-worker/worker.(*Worker).Start | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/worker/worker.go:232 +0x1b5 | |
| watchdog-stackdump: | |
| watchdog-stackdump: goroutine 159 [semacquire, 69 minutes]: | |
| watchdog-stackdump: sync.runtime_SemacquireMutex(0xc4203da1c4, 0x479a00) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/sema.go:71 +0x3d | |
| watchdog-stackdump: sync.(*Mutex).Lock(0xc4203da1c0) | |
| watchdog-stackdump: /home/travis/.gimme/versions/go1.10.linux.amd64/src/sync/mutex.go:134 +0x108 | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/runtime/ioext.(*threadSafeWriteFlusher).Flush(0xc4203da1c0) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/runtime/ioext/copyflush.go:41 +0x2d | |
| watchdog-stackdump: github.com/taskcluster/taskcluster-worker/runtime/ioext.CopyAndFlush.func1(0x5f5e100, 0xc4203da1c0, 0xc42022c0c0, 0xc4203ae110) | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/runtime/ioext/copyflush.go:59 +0x42 | |
| watchdog-stackdump: created by github.com/taskcluster/taskcluster-worker/runtime/ioext.CopyAndFlush | |
| watchdog-stackdump: /home/travis/gopath/src/github.com/taskcluster/taskcluster-worker/runtime/ioext/copyflush.go:55 +0xf4 | |
| watchdog-stackdump: error="watchdog: worker timeout exceeded" incidentId=30f550aa-ac10-413b-a97b-305b63dd019b plugin=watchdog prefix=plugin.watchdog | |
| ERRO[5301] error="Worker.Start(): livelock detected - didn't stop 5 min after StopNow()" incidentId=03129d7f-50ad-4a2a-996c-7f019aa26d28 prefix=worker |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment