Skip to content

Instantly share code, notes, and snippets.

@divanikus
Created April 9, 2021 10:15
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save divanikus/88a555b18ca2d4cd36b05a87034a6cdf to your computer and use it in GitHub Desktop.
Save divanikus/88a555b18ca2d4cd36b05a87034a6cdf to your computer and use it in GitHub Desktop.
Apr 09 13:10:09 wm-mb-00 weed[9209]: SIGQUIT: quit
Apr 09 13:10:09 wm-mb-00 weed[9209]: PC=0x474821 m=0 sigcode=0
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 0 [idle]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: runtime.futex(0x34fed50, 0x80, 0x0, 0x0, 0x0, 0x7ffdd8fb3558, 0x83fe3, 0x25ac580, 0x7ffdd8fb3568, 0x40c3ff, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/sys_linux_amd64.s:579 +0x21
Apr 09 13:10:09 wm-mb-00 weed[9209]: runtime.futexsleep(0x34fed50, 0x0, 0xffffffffffffffff)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/os_linux.go:44 +0x46
Apr 09 13:10:09 wm-mb-00 weed[9209]: runtime.notesleep(0x34fed50)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/lock_futex.go:159 +0x9f
Apr 09 13:10:09 wm-mb-00 weed[9209]: runtime.mPark()
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/proc.go:1340 +0x39
Apr 09 13:10:09 wm-mb-00 weed[9209]: runtime.stopm()
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/proc.go:2257 +0x92
Apr 09 13:10:09 wm-mb-00 weed[9209]: runtime.findrunnable(0xc00005c000, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/proc.go:2916 +0x72e
Apr 09 13:10:09 wm-mb-00 weed[9209]: runtime.schedule()
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/proc.go:3125 +0x2d7
Apr 09 13:10:09 wm-mb-00 weed[9209]: runtime.park_m(0xc034ef7380)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/proc.go:3274 +0x9d
Apr 09 13:10:09 wm-mb-00 weed[9209]: runtime.mcall(0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/asm_amd64.s:327 +0x5b
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 1 [IO wait]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.runtime_pollWait(0x7fbff14debd8, 0x72, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/netpoll.go:222 +0x55
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).wait(0xc00047a218, 0x72, 0x0, 0x0, 0x220f60c)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x45
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).waitRead(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*FD).Accept(0xc00047a200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_unix.go:401 +0x212
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*netFD).accept(0xc00047a200, 0x28, 0x7fc0181ab108, 0x20300b)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/fd_unix.go:172 +0x45
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*TCPListener).accept(0xc000117410, 0x40e2f8, 0x28, 0x412f5b)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/tcpsock_posix.go:139 +0x32
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*TCPListener).Accept(0xc000117410, 0xc0053f9aa8, 0x46fb7b, 0x25ff339294f8b5, 0xc000acd3bf)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/tcpsock.go:261 +0x65
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util.(*Listener).Accept(0xc000456360, 0xc0053f9b20, 0x18, 0xc000000180, 0x72f1fb)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/net_timeout.go:19 +0x37
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*Server).Serve(0xc000657340, 0x2617bf0, 0xc000456360, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2981 +0x285
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/command.(*FilerOptions).startFiler(0x34fd820)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/command/filer.go:220 +0x925
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/command.runFiler(0x34e10c0, 0xc00004c1f0, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/command/filer.go:142 +0x92
Apr 09 13:10:09 wm-mb-00 weed[9209]: main.main()
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/weed.go:66 +0x315
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 6 [chan receive]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/glog.(*loggingT).flushDaemon(0x34fdaa0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:883 +0x8b
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/glog.init.0
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:410 +0x274
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 27 [syscall, 41 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: os/signal.signal_recv(0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/sigqueue.go:168 +0xa5
Apr 09 13:10:09 wm-mb-00 weed[9209]: os/signal.loop()
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/os/signal/signal_unix.go:23 +0x25
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by os/signal.Notify.func1.1
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/os/signal/signal.go:151 +0x45
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 14 [chan receive, 41 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util/grace.init.0.func1()
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/grace/signal_handling.go:29 +0x8a
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/util/grace.init.0
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/grace/signal_handling.go:28 +0x185
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 16 [select]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: go.opencensus.io/stats/view.(*worker).start(0xc00004c780)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/go.opencensus.io@v0.22.4/stats/view/worker.go:276 +0xcd
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by go.opencensus.io/stats/view.init.0
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/go.opencensus.io@v0.22.4/stats/view/worker.go:34 +0x68
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 38 [select, 41 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.newClientStream.func5(0xc000610e00, 0xc000558000, 0x2626e60, 0xc00053a090)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:318 +0xaa
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc.newClientStream
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:317 +0xbf0
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 35 [sync.Cond.Wait, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: sync.runtime_notifyListWait(0xc00001c310, 0x21514)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/sema.go:513 +0xf8
Apr 09 13:10:09 wm-mb-00 weed[9209]: sync.(*Cond).Wait(0xc00001c300)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/sync/cond.go:56 +0x99
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util.NewLimitedOutOfOrderProcessor.func1(0xc000389100, 0x20, 0x20, 0xc00011a410)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/limiter.go:91 +0xef
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/util.NewLimitedOutOfOrderProcessor
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/limiter.go:79 +0x3fb
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 36 [select]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/stats.(*ServerStats).Start(0xc00001cc00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/stats/stats.go:92 +0x216
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/server.init.0
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/common.go:35 +0x57
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 82 [chan receive, 41 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*addrConn).resetTransport(0xc00024a840)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/clientconn.go:1156 +0x465
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc.(*addrConn).connect
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/clientconn.go:799 +0x12a
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 31 [IO wait]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.runtime_pollWait(0x7fbff14de838, 0x72, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/netpoll.go:222 +0x55
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).wait(0xc00047ae18, 0x72, 0x0, 0x0, 0x220f60c)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x45
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).waitRead(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*FD).Accept(0xc00047ae00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_unix.go:401 +0x212
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*netFD).accept(0xc00047ae00, 0x28, 0x7fc0181ab5b8, 0x203009)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/fd_unix.go:172 +0x45
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*TCPListener).accept(0xc000117ea8, 0x40e2f8, 0x28, 0x412f5b)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/tcpsock_posix.go:139 +0x32
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*TCPListener).Accept(0xc000117ea8, 0xc002787c20, 0x46fb7b, 0x25ff33abdab880, 0xc019f2938a)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/tcpsock.go:261 +0x65
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util.(*Listener).Accept(0xc00491eee0, 0xc002787c98, 0x18, 0xc000103380, 0x72f1fb)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/net_timeout.go:19 +0x37
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*Server).Serve(0xc000657500, 0x2617bf0, 0xc00491eee0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2981 +0x285
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/command.(*S3Options).startS3Server(0x34fcb80, 0x20)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/command/s3.go:207 +0xa70
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/command.runFiler.func1()
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/command/filer.go:129 +0x3a
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/command.runFiler
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/command/filer.go:127 +0x2ae
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 32 [chan receive]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc000660a50)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:128 +0x55
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:52 +0x2af
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 49 [sleep]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: time.Sleep(0xdf8475800)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/time.go:193 +0xd2
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopInterval(0xc000660a50)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:140 +0x85
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:53 +0x2d1
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 50 [sleep]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: time.Sleep(0x42ef9ec0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/time.go:193 +0xd2
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/filer.(*Filer).loopProcessingDeletion(0xc0003d33f0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filer/filer_deletion.go:66 +0xc6
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/filer.NewFiler
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filer/filer.go:59 +0x418
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 51 [select, 41 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000479500)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/balancer_conn_wrappers.go:69 +0xac
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc.newCCBalancerWrapper
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/balancer_conn_wrappers.go:60 +0x172
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 52 [chan receive, 41 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*addrConn).resetTransport(0xc000609080)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/clientconn.go:1156 +0x465
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc.(*addrConn).connect
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/clientconn.go:799 +0x12a
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 55 [select]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc000238c40)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:1339 +0x174
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc/internal/transport.newHTTP2Client
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:295 +0x173c
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 56 [IO wait]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.runtime_pollWait(0x7fbff14deda8, 0x72, 0xffffffffffffffff)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/netpoll.go:222 +0x55
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).wait(0xc000460218, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x45
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).waitRead(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*FD).Read(0xc000460200, 0xc00009a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_unix.go:166 +0x1d5
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*netFD).Read(0xc000460200, 0xc00009a000, 0x8000, 0x8000, 0x9814de, 0x800000601, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/fd_posix.go:55 +0x4f
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*conn).Read(0xc00000f330, 0xc00009a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/net.go:183 +0x91
Apr 09 13:10:09 wm-mb-00 weed[9209]: bufio.(*Reader).Read(0xc0003cc660, 0xc0000fa118, 0x9, 0x9, 0xc00027d200, 0xc008fc7dd8, 0x405cc5)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/bufio/bufio.go:227 +0x222
Apr 09 13:10:09 wm-mb-00 weed[9209]: io.ReadAtLeast(0x25ea380, 0xc0003cc660, 0xc0000fa118, 0x9, 0x9, 0x9, 0x18, 0xc02faafa28, 0x4669a0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/io/io.go:328 +0x87
Apr 09 13:10:09 wm-mb-00 weed[9209]: io.ReadFull(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/io/io.go:347
Apr 09 13:10:09 wm-mb-00 weed[9209]: golang.org/x/net/http2.readFrameHeader(0xc0000fa118, 0x9, 0x9, 0x25ea380, 0xc0003cc660, 0x0, 0xc000000000, 0xc008fc7e3f, 0xc008fc7e01)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/golang.org/x/net@v0.0.0-20201202161906-c7110b5ffcbb/http2/frame.go:237 +0x89
Apr 09 13:10:09 wm-mb-00 weed[9209]: golang.org/x/net/http2.(*Framer).ReadFrame(0xc0000fa0e0, 0xc02cb92408, 0x34fd3c0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/golang.org/x/net@v0.0.0-20201202161906-c7110b5ffcbb/http2/frame.go:492 +0xa5
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000238c40)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:1273 +0x185
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc/internal/transport.newHTTP2Client
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:300 +0xd51
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 57 [select]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00046e690, 0x1, 0x0, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/controlbuf.go:395 +0xff
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0003cc720, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/controlbuf.go:513 +0x1dd
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000238c40)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:346 +0x7b
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc/internal/transport.newHTTP2Client
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:344 +0xf27
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 58 [sleep]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: time.Sleep(0x37e11d600)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/time.go:193 +0xd2
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/stats.LoopPushingMetric(0x220c854, 0x5, 0xc000408620, 0xd, 0xc000055260, 0x21, 0xf)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/stats/metrics.go:167 +0x253
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/server.NewFilerServer
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server.go:109 +0x393
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 59 [select, 31 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc000118050, 0xc000116220, 0x5, 0x5, 0x31, 0x31, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:186 +0xb1
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000118050, 0xc000116220, 0x5, 0x5, 0x31, 0xc00029a018, 0x22c31c8)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:166 +0x190
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00053a120, 0xc000116220, 0x5, 0x5, 0x31, 0xc0004e3530, 0x9a787c)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:479 +0x55
Apr 09 13:10:09 wm-mb-00 weed[9209]: io.ReadAtLeast(0x25f2a20, 0xc00053a120, 0xc000116220, 0x5, 0x5, 0x5, 0xc007e676c0, 0x31, 0x31)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/io/io.go:328 +0x87
Apr 09 13:10:09 wm-mb-00 weed[9209]: io.ReadFull(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/io/io.go:347
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0003d0200, 0xc000116220, 0x5, 0x5, 0x31, 0x31, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:463 +0xd2
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*parser).recvMsg(0xc000116210, 0x40000000, 0x34d3dc0, 0x218e1a0, 0xc00796d1d0, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/rpc_util.go:510 +0x63
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.recvAndDecompress(0xc000116210, 0xc0003d0200, 0x0, 0x0, 0x40000000, 0x0, 0x0, 0x0, 0xc007e676c0, 0x31, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/rpc_util.go:641 +0x4d
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.recv(0xc000116210, 0x7fbff11f3b08, 0x352fc90, 0xc0003d0200, 0x0, 0x0, 0x20b2f40, 0xc009ff4460, 0x40000000, 0x0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/rpc_util.go:709 +0x9b
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc000128100, 0x20b2f40, 0xc009ff4460, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:884 +0xee
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc000128100, 0x8, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:735 +0x46
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*clientStream).withRetry(0xc000558000, 0xc0004e39d0, 0xc0004e39a0, 0xa0, 0x98)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:589 +0x3df
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc000558000, 0x20b2f40, 0xc009ff4460, 0x0, 0xc000476380)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:734 +0x105
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/pb/master_pb.(*seaweedKeepConnectedClient).Recv(0xc0002980c0, 0xc000000b9f, 0xc008752600, 0x12)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/pb/master_pb/master.pb.go:4157 +0x62
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/wdclient.(*MasterClient).tryConnectToMaster.func1(0x2646368, 0xc000298050, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/wdclient/masterclient.go:118 +0x952
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/pb.WithMasterClient.func1(0xc000610e00, 0x22, 0xc000442d90)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/pb/grpc_client_server.go:172 +0x76
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/pb.WithCachedGrpcClient(0xc000442da0, 0xc00051c000, 0x22, 0xc000442d90, 0x1, 0x1, 0x0, 0x10)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/pb/grpc_client_server.go:108 +0x15f
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/pb.WithMasterClient(0x7ffdd8fb3dfa, 0x21, 0x25f22c0, 0xc00000fd30, 0xc000442e58, 0x418573, 0x3531280)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/pb/grpc_client_server.go:170 +0x1ba
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/wdclient.(*MasterClient).tryConnectToMaster(0xc000476320, 0x7ffdd8fb3dfa, 0x21, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/wdclient/masterclient.go:98 +0x165
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/wdclient.(*MasterClient).tryAllMasters(0xc000476320)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/wdclient/masterclient.go:86 +0x14e
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/wdclient.(*MasterClient).KeepConnectedToMaster(0xc000476320)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/wdclient/masterclient.go:51 +0xd2
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/filer.(*Filer).KeepConnectedToMaster(0xc0003d33f0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filer/filer.go:116 +0x2f
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/server.NewFilerServer
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server.go:110 +0x3bf
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 60 [select]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc0000fae00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206 +0x105
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x18c
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 48 [select]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*persistConn).writeLoop(0xc0004867e0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:2382 +0xf7
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Transport).dialConn
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:1744 +0xc9c
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 47 [IO wait]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.runtime_pollWait(0x7fbff14decc0, 0x72, 0xffffffffffffffff)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/netpoll.go:222 +0x55
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).wait(0xc000128b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x45
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).waitRead(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*FD).Read(0xc000128b00, 0xc000480000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_unix.go:166 +0x1d5
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*netFD).Read(0xc000128b00, 0xc000480000, 0x1000, 0x1000, 0xc000442e88, 0xc000442d18, 0x2)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/fd_posix.go:55 +0x4f
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*conn).Read(0xc00000e058, 0xc000480000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/net.go:183 +0x91
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*persistConn).Read(0xc0004867e0, 0xc000480000, 0x1000, 0x1000, 0xc000442dc4, 0x2, 0x2)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:1922 +0x77
Apr 09 13:10:09 wm-mb-00 weed[9209]: bufio.(*Reader).fill(0xc00008d560)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/bufio/bufio.go:101 +0x108
Apr 09 13:10:09 wm-mb-00 weed[9209]: bufio.(*Reader).Peek(0xc00008d560, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/bufio/bufio.go:139 +0x4f
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*persistConn).readLoop(0xc0004867e0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:2083 +0x1a8
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Transport).dialConn
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:1743 +0xc77
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 87 [IO wait]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.runtime_pollWait(0x7fbff14dea08, 0x72, 0xffffffffffffffff)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/netpoll.go:222 +0x55
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).wait(0xc000129018, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x45
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).waitRead(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*FD).Read(0xc000129000, 0xc00279c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_unix.go:166 +0x1d5
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*netFD).Read(0xc000129000, 0xc00279c000, 0x8000, 0x8000, 0x9814de, 0x800000601, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/fd_posix.go:55 +0x4f
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*conn).Read(0xc00000e950, 0xc00279c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/net.go:183 +0x91
Apr 09 13:10:09 wm-mb-00 weed[9209]: bufio.(*Reader).Read(0xc0003ccde0, 0xc0000faab8, 0x9, 0x9, 0xc0027b6000, 0xc0004e6dd8, 0x405cc5)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/bufio/bufio.go:227 +0x222
Apr 09 13:10:09 wm-mb-00 weed[9209]: io.ReadAtLeast(0x25ea380, 0xc0003ccde0, 0xc0000faab8, 0x9, 0x9, 0x9, 0x18, 0xc0317d29c0, 0x4669a0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/io/io.go:328 +0x87
Apr 09 13:10:09 wm-mb-00 weed[9209]: io.ReadFull(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/io/io.go:347
Apr 09 13:10:09 wm-mb-00 weed[9209]: golang.org/x/net/http2.readFrameHeader(0xc0000faab8, 0x9, 0x9, 0x25ea380, 0xc0003ccde0, 0x0, 0xc000000000, 0xc0004e6e3f, 0xc0004e6e01)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/golang.org/x/net@v0.0.0-20201202161906-c7110b5ffcbb/http2/frame.go:237 +0x89
Apr 09 13:10:09 wm-mb-00 weed[9209]: golang.org/x/net/http2.(*Framer).ReadFrame(0xc0000faa80, 0xc01a7fd260, 0x34fd3c0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/golang.org/x/net@v0.0.0-20201202161906-c7110b5ffcbb/http2/frame.go:492 +0xa5
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0027b4000)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:1273 +0x185
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc/internal/transport.newHTTP2Client
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:300 +0xd51
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 86 [select]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0027b4000)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:1339 +0x174
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc/internal/transport.newHTTP2Client
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:295 +0x173c
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 81 [select, 41 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00044be80)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/balancer_conn_wrappers.go:69 +0xac
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc.newCCBalancerWrapper
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/balancer_conn_wrappers.go:60 +0x172
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 62 [select, 41 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc0005e2340)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90 +0xb1
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/syndtr/goleveldb/leveldb.openDB
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x429
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 63 [select]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc0005e2340)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101 +0xd7
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/syndtr/goleveldb/leveldb.openDB
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x44b
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 64 [select, 41 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc0005e2340)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825 +0x228
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/syndtr/goleveldb/leveldb.openDB
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x5a5
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 65 [select, 41 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc0005e2340)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762 +0x125
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/syndtr/goleveldb/leveldb.openDB
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x5c7
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 66 [select]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/go-redis/redis/v8/internal/pool.(*ConnPool).reaper(0xc0001a83c0, 0xdf8475800)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/go-redis/redis/v8@v8.4.4/internal/pool/pool.go:453 +0xd7
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/go-redis/redis/v8/internal/pool.NewConnPool
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/go-redis/redis/v8@v8.4.4/internal/pool/pool.go:107 +0x1fc
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 67 [chan receive]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0006304b0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:128 +0x55
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:52 +0x2af
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 68 [sleep]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: time.Sleep(0xdf8475800)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/time.go:193 +0xd2
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopInterval(0xc0006304b0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:140 +0x85
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:53 +0x2d1
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 69 [select, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc00278d270, 0xc00000d8e8, 0x5, 0x5, 0xc00491d848, 0xc000231138, 0xc00028ca02)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:186 +0xb1
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00278d270, 0xc00000d8e8, 0x5, 0x5, 0xc001997ed8, 0xc00029a1e0, 0x22c31c8)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:166 +0x190
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00279b0b0, 0xc00000d8e8, 0x5, 0x5, 0x903fac, 0xc002789610, 0x9a787c)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:479 +0x55
Apr 09 13:10:09 wm-mb-00 weed[9209]: io.ReadAtLeast(0x25f2a20, 0xc00279b0b0, 0xc00000d8e8, 0x5, 0x5, 0x5, 0xc001bb06bf, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/io/io.go:328 +0x87
Apr 09 13:10:09 wm-mb-00 weed[9209]: io.ReadFull(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/io/io.go:347
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0000efb00, 0xc00000d8e8, 0x5, 0x5, 0x0, 0x0, 0xc0027896f8)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:463 +0xd2
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*parser).recvMsg(0xc00000d8d8, 0x40000000, 0xc001bb05a0, 0x11f, 0x11f, 0x0, 0x0, 0xc0027897b0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/rpc_util.go:510 +0x63
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.recvAndDecompress(0xc00000d8d8, 0xc0000efb00, 0x0, 0x0, 0x40000000, 0x0, 0x0, 0x0, 0x0, 0xc002789840, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/rpc_util.go:641 +0x4d
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.recv(0xc00000d8d8, 0x7fbff11f3b08, 0x352fc90, 0xc0000efb00, 0x0, 0x0, 0x2040300, 0xc004cd7590, 0x40000000, 0x0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/rpc_util.go:709 +0x9b
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc000129900, 0x2040300, 0xc004cd7590, 0x0, 0xea99be, 0xc0006304b0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:884 +0xee
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc000129900, 0x8, 0x34fd3c0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:735 +0x46
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*clientStream).withRetry(0xc000486b40, 0xc002789ab0, 0xc002789a80, 0x50, 0x48)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:589 +0x3df
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc000486b40, 0x2040300, 0xc004cd7590, 0xec59ad, 0xc0003d33f0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:734 +0x105
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/pb/filer_pb.(*seaweedFilerSubscribeLocalMetadataClient).Recv(0xc0003fca40, 0xc004cd6640, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/pb/filer_pb/filer.pb.go:4728 +0x62
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/filer.(*MetaAggregator).subscribeToOneFiler.func3(0x264bb68, 0xc0003fc9d0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filer/meta_aggregator.go:133 +0x2dc
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/pb.WithGrpcFilerClient.func1(0xc002792000, 0x13, 0xc002789d28)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/pb/grpc_client_server.go:201 +0x76
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/pb.WithCachedGrpcClient(0xc002789d38, 0xc00029a1b0, 0x13, 0xc002789d28, 0x1, 0x1, 0xc00000fd30, 0xc002789db0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/pb/grpc_client_server.go:108 +0x15f
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/pb.WithGrpcFilerClient(0xc00029a1b0, 0x13, 0x25f22c0, 0xc00000fd30, 0xc002789f28, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/pb/grpc_client_server.go:199 +0x9f
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/pb.WithFilerClient(0xc0000533f8, 0x12, 0x25f22c0, 0xc00000fd30, 0xc002789f28, 0x0, 0x2)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/pb/grpc_client_server.go:193 +0x15e
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/filer.(*MetaAggregator).subscribeToOneFiler(0xc00013d5c0, 0xc0003d33f0, 0xc0000533f8, 0x12, 0xc0000533f8, 0x12)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filer/meta_aggregator.go:120 +0x3ca
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/filer.(*MetaAggregator).StartLoopSubscribe
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filer/meta_aggregator.go:45 +0xa9
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 70 [IO wait, 41 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.runtime_pollWait(0x7fbff14deaf0, 0x72, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/netpoll.go:222 +0x55
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).wait(0xc00047a298, 0x72, 0x0, 0x0, 0x220f60c)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x45
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).waitRead(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*FD).Accept(0xc00047a280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_unix.go:401 +0x212
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*netFD).accept(0xc00047a280, 0x28, 0x7fc0181ab108, 0x30)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/fd_unix.go:172 +0x45
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*TCPListener).accept(0xc000117428, 0x40e2f8, 0x28, 0x20dc320)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/tcpsock_posix.go:139 +0x32
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*TCPListener).Accept(0xc000117428, 0xc00279a240, 0x34fd3c0, 0x44356e, 0xc0004e9e68)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/tcpsock.go:261 +0x65
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util.(*Listener).Accept(0xc000456380, 0x22c12a0, 0xc0000a71e0, 0x263c028, 0xc00279a240)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/net_timeout.go:19 +0x37
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*Server).Serve(0xc0000a71e0, 0x2617bf0, 0xc000456380, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:621 +0x27f
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/command.(*FilerOptions).startFiler
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/command/filer.go:217 +0x8c5
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 88 [select]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00278c500, 0x1, 0x0, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/controlbuf.go:395 +0xff
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0003ccea0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/controlbuf.go:513 +0x1dd
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0027b4000)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:346 +0x7b
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc/internal/transport.newHTTP2Client
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:344 +0xf27
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 94 [select, 41 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.newClientStream.func5(0xc002792000, 0xc000486b40, 0x2626e60, 0xc00279b020)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:318 +0xaa
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc.newClientStream
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:317 +0xbf0
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 90 [select]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00278c640, 0x1, 0x0, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/controlbuf.go:395 +0xff
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0003cd080, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/controlbuf.go:513 +0x1dd
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0027b6300)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_server.go:291 +0xd7
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc/internal/transport.newHTTP2Server
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_server.go:288 +0x1110
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 91 [select]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0027b6300)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_server.go:966 +0x265
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc/internal/transport.newHTTP2Server
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_server.go:297 +0x1135
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 92 [IO wait]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.runtime_pollWait(0x7fbff14de920, 0x72, 0xffffffffffffffff)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/netpoll.go:222 +0x55
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).wait(0xc000129118, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x45
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).waitRead(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*FD).Read(0xc000129100, 0xc0027c0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_unix.go:166 +0x1d5
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*netFD).Read(0xc000129100, 0xc0027c0000, 0x8000, 0x8000, 0xc0027bfbd0, 0x405755, 0xc000114660)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/fd_posix.go:55 +0x4f
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*conn).Read(0xc00000f340, 0xc0027c0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/net.go:183 +0x91
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util.(*Conn).Read(0xc00279a240, 0xc0027c0000, 0x8000, 0x8000, 0x11, 0x40db9b, 0xc01a7fd290)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/net_timeout.go:48 +0x6c
Apr 09 13:10:09 wm-mb-00 weed[9209]: bufio.(*Reader).Read(0xc0003ccf00, 0xc0000fab98, 0x9, 0x9, 0x14, 0x7fc0181ab5b8, 0x18)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/bufio/bufio.go:227 +0x222
Apr 09 13:10:09 wm-mb-00 weed[9209]: io.ReadAtLeast(0x25ea380, 0xc0003ccf00, 0xc0000fab98, 0x9, 0x9, 0x9, 0x979a45, 0xc01a7fd29c, 0xc0027949c0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/io/io.go:328 +0x87
Apr 09 13:10:09 wm-mb-00 weed[9209]: io.ReadFull(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/io/io.go:347
Apr 09 13:10:09 wm-mb-00 weed[9209]: golang.org/x/net/http2.readFrameHeader(0xc0000fab98, 0x9, 0x9, 0x25ea380, 0xc0003ccf00, 0x0, 0x0, 0xc01a7fd290, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/golang.org/x/net@v0.0.0-20201202161906-c7110b5ffcbb/http2/frame.go:237 +0x89
Apr 09 13:10:09 wm-mb-00 weed[9209]: golang.org/x/net/http2.(*Framer).ReadFrame(0xc0000fab60, 0xc01a7fd290, 0x34fd3c0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/golang.org/x/net@v0.0.0-20201202161906-c7110b5ffcbb/http2/frame.go:492 +0xa5
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0027b6300, 0xc00279a390, 0x22c12e0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_server.go:453 +0x9b
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*Server).serveStreams(0xc0000a71e0, 0x263e798, 0xc0027b6300)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:742 +0xe2
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*Server).handleRawConn.func1(0xc0000a71e0, 0x263e798, 0xc0027b6300)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:703 +0x3f
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc.(*Server).handleRawConn
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:702 +0x505
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 95 [sync.Cond.Wait, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: sync.runtime_notifyListWait(0xc000479ed0, 0xa059)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/sema.go:513 +0xf8
Apr 09 13:10:09 wm-mb-00 weed[9209]: sync.(*Cond).Wait(0xc000479ec0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/sync/cond.go:56 +0x99
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).SubscribeLocalMetadata.func1(0xc000660a50)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_grpc_server_sub_meta.go:102 +0x53
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).LoopProcessLogData(0xc000660a50, 0x1314e7c1, 0xed8021523, 0x34fd3c0, 0xc0004dfb70, 0xc0003fcc90, 0x3bfcc3b, 0xed80215d3, 0x34fd3c0, 0x0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_read.go:44 +0x796
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).SubscribeLocalMetadata(0xc000460e80, 0xc0003cdce0, 0x2637b78, 0xc0003fcc50, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_grpc_server_sub_meta.go:100 +0x4d0
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/pb/filer_pb._SeaweedFiler_SubscribeLocalMetadata_Handler(0x21ce2c0, 0xc000460e80, 0x26347e8, 0xc0001c8c00, 0x352fc90, 0xc0000efc00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/pb/filer_pb/filer.pb.go:5144 +0x116
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc0000a71e0, 0x263e798, 0xc0027b6300, 0xc0000efc00, 0xc0004507e0, 0x34df360, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1329 +0xcd8
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*Server).handleStream(0xc0000a71e0, 0x263e798, 0xc0027b6300, 0xc0000efc00, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1409 +0xc68
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00029cc80, 0xc0000a71e0, 0x263e798, 0xc0027b6300, 0xc0000efc00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:746 +0xab
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:744 +0xa5
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 72 [sleep]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: time.Sleep(0x37e11d600)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/time.go:193 +0xd2
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/stats.LoopPushingMetric(0x220a1dc, 0x2, 0xc000408f60, 0xd, 0xc00051cf00, 0x21, 0xf)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/stats/metrics.go:167 +0x253
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/command.(*S3Options).startS3Server
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/command/s3.go:174 +0x3ea
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 73 [select, 41 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc0003d1000)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:318 +0x99
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:333
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc00047ae80, 0x2040300, 0xc004914c80, 0x0, 0x64, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:870 +0x731
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc00047ae80, 0x8, 0x26)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:735 +0x46
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*clientStream).withRetry(0xc000559680, 0xc002783c58, 0xc002783c28, 0x50, 0x48)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:593 +0x9f
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc000559680, 0x2040300, 0xc004914c80, 0xc0003bbc80, 0xc000559680)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:734 +0x105
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/pb/filer_pb.(*seaweedFilerSubscribeMetadataClient).Recv(0xc0003bbc80, 0x2626db8, 0xc00490ae40, 0xc00492a180)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/pb/filer_pb/filer.pb.go:4696 +0x62
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/s3api.(*S3ApiServer).subscribeMetaEvents.func2(0x264bb68, 0xc0003bbc10, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials_subscribe.go:51 +0x28a
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/s3api.(*S3ApiServer).WithFilerClient.func1(0xc002792000, 0x13, 0xc0000116d0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/s3api/s3api_handlers.go:49 +0x76
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/pb.WithCachedGrpcClient(0xc0000116e0, 0xc000053c38, 0x13, 0xc0000116d0, 0x1, 0x1, 0xc0004761c8, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/pb/grpc_client_server.go:108 +0x15f
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/s3api.(*S3ApiServer).WithFilerClient(0xc000233d50, 0xc000011758, 0xc0000a7300, 0xc000117440)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/s3api/s3api_handlers.go:47 +0x9d
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/s3api.(*S3ApiServer).subscribeMetaEvents(0xc000233d50, 0x220a1dc, 0x2, 0x222c08a, 0x16, 0x1674268b2bccd68f, 0x70, 0x20e9580)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials_subscribe.go:38 +0xf5
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/s3api.NewS3ApiServer
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/s3api/s3api_server.go:39 +0x128
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 74 [select, 41 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.newClientStream.func5(0xc002792000, 0xc000559680, 0x2626e60, 0xc00491bda0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:318 +0xaa
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc.newClientStream
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:317 +0xbf0
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 96 [sync.Cond.Wait, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: sync.runtime_notifyListWait(0xc00013d610, 0x5292)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/sema.go:513 +0xf8
Apr 09 13:10:09 wm-mb-00 weed[9209]: sync.(*Cond).Wait(0xc00013d600)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/sync/cond.go:56 +0x99
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).SubscribeMetadata.func1(0xc0006304b0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_grpc_server_sub_meta.go:48 +0x65
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).LoopProcessLogData(0xc0006304b0, 0x127a4c8f, 0xed8021561, 0x34fd3c0, 0xc004895b70, 0xc0048860d0, 0x3c0dddf, 0xed80215d3, 0x34fd3c0, 0x0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_read.go:44 +0x796
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).SubscribeMetadata(0xc000460e80, 0xc0027f06c0, 0x2637c18, 0xc004886090, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_grpc_server_sub_meta.go:46 +0x4b7
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/pb/filer_pb._SeaweedFiler_SubscribeMetadata_Handler(0x21ce2c0, 0xc000460e80, 0x26347e8, 0xc000154d80, 0x352fc90, 0xc0000eff00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/pb/filer_pb/filer.pb.go:5123 +0x116
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc0000a71e0, 0x263e798, 0xc0027b6300, 0xc0000eff00, 0xc0004507e0, 0x34df340, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1329 +0xcd8
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*Server).handleStream(0xc0000a71e0, 0x263e798, 0xc0027b6300, 0xc0000eff00, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1409 +0xc68
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00029cc80, 0xc0000a71e0, 0x263e798, 0xc0027b6300, 0xc0000eff00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:746 +0xab
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:744 +0xa5
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 328154 [chan send, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util.(*LimitedOutOfOrderProcessor).Execute(0xc00001c340, 0xc00897d050)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/limiter.go:113 +0x87
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).uploadReaderToChunks(0xc000460e80, 0x261c9f0, 0xc0086556c0, 0xc008985200, 0x7fbff11a8518, 0xc0088e0e00, 0x9903d96700400000, 0xc00860916c, 0x7, 0x0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_upload.go:116 +0x2a5
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).doPutAutoChunk(0xc000460e80, 0x2626df0, 0xc00004e060, 0x261c9f0, 0xc0086556c0, 0xc008985200, 0xc000400000, 0x0, 0xc0089b4480, 0xc0089ad8d0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_autochunk.go:111 +0x1ea
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).autoChunk(0xc000460e80, 0x2626df0, 0xc00004e060, 0x261c9f0, 0xc0086556c0, 0xc008985200, 0x0, 0xc0089b4480)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_autochunk.go:53 +0x1ef
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).PostHandler(0xc000460e80, 0x261c9f0, 0xc0086556c0, 0xc008985200, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write.go:69 +0x1ed
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).filerHandler(0xc000460e80, 0x261c9f0, 0xc0086556c0, 0xc008985200)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers.go:71 +0x75d
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.HandlerFunc.ServeHTTP(0xc00278a6f0, 0x261c9f0, 0xc0086556c0, 0xc008985200)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2069 +0x44
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*ServeMux).ServeHTTP(0xc000479e00, 0x261c9f0, 0xc0086556c0, 0xc008985200)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2448 +0x1ad
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.serverHandler.ServeHTTP(0xc000657340, 0x261c9f0, 0xc0086556c0, 0xc008985200)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2887 +0xa3
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*conn).serve(0xc00844ba40, 0x2626e60, 0xc0053c5b40)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:1952 +0x8cd
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Server).Serve
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:3013 +0x39b
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 100 [chan send, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util.(*LimitedOutOfOrderProcessor).Execute(0xc00001c340, 0xc00734d680)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/limiter.go:113 +0x87
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).uploadReaderToChunks(0xc000460e80, 0x261c9f0, 0xc0053bd5e0, 0xc005dbf000, 0x7fbff11a8518, 0xc006150100, 0x9903d96700400000, 0xc0095921cc, 0x7, 0x0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_upload.go:116 +0x2a5
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).doPutAutoChunk(0xc000460e80, 0x2626df0, 0xc00004e060, 0x261c9f0, 0xc0053bd5e0, 0xc005dbf000, 0xc000400000, 0x0, 0xc00517e480, 0xc0008a78d0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_autochunk.go:111 +0x1ea
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).autoChunk(0xc000460e80, 0x2626df0, 0xc00004e060, 0x261c9f0, 0xc0053bd5e0, 0xc005dbf000, 0x0, 0xc00517e480)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_autochunk.go:53 +0x1ef
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).PostHandler(0xc000460e80, 0x261c9f0, 0xc0053bd5e0, 0xc005dbf000, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write.go:69 +0x1ed
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).filerHandler(0xc000460e80, 0x261c9f0, 0xc0053bd5e0, 0xc005dbf000)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers.go:71 +0x75d
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.HandlerFunc.ServeHTTP(0xc00278a6f0, 0x261c9f0, 0xc0053bd5e0, 0xc005dbf000)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2069 +0x44
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*ServeMux).ServeHTTP(0xc000479e00, 0x261c9f0, 0xc0053bd5e0, 0xc005dbf000)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2448 +0x1ad
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.serverHandler.ServeHTTP(0xc000657340, 0x261c9f0, 0xc0053bd5e0, 0xc005dbf000)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2887 +0xa3
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*conn).serve(0xc0048ef400, 0x2626e60, 0xc00498c040)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:1952 +0x8cd
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Server).Serve
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:3013 +0x39b
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 328380 [select, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*persistConn).writeLoop(0xc0090767e0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:2382 +0xf7
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Transport).dialConn
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:1744 +0xc9c
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 1162 [select, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*persistConn).roundTrip(0xc0049d27e0, 0xc008886e80, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:2610 +0x765
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*Transport).roundTrip(0xc0005a6dc0, 0xc006e50700, 0x7fc0181ab5b8, 0x2c0, 0xc004e72dc0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:592 +0xacb
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*Transport).RoundTrip(0xc0005a6dc0, 0xc006e50700, 0xc0005a6dc0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/roundtrip.go:17 +0x35
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.send(0xc006e50700, 0x25f34a0, 0xc0005a6dc0, 0x0, 0x0, 0x0, 0xc008265698, 0x203000, 0x1, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/client.go:251 +0x454
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*Client).send(0xc00043df20, 0xc006e50700, 0x0, 0x0, 0x0, 0xc008265698, 0x0, 0x1, 0x40db9b)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/client.go:175 +0xff
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*Client).do(0xc00043df20, 0xc006e50700, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/client.go:717 +0x45f
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*Client).Do(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/client.go:585
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/s3api.(*S3ApiServer).putToFiler(0xc000233d50, 0xc006e50400, 0xc006f788c0, 0x4d, 0x7fbff11a8538, 0xc0051ffb00, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/s3api/s3api_object_handlers.go:358 +0x605
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/s3api.(*S3ApiServer).PutObjectHandler(0xc000233d50, 0x2617290, 0xc00675ced0, 0xc006e50400)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/s3api/s3api_object_handlers.go:76 +0x45a
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/s3api.(*IdentityAccessManagement).Auth.func1(0x2617290, 0xc00675ced0, 0xc006e50400)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials.go:150 +0xb9
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/s3api.track.func1(0x261c9f0, 0xc0020f27e0, 0xc006e50400)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/s3api/stats.go:34 +0x239
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.HandlerFunc.ServeHTTP(0xc00491e8e0, 0x261c9f0, 0xc0020f27e0, 0xc006e50400)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2069 +0x44
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/gorilla/mux.(*Router).ServeHTTP(0xc000154b40, 0x261c9f0, 0xc0020f27e0, 0xc008f3b400)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0xd3
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.serverHandler.ServeHTTP(0xc000657500, 0x261c9f0, 0xc0020f27e0, 0xc008f3b400)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2887 +0xa3
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*conn).serve(0xc0001a8640, 0x2626e60, 0xc0027fe000)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:1952 +0x8cd
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Server).Serve
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:3013 +0x39b
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 117464 [select, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*persistConn).writeLoop(0xc0003c9b00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:2382 +0xf7
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Transport).dialConn
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:1744 +0xc9c
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 98 [IO wait, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.runtime_pollWait(0x7fbff14de668, 0x72, 0xffffffffffffffff)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/netpoll.go:222 +0x55
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).wait(0xc00047b298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x45
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).waitRead(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*FD).Read(0xc00047b280, 0xc0048ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_unix.go:166 +0x1d5
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*netFD).Read(0xc00047b280, 0xc0048ff000, 0x1000, 0x1000, 0x43b05c, 0xc0027bec38, 0x469da0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/fd_posix.go:55 +0x4f
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*conn).Read(0xc00011ba08, 0xc0048ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/net.go:183 +0x91
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*persistConn).Read(0xc000177560, 0xc0048ff000, 0x1000, 0x1000, 0xc0027953e0, 0xc0027bed40, 0x405755)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:1922 +0x77
Apr 09 13:10:09 wm-mb-00 weed[9209]: bufio.(*Reader).fill(0xc0027f0f00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/bufio/bufio.go:101 +0x108
Apr 09 13:10:09 wm-mb-00 weed[9209]: bufio.(*Reader).Peek(0xc0027f0f00, 0x1, 0x0, 0x1, 0x4, 0xc006744f01, 0x3)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/bufio/bufio.go:139 +0x4f
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*persistConn).readLoop(0xc000177560)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:2083 +0x1a8
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Transport).dialConn
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:1743 +0xc77
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 99 [select, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*persistConn).writeLoop(0xc000177560)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:2382 +0xf7
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Transport).dialConn
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:1744 +0xc9c
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 117 [select]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc0000fb880)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206 +0x105
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x18c
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 328377 [select, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*persistConn).roundTrip(0xc0090767e0, 0xc006e54500, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:2610 +0x765
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*Transport).roundTrip(0xc0005a6dc0, 0xc009146100, 0x20300200000000, 0x7fbff0eb7fff, 0x7fbff0f41530)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:592 +0xacb
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*Transport).RoundTrip(0xc0005a6dc0, 0xc009146100, 0xc0005a6dc0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/roundtrip.go:17 +0x35
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.send(0xc009146100, 0x25f34a0, 0xc0005a6dc0, 0x0, 0x0, 0x0, 0xc005c7e470, 0x203002, 0x1, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/client.go:251 +0x454
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*Client).send(0xc00043df20, 0xc009146100, 0x0, 0x0, 0x0, 0xc005c7e470, 0x0, 0x1, 0x40db9b)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/client.go:175 +0xff
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*Client).do(0xc00043df20, 0xc009146100, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/client.go:717 +0x45f
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*Client).Do(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/client.go:585
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/s3api.(*S3ApiServer).putToFiler(0xc000233d50, 0xc00904be00, 0xc00911a480, 0x58, 0x7fbff11a8538, 0xc009102d80, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/s3api/s3api_object_handlers.go:358 +0x605
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/s3api.(*S3ApiServer).PutObjectHandler(0xc000233d50, 0x2617290, 0xc008747f38, 0xc00904be00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/s3api/s3api_object_handlers.go:76 +0x45a
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/s3api.(*IdentityAccessManagement).Auth.func1(0x2617290, 0xc008747f38, 0xc00904be00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials.go:150 +0xb9
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/s3api.track.func1(0x261c9f0, 0xc009100540, 0xc00904be00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/s3api/stats.go:34 +0x239
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.HandlerFunc.ServeHTTP(0xc00491e8e0, 0x261c9f0, 0xc009100540, 0xc00904be00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2069 +0x44
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/gorilla/mux.(*Router).ServeHTTP(0xc000154b40, 0x261c9f0, 0xc009100540, 0xc00904bc00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0xd3
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.serverHandler.ServeHTTP(0xc000657500, 0x261c9f0, 0xc009100540, 0xc00904bc00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2887 +0xa3
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*conn).serve(0xc008bdb7c0, 0x2626e60, 0xc006e54280)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:1952 +0x8cd
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Server).Serve
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:3013 +0x39b
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 119 [select]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc0005e2d00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101 +0xd7
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/syndtr/goleveldb/leveldb.openDB
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x44b
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 328351 [chan send, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util.(*LimitedOutOfOrderProcessor).Execute(0xc00001c340, 0xc008e9fb90)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/limiter.go:113 +0x87
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).uploadReaderToChunks(0xc000460e80, 0x261c9f0, 0xc008fcc1c0, 0xc008fe6300, 0x7fbff11a8518, 0xc008fa4ae0, 0x9903d96700400000, 0xc008d40e4c, 0x7, 0x0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_upload.go:116 +0x2a5
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).doPutAutoChunk(0xc000460e80, 0x2626df0, 0xc00004e060, 0x261c9f0, 0xc008fcc1c0, 0xc008fe6300, 0xc000400000, 0x0, 0xc008fa9800, 0xc00912f8d0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_autochunk.go:111 +0x1ea
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).autoChunk(0xc000460e80, 0x2626df0, 0xc00004e060, 0x261c9f0, 0xc008fcc1c0, 0xc008fe6300, 0x0, 0xc008fa9800)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_autochunk.go:53 +0x1ef
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).PostHandler(0xc000460e80, 0x261c9f0, 0xc008fcc1c0, 0xc008fe6300, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write.go:69 +0x1ed
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).filerHandler(0xc000460e80, 0x261c9f0, 0xc008fcc1c0, 0xc008fe6300)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers.go:71 +0x75d
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.HandlerFunc.ServeHTTP(0xc00278a6f0, 0x261c9f0, 0xc008fcc1c0, 0xc008fe6300)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2069 +0x44
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*ServeMux).ServeHTTP(0xc000479e00, 0x261c9f0, 0xc008fcc1c0, 0xc008fe6300)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2448 +0x1ad
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.serverHandler.ServeHTTP(0xc000657340, 0x261c9f0, 0xc008fcc1c0, 0xc008fe6300)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2887 +0xa3
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*conn).serve(0xc008b4d720, 0x2626e60, 0xc0005af640)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:1952 +0x8cd
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Server).Serve
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:3013 +0x39b
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 118 [select, 41 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc0005e2d00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90 +0xb1
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/syndtr/goleveldb/leveldb.openDB
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x429
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 120 [select, 41 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc0005e2d00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825 +0x228
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/syndtr/goleveldb/leveldb.openDB
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x5a5
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 121 [select, 41 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc0005e2d00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762 +0x125
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/syndtr/goleveldb/leveldb.openDB
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x5c7
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 327769 [chan send, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util.(*LimitedOutOfOrderProcessor).Execute(0xc00001c340, 0xc007c2c1b0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/limiter.go:113 +0x87
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).uploadReaderToChunks(0xc000460e80, 0x261c9f0, 0xc007b1c380, 0xc007b96900, 0x7fbff11a8518, 0xc0079bd4a0, 0x9903d96700400000, 0xc00524f0ab, 0xa, 0x0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_upload.go:116 +0x2a5
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).doPutAutoChunk(0xc000460e80, 0x2626df0, 0xc00004e060, 0x261c9f0, 0xc007b1c380, 0xc007b96900, 0xc000400000, 0x0, 0xc007b185a0, 0xc007c098d0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_autochunk.go:111 +0x1ea
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).autoChunk(0xc000460e80, 0x2626df0, 0xc00004e060, 0x261c9f0, 0xc007b1c380, 0xc007b96900, 0x0, 0xc007b185a0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_autochunk.go:53 +0x1ef
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).PostHandler(0xc000460e80, 0x261c9f0, 0xc007b1c380, 0xc007b96900, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write.go:69 +0x1ed
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).filerHandler(0xc000460e80, 0x261c9f0, 0xc007b1c380, 0xc007b96900)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers.go:71 +0x75d
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.HandlerFunc.ServeHTTP(0xc00278a6f0, 0x261c9f0, 0xc007b1c380, 0xc007b96900)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2069 +0x44
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*ServeMux).ServeHTTP(0xc000479e00, 0x261c9f0, 0xc007b1c380, 0xc007b96900)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2448 +0x1ad
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.serverHandler.ServeHTTP(0xc000657340, 0x261c9f0, 0xc007b1c380, 0xc007b96900)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2887 +0xa3
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*conn).serve(0xc007a005a0, 0x2626e60, 0xc008755940)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:1952 +0x8cd
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Server).Serve
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:3013 +0x39b
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 327618 [chan send, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util.(*LimitedOutOfOrderProcessor).Execute(0xc00001c340, 0xc006343b00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/limiter.go:113 +0x87
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).uploadReaderToChunks(0xc000460e80, 0x261c9f0, 0xc006e0a1c0, 0xc006aee800, 0x7fbff11a8518, 0xc006398c40, 0x9903d96700400000, 0xc005945abe, 0xa, 0x0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_upload.go:116 +0x2a5
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).doPutAutoChunk(0xc000460e80, 0x2626df0, 0xc00004e060, 0x261c9f0, 0xc006e0a1c0, 0xc006aee800, 0xc000400000, 0x0, 0xc002734e40, 0xc005bc58d0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_autochunk.go:111 +0x1ea
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).autoChunk(0xc000460e80, 0x2626df0, 0xc00004e060, 0x261c9f0, 0xc006e0a1c0, 0xc006aee800, 0x0, 0xc002734e40)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_autochunk.go:53 +0x1ef
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).PostHandler(0xc000460e80, 0x261c9f0, 0xc006e0a1c0, 0xc006aee800, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write.go:69 +0x1ed
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).filerHandler(0xc000460e80, 0x261c9f0, 0xc006e0a1c0, 0xc006aee800)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers.go:71 +0x75d
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.HandlerFunc.ServeHTTP(0xc00278a6f0, 0x261c9f0, 0xc006e0a1c0, 0xc006aee800)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2069 +0x44
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*ServeMux).ServeHTTP(0xc000479e00, 0x261c9f0, 0xc006e0a1c0, 0xc006aee800)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2448 +0x1ad
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.serverHandler.ServeHTTP(0xc000657340, 0x261c9f0, 0xc006e0a1c0, 0xc006aee800)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2887 +0xa3
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*conn).serve(0xc0071cff40, 0x2626e60, 0xc008cba300)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:1952 +0x8cd
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Server).Serve
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:3013 +0x39b
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 1158 [select, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*persistConn).roundTrip(0xc0022a7e60, 0xc005a1e900, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:2610 +0x765
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*Transport).roundTrip(0xc0005a6dc0, 0xc0064caa00, 0x7fc0181ab5b8, 0xc00006e800, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:592 +0xacb
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*Transport).RoundTrip(0xc0005a6dc0, 0xc0064caa00, 0xc0005a6dc0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/roundtrip.go:17 +0x35
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.send(0xc0064caa00, 0x25f34a0, 0xc0005a6dc0, 0x0, 0x0, 0x0, 0xc0082654a8, 0x203001, 0x1, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/client.go:251 +0x454
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*Client).send(0xc00043df20, 0xc0064caa00, 0x0, 0x0, 0x0, 0xc0082654a8, 0x0, 0x1, 0x40db9b)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/client.go:175 +0xff
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*Client).do(0xc00043df20, 0xc0064caa00, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/client.go:717 +0x45f
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*Client).Do(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/client.go:585
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/s3api.(*S3ApiServer).putToFiler(0xc000233d50, 0xc0064ca100, 0xc000137180, 0x4d, 0x7fbff11a8538, 0xc001f585a0, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/s3api/s3api_object_handlers.go:358 +0x605
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/s3api.(*S3ApiServer).PutObjectHandler(0xc000233d50, 0x2617290, 0xc00675c000, 0xc0064ca100)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/s3api/s3api_object_handlers.go:76 +0x45a
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/s3api.(*IdentityAccessManagement).Auth.func1(0x2617290, 0xc00675c000, 0xc0064ca100)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials.go:150 +0xb9
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/s3api.track.func1(0x261c9f0, 0xc001b7e540, 0xc0064ca100)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/s3api/stats.go:34 +0x239
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.HandlerFunc.ServeHTTP(0xc00491e8e0, 0x261c9f0, 0xc001b7e540, 0xc0064ca100)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2069 +0x44
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/gorilla/mux.(*Router).ServeHTTP(0xc000154b40, 0x261c9f0, 0xc001b7e540, 0xc007a1ee00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0xd3
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.serverHandler.ServeHTTP(0xc000657500, 0x261c9f0, 0xc001b7e540, 0xc007a1ee00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2887 +0xa3
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*conn).serve(0xc0000bef00, 0x2626e60, 0xc00498d740)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:1952 +0x8cd
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Server).Serve
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:3013 +0x39b
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 138190 [chan send, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util.(*LimitedOutOfOrderProcessor).Execute(0xc00001c340, 0xc0021542d0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/limiter.go:113 +0x87
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).uploadReaderToChunks(0xc000460e80, 0x261c9f0, 0xc006e4f500, 0xc0060ed400, 0x7fbff11a8518, 0xc0068e02e0, 0x9903d96700400000, 0xc00519aace, 0xa, 0x0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_upload.go:116 +0x2a5
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).doPutAutoChunk(0xc000460e80, 0x2626df0, 0xc00004e060, 0x261c9f0, 0xc006e4f500, 0xc0060ed400, 0xc000400000, 0x0, 0xc004ec4cc0, 0xc0019418d0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_autochunk.go:111 +0x1ea
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).autoChunk(0xc000460e80, 0x2626df0, 0xc00004e060, 0x261c9f0, 0xc006e4f500, 0xc0060ed400, 0x0, 0xc004ec4cc0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_autochunk.go:53 +0x1ef
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).PostHandler(0xc000460e80, 0x261c9f0, 0xc006e4f500, 0xc0060ed400, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write.go:69 +0x1ed
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).filerHandler(0xc000460e80, 0x261c9f0, 0xc006e4f500, 0xc0060ed400)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers.go:71 +0x75d
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.HandlerFunc.ServeHTTP(0xc00278a6f0, 0x261c9f0, 0xc006e4f500, 0xc0060ed400)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2069 +0x44
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*ServeMux).ServeHTTP(0xc000479e00, 0x261c9f0, 0xc006e4f500, 0xc0060ed400)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2448 +0x1ad
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.serverHandler.ServeHTTP(0xc000657340, 0x261c9f0, 0xc006e4f500, 0xc0060ed400)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2887 +0xa3
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*conn).serve(0xc006b137c0, 0x2626e60, 0xc008887b80)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:1952 +0x8cd
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Server).Serve
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:3013 +0x39b
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 327600 [IO wait, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.runtime_pollWait(0x7fbff0ed0a60, 0x72, 0xffffffffffffffff)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/netpoll.go:222 +0x55
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).wait(0xc001e8ac18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x45
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).waitRead(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*FD).Read(0xc001e8ac00, 0xc006e3d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_unix.go:166 +0x1d5
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*netFD).Read(0xc001e8ac00, 0xc006e3d000, 0x1000, 0x1000, 0xc001b4a800, 0x200, 0x200)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/fd_posix.go:55 +0x4f
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*conn).Read(0xc0082647e0, 0xc006e3d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/net.go:183 +0x91
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*persistConn).Read(0xc006e9e000, 0xc006e3d000, 0x1000, 0x1000, 0x405690, 0x60, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:1922 +0x77
Apr 09 13:10:09 wm-mb-00 weed[9209]: bufio.(*Reader).fill(0xc002734ba0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/bufio/bufio.go:101 +0x108
Apr 09 13:10:09 wm-mb-00 weed[9209]: bufio.(*Reader).Peek(0xc002734ba0, 0x1, 0xc006bf2480, 0x203001, 0xc005ff1400, 0xc0024b2da8, 0x203001)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/bufio/bufio.go:139 +0x4f
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*persistConn).readLoop(0xc006e9e000)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:2083 +0x1a8
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Transport).dialConn
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:1743 +0xc77
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 327660 [select, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*persistConn).writeLoop(0xc006e9e900)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:2382 +0xf7
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Transport).dialConn
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:1744 +0xc9c
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 326638 [chan receive, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).uploadReaderToChunks.func1(0xc005bf44f0, 0xc0022708a0, 0xc009ba1350, 0xc00488d590)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_upload.go:46 +0x6d
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).uploadReaderToChunks
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_upload.go:43 +0x23b
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 327716 [select, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*persistConn).writeLoop(0xc006e9f440)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:2382 +0xf7
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Transport).dialConn
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/transport.go:1744 +0xc9c
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 327033 [chan receive, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).uploadReaderToChunks.func1(0xc0025c3980, 0xc004e748a0, 0xc00012c0f0, 0xc008f20e40)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_upload.go:46 +0x6d
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).uploadReaderToChunks
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_upload.go:43 +0x23b
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 453 [IO wait]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.runtime_pollWait(0x7fbff11ae780, 0x72, 0xffffffffffffffff)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/runtime/netpoll.go:222 +0x55
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).wait(0xc005a7cb98, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x45
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*pollDesc).waitRead(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
Apr 09 13:10:09 wm-mb-00 weed[9209]: internal/poll.(*FD).Read(0xc005a7cb80, 0xc00204a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/internal/poll/fd_unix.go:166 +0x1d5
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*netFD).Read(0xc005a7cb80, 0xc00204a000, 0x8000, 0x8000, 0x9814de, 0x800000601, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/fd_posix.go:55 +0x4f
Apr 09 13:10:09 wm-mb-00 weed[9209]: net.(*conn).Read(0xc005bf72a0, 0xc00204a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/net.go:183 +0x91
Apr 09 13:10:09 wm-mb-00 weed[9209]: bufio.(*Reader).Read(0xc0064cdbc0, 0xc0068e88f8, 0x9, 0x9, 0xc001fd3980, 0xc00217edd8, 0x405cc5)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/bufio/bufio.go:227 +0x222
Apr 09 13:10:09 wm-mb-00 weed[9209]: io.ReadAtLeast(0x25ea380, 0xc0064cdbc0, 0xc0068e88f8, 0x9, 0x9, 0x9, 0x18, 0xc02faafa10, 0x4669a0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/io/io.go:328 +0x87
Apr 09 13:10:09 wm-mb-00 weed[9209]: io.ReadFull(...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/io/io.go:347
Apr 09 13:10:09 wm-mb-00 weed[9209]: golang.org/x/net/http2.readFrameHeader(0xc0068e88f8, 0x9, 0x9, 0x25ea380, 0xc0064cdbc0, 0x0, 0xc000000000, 0xc00217ee3f, 0xc00217ee01)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/golang.org/x/net@v0.0.0-20201202161906-c7110b5ffcbb/http2/frame.go:237 +0x89
Apr 09 13:10:09 wm-mb-00 weed[9209]: golang.org/x/net/http2.(*Framer).ReadFrame(0xc0068e88c0, 0xc02cb923f0, 0x34fd3c0, 0x0, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/golang.org/x/net@v0.0.0-20201202161906-c7110b5ffcbb/http2/frame.go:492 +0xa5
Apr 09 13:10:09 wm-mb-00 weed[9209]: google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0002388c0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:1273 +0x185
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by google.golang.org/grpc/internal/transport.newHTTP2Client
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:300 +0xd51
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 327255 [chan receive, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).uploadReaderToChunks.func1(0xc006154a70, 0xc0026ef8c0, 0xc009ba0b40, 0xc008f39c50)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_upload.go:46 +0x6d
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).uploadReaderToChunks
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_upload.go:43 +0x23b
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 153 [chan send, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util.(*LimitedOutOfOrderProcessor).Execute(0xc00001c340, 0xc004c6b8c0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/limiter.go:113 +0x87
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).uploadReaderToChunks(0xc000460e80, 0x261c9f0, 0xc006e65dc0, 0xc0060edd00, 0x7fbff11a8518, 0xc005f71c40, 0x9903d96700400000, 0xc0024de8eb, 0xa, 0x0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_upload.go:116 +0x2a5
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).doPutAutoChunk(0xc000460e80, 0x2626df0, 0xc00004e060, 0x261c9f0, 0xc006e65dc0, 0xc0060edd00, 0xc000400000, 0x0, 0xc002271500, 0xc001a3f8d0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_autochunk.go:111 +0x1ea
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).autoChunk(0xc000460e80, 0x2626df0, 0xc00004e060, 0x261c9f0, 0xc006e65dc0, 0xc0060edd00, 0x0, 0xc002271500)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_autochunk.go:53 +0x1ef
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).PostHandler(0xc000460e80, 0x261c9f0, 0xc006e65dc0, 0xc0060edd00, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write.go:69 +0x1ed
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).filerHandler(0xc000460e80, 0x261c9f0, 0xc006e65dc0, 0xc0060edd00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers.go:71 +0x75d
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.HandlerFunc.ServeHTTP(0xc00278a6f0, 0x261c9f0, 0xc006e65dc0, 0xc0060edd00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2069 +0x44
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*ServeMux).ServeHTTP(0xc000479e00, 0x261c9f0, 0xc006e65dc0, 0xc0060edd00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2448 +0x1ad
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.serverHandler.ServeHTTP(0xc000657340, 0x261c9f0, 0xc006e65dc0, 0xc0060edd00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2887 +0xa3
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*conn).serve(0xc004952aa0, 0x2626e60, 0xc00494fd00)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:1952 +0x8cd
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Server).Serve
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:3013 +0x39b
Apr 09 13:10:09 wm-mb-00 weed[9209]: goroutine 327742 [chan send, 39 minutes]:
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/util.(*LimitedOutOfOrderProcessor).Execute(0xc00001c340, 0xc007c2c000)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/util/limiter.go:113 +0x87
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).uploadReaderToChunks(0xc000460e80, 0x261c9f0, 0xc007b1c1c0, 0xc007b12600, 0x7fbff11a8518, 0xc0079bd420, 0x9903d96700400000, 0xc005841e6b, 0xa, 0x0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_upload.go:116 +0x2a5
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).doPutAutoChunk(0xc000460e80, 0x2626df0, 0xc00004e060, 0x261c9f0, 0xc007b1c1c0, 0xc007b12600, 0xc000400000, 0x0, 0xc007b18480, 0xc007c2b8d0, ...)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_autochunk.go:111 +0x1ea
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).autoChunk(0xc000460e80, 0x2626df0, 0xc00004e060, 0x261c9f0, 0xc007b1c1c0, 0xc007b12600, 0x0, 0xc007b18480)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write_autochunk.go:53 +0x1ef
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).PostHandler(0xc000460e80, 0x261c9f0, 0xc007b1c1c0, 0xc007b12600, 0x0)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_write.go:69 +0x1ed
Apr 09 13:10:09 wm-mb-00 weed[9209]: github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).filerHandler(0xc000460e80, 0x261c9f0, 0xc007b1c1c0, 0xc007b12600)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers.go:71 +0x75d
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.HandlerFunc.ServeHTTP(0xc00278a6f0, 0x261c9f0, 0xc007b1c1c0, 0xc007b12600)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2069 +0x44
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*ServeMux).ServeHTTP(0xc000479e00, 0x261c9f0, 0xc007b1c1c0, 0xc007b12600)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2448 +0x1ad
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.serverHandler.ServeHTTP(0xc000657340, 0x261c9f0, 0xc007b1c1c0, 0xc007b12600)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:2887 +0xa3
Apr 09 13:10:09 wm-mb-00 weed[9209]: net/http.(*conn).serve(0xc007afa140, 0x2626e60, 0xc005fee400)
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:1952 +0x8cd
Apr 09 13:10:09 wm-mb-00 weed[9209]: created by net/http.(*Server).Serve
Apr 09 13:10:09 wm-mb-00 weed[9209]: /home/travis/.gimme/versions/go1.16.3.linux.amd64/src/net/http/server.go:3013 +0x39b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment