heap dump

gistfile1.txt

mjog@devinstance:~/mixer_leak_dumps$ go tool pprof  --base 1_hd_initial.txt mixs.0.2.9 hd_resting.txt 
Entering interactive mode (type "help" for commands)
(pprof) top
266.20MB of 267.18MB total (99.63%)
Dropped 616 nodes (cum <= 1.34MB)
Showing top 10 nodes out of 16 (cum >= 2.01MB)
      flat  flat%   sum%        cum   cum%
  254.10MB 95.10% 95.10%   254.10MB 95.10%  runtime.malg
   11.09MB  4.15% 99.26%    11.09MB  4.15%  runtime.allgadd
    1.01MB  0.38% 99.63%     2.01MB  0.75%  istio.io/api/mixer/v1.(*ReportRequest).Unmarshal
         0     0% 99.63%     2.01MB  0.75%  github.com/golang/protobuf/proto.(*Buffer).Unmarshal
         0     0% 99.63%     2.01MB  0.75%  google.golang.org/grpc.(*Server).handleStream
         0     0% 99.63%     2.01MB  0.75%  google.golang.org/grpc.(*Server).processUnaryRPC
         0     0% 99.63%     2.01MB  0.75%  google.golang.org/grpc.(*Server).processUnaryRPC.func3
         0     0% 99.63%     2.01MB  0.75%  google.golang.org/grpc.(*Server).serveStreams.func1.1
         0     0% 99.63%     2.01MB  0.75%  google.golang.org/grpc.(*protoCodec).Unmarshal
         0     0% 99.63%     2.01MB  0.75%  google.golang.org/grpc.protoCodec.Unmarshal
(pprof) list runtime.malg 
Total: 267.18MB
ROUTINE ======================== runtime.malg in /builder/home/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/go1_8_3_linux_amd64/src/runtime/proc.go
  254.10MB   254.10MB (flat, cum) 95.10% of Total
         .          .   2816:   systemstack(afterfork)
         .          .   2817:}
         .          .   2818:
         .          .   2819:// Allocate a new g, with a stack big enough for stacksize bytes.
         .          .   2820:func malg(stacksize int32) *g {
  254.10MB   254.10MB   2821:   newg := new(g)
         .          .   2822:   if stacksize >= 0 {
         .          .   2823:           stacksize = round2(_StackSystem + stacksize)
         .          .   2824:           systemstack(func() {
         .          .   2825:                   newg.stack, newg.stkbar = stackalloc(uint32(stacksize))
         .          .   2826:           })

pprof) inuse_objects           
(pprof) top
1240114 of 1298110 total (95.53%)
Dropped 459 nodes (cum <= 6490)
Showing top 10 nodes out of 116 (cum >= 19114)
      flat  flat%   sum%        cum   cum%
    645533 49.73% 49.73%     645533 49.73%  runtime.malg
    426015 32.82% 82.55%     426015 32.82%  sync.runtime_SemacquireMutex
     32770  2.52% 85.07%      32770  2.52%  istio.io/mixer/pkg/adapterManager.(*Manager).dispatch
     32768  2.52% 87.60%      32768  2.52%  github.com/prometheus/client_golang/prometheus.makeLabelPairs
     32768  2.52% 90.12%      32768  2.52%  github.com/ugorji/go/codec.(*jsonDecDriver).DecodeString
     16427  1.27% 91.39%      32031  2.47%  istio.io/api/mixer/v1.(*ReportRequest).Unmarshal
     16384  1.26% 92.65%      16384  1.26%  istio.io/mixer/adapter/kubernetes.(*controllerImpl).updateIPPodMap
     15604  1.20% 93.85%      15604  1.20%  istio.io/api/mixer/v1.(*Attributes).Unmarshal
     10923  0.84% 94.69%      10923  0.84%  net.unpackStruct
     10922  0.84% 95.53%      19114  1.47%  context.WithDeadline