Skip to content

Instantly share code, notes, and snippets.

@patrick0057
Last active May 1, 2021 18:05
Show Gist options
  • Save patrick0057/0f6586aba1d213a6b0dac265f2f4ead5 to your computer and use it in GitHub Desktop.
Save patrick0057/0f6586aba1d213a6b0dac265f2f4ead5 to your computer and use it in GitHub Desktop.
go dumps for Rancher 2.x support

go dumps for Rancher 2.x support

If you are experiencing random crashing of your Rancher 2.x pods or docker container, sometimes Rancher support will ask you to take a go routine dump or a memory dump. Below are the commmands you need to run inside of your Rancher container or pod.

#exec into Rancher pod or container first
mkdir dumps
curl localhost:6060/debug/pprof/goroutine -o dumps/goroutine
curl localhost:6060/debug/pprof/heap -o dumps/heap
curl localhost:6060/debug/pprof/threadcreate -o dumps/threadcreate
curl localhost:6060/debug/pprof/block -o dumps/block
curl localhost:6060/debug/pprof/mutex -o dumps/mutex
tar -zcf dumps.tar.gz dumps
#exit container
exit
#copy file out of container
docker cp <id>:/var/lib/rancher/dumps.tar.gz /tmp/

Single server rancher k3s profile dump

docker exec -i CONTAINER kubectl get --raw /debug/pprof/profile > profile
docker exec -i $CONTAINER kubectl get --raw /debug/pprof/goroutine > goroutine
docker exec -i $CONTAINER kubectl get --raw /debug/pprof/heap > heap
docker exec -i $CONTAINER kubectl get --raw /debug/pprof/threadcreate > threadcreate
docker exec -i $CONTAINER kubectl get --raw /debug/pprof/block > block
docker exec -i $CONTAINER kubectl get --raw /debug/pprof/mutex > mutex
@aemneina
Copy link

aemneina commented Oct 5, 2020

Deprecate use of this script in favor for:

for pod in $(kubectl get pods --no-headers -l app=rancher | cut -d ' ' -f1); do
  echo getting heap for $pod
  kubectl exec $pod -- curl -s http://localhost:6060/debug/pprof/heap -o heap;
  kubectl cp $pod:heap $pod-heap;
  echo saved heap $pod-heap
done```

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment