jvm-heap-thread-dump-analyze_tips

Production is awesome place to learn and know about the system. Team should feel lucky to work on those systems.

Make sure to have these parameters defined 
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=<path>

lets say that you are experiencing OOM, but you still don't determine the root cause of the issue, 
You might probably need to restart as soon as you see performance issues or even better daily thats perfectly fine & taking heap
dump will not practical. try to take heap dumpl.

The tools like springboot admin that do provide options to take heap/thread dumpare not useful here, please rely on below commands
Once you restarted start looking into pattern, leak usage taking incremental heapdump whenever you see a jump.
We will compare the multiple snapshots to find the culprit causing the leak OR holding our services for ransom


$JAVA_HOME/bin/jmap -dump:format=b,file=~\heap.hprof <PID>

Or if you have already a core file and you would like to have a heap dump as well, you can also get a heap dump:
jmap  -dump:format=b,file=heap.hprof $JAVA_HOME/bin/java /home/core.<pid>

use jps
jstack -f <pid> >> thread_dump.txt

ps -ef | grep java
OR
kill -3 <pid>

Hints:
1. Performance is Slow - Look into blocked
stack trace from the thread currently holding the lock - usually database connections.
The longer the thread stack, the more suspicious it is.
Look at the DC cycles if there are huge spikes.

2. Memory is high - Look into total number of threads 
Generate multile times to check how the thread us moving, check if you are running multiple services
it will reveal any long running threads.

for i in `seq 1 10` ; do
  echo ${i}
   jstack `ps aux | grep java | grep <particularClass> | grep -v grep | awk '{print $2}'` >> threaddump.log
  sleep 10
done

3. Look for hot threads
Generally scheduled threads and HTTP threads making requests will cause problem if they don;t get response.
possible states (NEW, RUNNABLE, BLOCKED, WAITING, TIMED_WAITING, TERMINATED),

waiting are not that useful, timed one will have timeouts. blocked and runnable are important for performance check

Tools:
https://fastthread.io - Use this tool for analyzing the dumps
https://gceasy.io/
JvisualVM
JMC