Tool notes

jvm_garbage_analysis.txt

Garbage output options:
  "-verbose:gc",
  "-XX:+PrintHeapAtGC",
  "-XX:+PrintGCDetails",
  "-XX:+PrintGCDateStamps",
  "-XX:+PrintGCApplicationStoppedTime",
  "-XX:+PrintTenuringDistribution",
  "-Xloggc:/tmp/gc.log}", 
  "-XX:+UseGCLogFileRotation",
  "-XX:NumberOfGCLogFiles=3",
  "-XX:GCLogFileSize=100M",
  "-XX:+UnlockDiagnosticVMOptions",
  "-XX:ParGCCardsPerStrideChunk=32768"

GCViewer for analysis:
  https://github.com/chewiebug/GCViewer
  https://github.com/HubSpot/gc_log_visualizer (mostly G1)

ttop for garbage allocation rate (by thread)
  https://github.com/aragozin/jvm-tools
  
Benchmarking tools:
  http://openjdk.java.net/projects/code-tools/jmh/
  https://github.com/OpenHFT/Chronicle-Core/tree/master/src/main/java/net/openhft/chronicle/core/jlbh
  
Analysis of pause time:
  https://www.azul.com/jhiccup/

linux_system_analysis.txt

General "what's going on":
  dstat -lrvn 10
  nmon
  vmstat
  atop

Performance

Java:
  http://java-performance.com/
  
Linux:
  perf (https://perf.wiki.kernel.org/index.php/Main_Page)
  
  perf can be a bit finicky to set up though, One of the most interesting payoffs for the effort is getting cool flame graphs of cpu usage, like this example:
http://www.brendangregg.com/FlameGraphs/cpu-mixedmode-flamegraph-java.svg (from the blog post http://techblog.netflix.com/2015/07/java-in-flames.html)
 
Install perf:
·         linux-tools-common
·         linux-tools-generic (and perhaps a linux-tools-<version>-generic if the generic kernel version isn’t the one you’re currently running - perhaps because you haven’t restarted lately)
 
That should get perf working. There’s a good outline of how to use perf directly here: http://www.brendangregg.com/perf.html So far I’ve mostly used “perf record” then “perf report”.  The rest of this is specific to recognizing Java symbols so you get more interesting stack traces than “libjvm.so” in your output. Other languages can be profiled too, and arbitrary third-party code, you just might not have all the symbols defined for you.
 
If you’ll be debugging a Java process, you’ll need some more stuff:
·         openjdk-8-jdk (This ONLY works on JDK8 update 60 build 19 and higher)
·         openjdk-8-dbg
·         You’ll also need to start your java process with the -XX:+PreserveFramePointer option, which is why you need Java8.
·         A checkout of https://github.com/jvm-profiling-tools/perf-map-agent
·         A checkout of https://github.com/brendangregg/FlameGraph if you want perf-map-agent’s Flame Graph generation
 
The easiest thing is to use perf-map-agent’s scripts to do the right combination of calls for you. Follow the setup instructions in the readme for that repo to create the helper scripts.
 
To run those scripts, you’ll need some environment variables set:
·         FLAMEGRAPH_DIR=<dir of your FlameGraph repo checkout> (if you’re doing that)
·         PERF_MAP_OPTIONS=sourcepos   # helps with identifying Scala dynamically generated classes
·         PERF_RECORD_SECONDS=<how long to monitor the process>
·         JAVA_HOME=<the same java home your app is using>
Then something like:
                “perf-java-flames <Java PID> -F 99 -a -g”