Useful JMX metric to monitor Java GC

Recently, I use JMXterm to collect info about Java JVM via JMX and MBeans. Here is a short note.

- Download JMXterm
https://sourceforge.net/projects/cyclops-group/files/jmxterm/1.0.0/jmxterm-1.0.0-uber.jar/download

- Run JMXterm
java -jar jmxterm-1.0.0-uber.jar --url localhost:<jmx listen port>

- All MBeans from java.lang
$>domain java.lang
$>beans
java.lang:name=CMS Old Gen,type=MemoryPool
java.lang:name=Code Cache,type=MemoryPool
java.lang:name=CodeCacheManager,type=MemoryManager
java.lang:name=Compressed Class Space,type=MemoryPool
java.lang:name=ConcurrentMarkSweep,type=GarbageCollector
java.lang:name=Metaspace Manager,type=MemoryManager
java.lang:name=Metaspace,type=MemoryPool
java.lang:name=Par Eden Space,type=MemoryPool
java.lang:name=Par Survivor Space,type=MemoryPool
java.lang:name=ParNew,type=GarbageCollector
java.lang:type=ClassLoading
java.lang:type=Compilation
java.lang:type=Memory
java.lang:type=OperatingSystem
java.lang:type=Runtime
java.lang:type=Threading

- Usually, we want to know how many time a garbage collection happen and how long a garbage collection is, so we use 2 MBeans
java.lang:name=ParNew,type=GarbageCollector  (young GC)
java.lang:name=ConcurrentMarkSweep,type=GarbageCollector  (in this case, I use CMS as major GC)

Now, let's explore inside
$>bean java.lang:name=ParNew,type=GarbageCollector$>info
#mbean = java.lang:name=ParNew,type=GarbageCollector
#class name = sun.management.GarbageCollectorImpl
# attributes
  %0   - CollectionCount (long, r)
  %1   - CollectionTime (long, r)
  %2   - LastGcInfo (javax.management.openmbean.CompositeData, r)
  %3   - MemoryPoolNames ([Ljava.lang.String;, r)
  %4   - Name (java.lang.String, r)
  %5   - ObjectName (javax.management.ObjectName, r)
  %6   - Valid (boolean, r)
#there's no operations
# notifications
  %0   - javax.management.Notification(com.sun.management.gc.notification)

Here, we care about CollectionCount and CollectionTime. Let's do a simple query.
$>get CollectionCount
#mbean = java.lang:name=ParNew,type=GarbageCollector:
CollectionCount = 546;

So, we can do something similar to fetch the query result to a time-series database, then have grafana draw a graph for us

- Query by Zabbix Zabbix jmxagent  
Define an item with a key like this:
jmx["java.lang:type=GarbageCollector,name=ParNew","CollectionCount"]

- Query by collectd JMX plugin.
Well, this is more complicated to define, because of the plugin configuration. We can take an example here: https://gist.github.com/huynhbaoan/e2cf40654649237cd6ca495ce5a9e13b
And the offical document: https://collectd.org/wiki/index.php/Plugin:GenericJMX
I will update collectd configurations later, if we decide to use this for our monitor solution.

- TL; DR:
Here is a short list of key to apply on Zabbix jmx agent. You can use these MBeans info to configure collectd plugin.
jmx["java.lang:type=GarbageCollector,name=ParNew","CollectionCount"]
jmx["java.lang:type=GarbageCollector,name=ParNew","CollectionTime"]
jmx["java.lang:type=GarbageCollector,name=ConcurrentMarkSweep","CollectionCount"]	
jmx["java.lang:type=GarbageCollector,name=ConcurrentMarkSweep","CollectionTime"]	
jmx["java.lang:type=Memory","HeapMemoryUsage.used"]	
jmx["java.lang:type=Memory","HeapMemoryUsage.max"]
jmx["java.lang:type=Memory","NonHeapMemoryUsage.used"]	
jmx["java.lang:type=Memory","NonHeapMemoryUsage.max"]	
jmx["java.lang:type=MemoryPool,name=Par Eden Space","Usage.used"]	
jmx["java.lang:type=MemoryPool,name=Par Survivor Space","Usage.used"]	
jmx["java.lang:type=MemoryPool,name=CMS Old Gen","Usage.used"]	
jmx["java.lang:type=MemoryPool,name=CMS Perm Gen","Usage.used"]	
jmx["java.lang:type=MemoryPool,name=Code Cache","Usage.used"]	

jmx["java.lang:type=Threading","DaemonThreadCount"]	
jmx["java.lang:type=ClassLoading","LoadedClassCount"]	
jmx["java.lang:type=ClassLoading","UnloadedClassCount"]	
jmx["java.lang:type=OperatingSystem","ProcessCpuLoad"]	
jmx["java.lang:type=Threading","ThreadCount"]	
jmx["java.lang:type=Compilation","TotalCompilationTime"]