Troubleshooting Production JVMs with jcmd

troubleshooting-with-jcmd.md

Troubleshooting Production JVMs with jcmd

jcmd is a powerful new tool introduced in Java 7. Along with jstack and jps, it should be in your go-to tool for solving production problems on the JVM. (Come to think of it, with this tool you don't really need jps anymore)

Here's an example session with jcmd:

$ ssh wopr.qa.corp.local
$ jcmd -l
34739 sun.tools.jcmd.JCmd -l

jcmd -l lists the running Java processes on the machine. But I only see the JCmd tool itself. Where are my other processes? Turns out I can't see them because they are owned by another user ("prod").

Add a little sudo and they show up:

$ sudo jcmd -l        
24837 com.corp.app.web.HttpMain
24944 com.corp.app.workers.Main
34786 sun.tools.jcmd.JCmd -l

Great, there's our app's web server (com.corp.app.web.HttpMain) and a worker (com.corp.app.workers.Main). Let's see what we can do.

jcmd supports passing in a specific pid for a given process. Once connected, help tells you what operations are available:

$ sudo jcmd 24944 help
24944:
java.io.IOException: well-known file is not secure
        at sun.tools.attach.LinuxVirtualMachine.checkPermissions(Native Method)
        at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:117)
        at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)
        at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:213)
        at sun.tools.jcmd.JCmd.executeCommandForPid(JCmd.java:140)
        at sun.tools.jcmd.JCmd.main(JCmd.java:129)

Wait...what happened?

Google search provided the answer immediately: you can't connect to a Java process with jcmd unless you own the process. Again sudo to the rescue:

$ sudo -s -u prod jcmd 24944 help
24944:
The following commands are available:
VM.native_memory
VM.commercial_features
ManagementAgent.stop
ManagementAgent.start_local
ManagementAgent.start
Thread.print
GC.class_histogram
GC.heap_dump
GC.run_finalization
GC.run
VM.uptime
VM.flags
VM.system_properties
VM.command_line
VM.version
help
For more information about a specific command use 'help <command>'.

If you've ever been asked to answer the question "what version of Java are we running in production?", have puzzled over jhist or jmap command options, or needed to run full garbage collection across your entire cluster - and you manage dozens or hundreds (thousands?) of JVMs - that help output probably looks pretty good about now.

But wait! There's more!

If you think looking up the pids for your JVMs is tedious: you're right. Fortunately, jcmd also supports passing the "main" class of the JVM as the argument. The "main" class is of course the class that was used to launch the JVM.

It can be the full class name (including the package) or just the "simple" name (it matches using substring):

$ sudo -s -u prod jcmd HttpMain VM.version
24837:
Java HotSpot(TM) 64-Bit Server VM version 24.55-b03
JDK 7.0_55

Let's say you wanted to check the JVM version of all your applications in an environment. You could write a little shell script to do this, or - if you're into the devops thing and are using Chef - you could use knife:

$ knife ssh
 "role:app-worker AND chef_environment:qa" "sudo -u prod -s jcmd com.corp.app.workers.Main VM.version"                                                                                                                                     
wopr.qa.corp.local 24944:
wopr.qa.corp.local Java HotSpot(TM) 64-Bit Server VM version 24.55-b03
wopr.qa.corp.local JDK 7.0_55

My example just queried the JVM version, but you could use this tool to - for example - force garbage collection across all workers (GC.run).