Troubleshooting Production JVMs with
jcmd is a powerful new tool introduced in Java 7. Along with
jps, it should be in your go-to tool for solving production problems on the JVM. (Come to think of it, with this tool you don't really need
Here's an example session with
$ ssh wopr.qa.corp.local $ jcmd -l 34739 sun.tools.jcmd.JCmd -l
jcmd -l lists the running Java processes on the machine. But I only see the JCmd tool itself. Where are my other processes? Turns out I can't see them because they are owned by another user ("prod").
Add a little sudo and they show up:
$ sudo jcmd -l 24837 com.corp.app.web.HttpMain 24944 com.corp.app.workers.Main 34786 sun.tools.jcmd.JCmd -l
Great, there's our app's web server (
com.corp.app.web.HttpMain) and a worker (
com.corp.app.workers.Main). Let's see what we can do.
jcmd supports passing in a specific pid for a given process. Once connected, help tells you what operations are available:
$ sudo jcmd 24944 help 24944: java.io.IOException: well-known file is not secure at sun.tools.attach.LinuxVirtualMachine.checkPermissions(Native Method) at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:117) at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63) at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:213) at sun.tools.jcmd.JCmd.executeCommandForPid(JCmd.java:140) at sun.tools.jcmd.JCmd.main(JCmd.java:129)
Google search provided the answer immediately: you can't connect to a Java process with
jcmd unless you own the process. Again
sudo to the rescue:
$ sudo -s -u prod jcmd 24944 help 24944: The following commands are available: VM.native_memory VM.commercial_features ManagementAgent.stop ManagementAgent.start_local ManagementAgent.start Thread.print GC.class_histogram GC.heap_dump GC.run_finalization GC.run VM.uptime VM.flags VM.system_properties VM.command_line VM.version help For more information about a specific command use 'help <command>'.
If you've ever been asked to answer the question "what version of Java are we running in production?", have puzzled over
jmap command options, or needed to run full garbage collection across your entire cluster - and you manage dozens or hundreds (thousands?) of JVMs - that help output probably looks pretty good about now.
But wait! There's more!
If you think looking up the pids for your JVMs is tedious: you're right. Fortunately,
jcmd also supports passing the "main" class of the JVM as the argument. The "main" class is of course the class that was used to launch the JVM.
It can be the full class name (including the package) or just the "simple" name (it matches using substring):
$ sudo -s -u prod jcmd HttpMain VM.version 24837: Java HotSpot(TM) 64-Bit Server VM version 24.55-b03 JDK 7.0_55
Let's say you wanted to check the JVM version of all your applications in an environment. You could write a little shell script to do this, or - if you're into the devops thing and are using Chef - you could use
$ knife ssh "role:app-worker AND chef_environment:qa" "sudo -u prod -s jcmd com.corp.app.workers.Main VM.version" wopr.qa.corp.local 24944: wopr.qa.corp.local Java HotSpot(TM) 64-Bit Server VM version 24.55-b03 wopr.qa.corp.local JDK 7.0_55
My example just queried the JVM version, but you could use this tool to - for example - force garbage collection across all workers (