JMX and RMI

jmxrmi.md

Java Management Extensions

I think it is worthwhile providing some information about JMX, since it is very heavily used by all of our Java based entities in Brooklyn. JMX uses the RMI protocol to communicate, this is called JRMP and is implemented by the javax.management.remote.rmi.RMIConnector in the JVM.

Protocol Annoyances

There are a couple of problems with the RMI protocol as used by default in JMX.

1. It sends the remote address as data inside its protocol messages, causing problems for machines behind NAT or using split DNS.
2. The random port allocation for RMI means incorrect firewall configurations are generated.

The first problem is generally solved by setting the java.rmi.server.hostname system property being set appropriately. The second is still an issue, meaning deployments to AWS EC2 and other cloud providers are not possible for most JMX controlled entities.

Existing Solutions

Qpid solves this by ignoring the platform RMI server and starting its own, using the JMX port number plus 100 for the RMI server port. Other software simply uses the platform RMI server so a security group allowing all incoming TCP ports needs to be set to allow cloud deployments.

This was partially fixed by @pveentjer and @ahgittin with a new, secure JMX mechanism in 0.5.0, as part of the Betfair work. It uses an alternative transport called JMXMP instead of RMI. This optional connector is provided by Oracle [3] and uses JSSE, JAAS and SASL for security, as described in JSR-160.

New Feature

The new code I'm working on uses the same techniques as JMXMP, but for normal unsecured JMX over RMI, which is required for simplicity in demos and examples. Previously this was configured on an ad-hoc basis for a couple of entities only [4,5] but I have now made it the default for Java entities implementing the UsesJmx interface.

The underlying mechanism involves deploying an agent Jar file to the remote VM, which is then referenced at JVM startup and sets up the RMI server using well-known ports. For secure environments SSL over RMI is also available as an option, as well as the alternative JMXMP transport.

% java -javaagent:brooklyn-jmxrmi-0.5.0.jar \
    -Dbrooklyn.jmx.port=11099 \
    -Dbrooklyn.rmi.port=11199 \
    -Djava.rmi.server.hostname=$(hostname) \
    ...

On startup, the premain method in the agent Jar file is called by the JVM, and we can do our JMX and RMI setup here.

public static void premain(String agentArgs) throws IOException {

The work I have been doing recently to change the way JMX is used in Brooklyn will be included in 0.6.0 as well. I will start a discussion on brooklyn-dev about a back-port for a possible 0.5.1 release.

References

1. RMI is a horrible protocol!
2. https://github.com/brooklyncentral/brooklyn/tree/master/utils/jmx/jmxmp-ssl-agent
3. http://docs.oracle.com/javase/1.5.0/docs/guide/jmx/overview/connectors.html
4. https://github.com/brooklyncentral/brooklyn/tree/master/utils/jmx/jmxrmi-agent
5. https://github.com/brooklyncentral/brooklyn/blob/master/software/nosql/src/main/java/brooklyn/entity/nosql/cassandra/CassandraNodeSshDriver.java#L132