The last post talked to the ability to connect to a "remote" JVM, that is, a JVM not running behind a firewall.  This could either be a home or corporate network.   The key point being that all TCP/IP ports on the machine running the JVM are accessible by another machine that will be running JConsole or JVisualVM.

This post, however, covers accessing a JVM behind a firewall - specifically AWS.  This means that only certain ports will be open on the "Server" machine for the "Client" machine to attach to.

The problem with running a RMI server is fairly well documented, so we won't cover it here in detail.  The 2 cent tour of the issue is that when configuring the RMI port for JMX it actually uses 2 RMI ports, one port is for the RMI registry, and the other is for the RMI Objects.  The problem is the second port is selected randomly at runtime, and without the ability for the "Client" machine to attach to the second RMI port, JMX remote monitoring will not function.  The first port is assigned with: "<port>".  This port must be opened on the firewall.

To diverge just a little, I would recommend setting up a JMX configuration from the last post first.  Test it with 2 separate machines, niether of which should be behind a firewall, and make sure it functions.  This will save you some of headaches.

What we will do is define a premain Java Agent.  Here is a link that explains a little about what this is and how to create one.  Here is another link that has an example you can use as a base to start from.  It also has many details on further locking down access to JMX.  We will make a few changes based on my experience using this on AWS.

Follow the instructions for building the premain agent.  I used NetBeans and maven.  You could just as easily use any other IDE.

The change I made was caused by the fact that the line:
String hostname = InetAddress.getLocalHost().getHostname()
returns localhost.  This basically meant that it was only listening for requests on the loopback address, and not the DHCP assigned address.  So I added a line immediately following that line:
hostname = System.getProperty("example.rmi.agent.hostname", hostname);
This allowed me to manually set the hostname if need be.  It it IS needed on AWS.

Thats it.  Compile the source, create the JAR file, and stick it somewhere where the CQ start script has access to it.  Then modify the start script, adding the following shell statements and JVM parameters:

Shell Statements:

  • pubhost=`ec2metadata --public-hostname`
  • jvmhost=`ec2metadata --local-hostname`

JVM Parameters:

  • -Djava.rmi.server.hostname=${pubhost}
  • -Dexample.rmi.agent.port=8003
  • -javaagent:/opt/jmxfirewallagent.jar

We talked above about why "${jvmhost}" is needed.  What I found was that the RMI server was still not responding to requests.  This is why "-Djava.rmi.server.hostname=${pubhost}" is needed.

I have not confirmed why, but this is my guess why it's needed.  Without "java.rmi.server.hostname" - when the initial connection is made, the server will communicate to the client the "private" AWS servername for RMI connections to be connect to.  The private AWS address is not accessible outside of AWS.  Setting "java.rmi.server.hostname" forces JMX to respond with the public AWS servername for RMI connections to connect to.  Or something close to that.  I have not had time to play around with the configuration to fully confirm why.

One other note of warning using JVisualVM.  The nice feature is that it saves the configurations for you.  The problem is that without an ElasticIP and a DNS hosting service, the name and IP address will change upon every single run of the AWS AMI.  When you start JVisualVM later on, it will try a "light" connect to that server, and when it fails (due to the IP address change), it will remove your configuration.

So I recommend using an ElasticIP address with your AMI.  You get 1 for free, and they only charge you when you are NOT using it.  However, it is CRAZY cheap.  The other thing you likely want is a DNS hosting service.  I use DynDNS - which is great.  I have been using that service for 5+ years.   Very reliable.  I associate a my custom hostname with the ElasticIP address.

The last issue, related to the last paragraph is this: if your JVM is started at system startup, your ElasticIP has not been assigned to your AMI yet.  This means that when you associate the ElasticIP to your AMI, your public IP will change.  But by then the JVM has started, and already been bound (via the shell statement: pubhost=`ec2metadata --public-hostname) to a different public IP address.  Again, this will cause JVisualVM to remove your configuration.  So BEFORE you start JVisualVM up, make sure that you have restarted your JVM to get the correct public IP address bound to the RMI server.

In a subsequent POST I will address using nagios or some other monitoring software to allow for a less hand on approach to watching your JVM.