General description

For developers it is a huge help if they have an application insight possibility and can check out whats going on under the hood on a production or test environment. JAVA allows to monitor remotely the Java Virtual Machine (JVM) itself with client applications like JConsole, JAVA Mission Control (JMC), etc. In order to have access to the remote JVM information from a local environment you need to enable and configure the JAVA's built in JAVA Management Extensions (JMX) on the remote server.

The main goal of this post is to put under spotlight what are the different possibilities which we have in hand for securely monitor the remote JVM through the JMX. The next image demonstrates what we would like to achive. We have a remote server running a JAVA application and from a local machine we would like to monitor its behaviour in a secure way. Here we have 2 different security aspects. The first one is that the communication between the monitoring client and the remote server should happen on a secured channel. This is where the SSL comes in to the picture, SSL encrypts the communication between the server and the client. But using just the SSL doesn't mean you are protecting your remote application, because JMX client can accessed anonymously as well. In other words you need to have some sort of authentication such as basic or LDAP authentication to prevent random strangers to harm your environment.

Set up the JMX connection through SSL

Giving remote access for the JVM can holds risks, I advise you to do some analyzis first. To whom you want to give access and how much credentials you want to allow them. If you are not sure play it on the safe side, evaluate it first on a machine which is less important. The evaluation you can do also on a docker image.

Enabling and configuring the JMX settings you do it when you start the JVM. You have to do it by adding command line arguments as you start the JVM.

The post will cover the following:

• 1. Connect to JMX through SSL anonymously (Stage 1) : This is the best for evaluation, not advised to run on a production environment

• 2. Connect to JMX through SSL with basic authentication (Stage 2) : This is a good solution for small developer groups in case you want to use dedicated monitoring users which you share easily within the team.

• 3. Connect to JMX through SSL with LDAP authentication (Stage 3) : This is a good solution for organisations who have more developer groups and have a working LDAP server already

I advise you to go stage by stage as you set up the environment. If you follow that strategy and you get stuck you most probably can resolve the issues faster . Keep the issue complexity as simple as possible!

In this blog post I'm not going to cover how you can open firewall ports and how you should configure your application server to use SSL. The focus will be how you set up the JMX with SSL with JAVA command line arguments.

I have created a public github repository with descriptions where you have all the settings in one place. In the repository you will find a bit more elegant way of designing the JMX connection command line arguments

0. Configuring client application

Since you want to use SSL channel for communication between the JMX and the monitoring client you will need to trust the SSL host certificate of the remote server.

0.a. Create a trusted key store file on the machine where you want to run the monitoring client, add the remote server's host certificate to it.

0.b. Link the trusted keystore to the monitoring agent. For example:

• With JConsole you do it with command line arguments as you start the application

# Here my trusted store is called "trustedRemoteCertStore" and the password for the trusted store is "changeit". Please note that I'm running the JConsole from the directory where I have the trusted store . $JAVA_HOME is the home path of my JRE

#>$JAVA_HOME/bin/jconsole -debug -J-Djavax.net.ssl.trustStore=trustedRemoteCertStore -J-Djavax.net.ssl.trustStorePassword=changeit

• With JMC (Java Mission Control) you do the linking within the "jmc.ini" file which is located next to the jmc executable in the  "$JAVA_HOME/bin/" folder. Here you add them as command line arguments. Once you have added you need to start JMC in the regular way it will pick up the settings from the file.

-Djavax.net.ssl.trustStore=/opt/certs/trustedRemoteCertStore
-Djavax.net.ssl.trustStorePassword=changeit

1. Connect to JMX through SSL anonymously (Stage 1)

1.a. Requirements:

• you need to have a host SSL certificate (preferred type is PKCS12 format, see JEP 229)

• open the JMX and RMI ports on the firewall (it is advised to use numbers which have one number difference between each other for example :29601,29602)

• JDK/JRE version higher than 5.0  (the JMX has been a part of J2SE since 2004 september)

1.b. Steps which you need to do:

• Open the firewall ports which you want to use for monitoring and grant the http, https services rights if necessary

• As a second step add the host certificate command line arguments and check if your application server starts or not. In case you have some issues with the certificate the application server should fail to start. Please note that you might want to do the LDAP authentication through SSL as well. For this you will need to create a certificate key store. Very important that the type of the keystore (-Djavax.net.ssl.trustStore) needs to be the same as the host certificate type (-Djavax.net.ssl.keyStore) !!! If you have different configuration then the application server might fail to start (with apache Tomcat/TomEE I have had this behaviour)

# your certificate file with its location in the example "YourCertFileName.p12" is the file and "/opt/cert" is the location of the file

-Djavax.net.ssl.keyStore=/opt/cert/YourCertFileName.p12

# the password for accessing the certificate file
-Djavax.net.ssl.keyStorePassword=changeit

# the type of the certificate file
-Djavax.net.ssl.keyStoreType=PKCS12

• As a next step you need to enable the jmx remote settings with the following command line arguments

#enable JMX remote

-Dcom.sun.management.jmxremote

-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote=true

# if you don't set this command line argument then it will take the ip of the machine by default, but since we would like to connect via SSL here you have to specify the hostname. During the authentication process this value is going to checked if you have it wrongly configured you will fail to connect

-Djava.rmi.server.hostname=$IP_OR_HOSTNAME_WITH_DOMAIN

# disable client authentication

-Dcom.sun.management.jmxremote.ssl.need.client.auth=false

# this will use the host certificate which you specified

-Dcom.sun.management.jmxremote.ssl=true

# disable authentication to connect anonymously

-Dcom.sun.management.jmxremote.authenticate=false

# Monitoring port numbers, please be aware that by default the jmx remote port is less by one than the RMI port. In case you don't follow the numbering you have to specificy in the connection string which numbers are they listening

-Dcom.sun.management.jmxremote.port=$PortNumber1
-Dcom.sun.management.jmxremote.rmi.port=$PortNumber2

1.c. Achievements:

• From the monitoring client you can connect anonymously to the JMX 

1.d. Important note(s):

• in case you have different ports this is the connection string which you have to use  "service:jmx:rmi://$HostName:$PortNumberRMI/jndi/rmi://$HostName:$PortNumberJMX/jmxrmi". "$HostName" is the name of the server with domain, "$PortNumberRMI" port number of RMI, "$PortNumberJMX" port number of the JMX
• in case you don't use SSL the RMI port and the JMX port can be the same

2. Connect to JMX through SSL with basic authentication (Stage 2)

2.a. Requirements:

• successfully passed the (stage 1), which means you were able to connect to the remote JMX anonymously 

2.b. Steps which you need to do:

• define JMX monitoring groups

Create a  "jmxremote.access" file with the following content:

monitorRole   readonly
controlRole   readwrite \
                         create com.sun.management.*,com.oracle.jrockit.* \
                         unregister

Here you define 2 user group a readonly group named "monitorRole" and a group called "controlRole" with read/write permissions.

• define JMX users

Create a  "jmxremote.password" file with the following content:

monitorRole tomcat
controlRole tomcat

Please note that in case of basic authentication the login username should match with the group name. Here in this example the login user name are "monitorRole" and "controlRole" and the password for each of them is "tomcat".

• update command line arguments to use basic authentication

Change in the previous defined authentication required flag to true

# we would like to have authentication

-Dcom.sun.management.jmxremote.authenticate=true

Link the basic authentication configuration files

# link the access file where you have designed the user groups in my example the file location is in the "/opt/jmx_access" folder

-Dcom.sun.management.jmxremote.access.file=/opt/jmx_access/jmxremote.access

# link the basic authentication username and password file in my example the file location is in the "/opt/jmx_access" folder

-Dcom.sun.management.jmxremote.password.file=/opt/jmx_access/jmxremote.password

2.c. Achievements:

• After this configuration you are able to connect to the remote JMX client with basic authentication on SSL channel

3. Connect to JMX through SSL with LDAP authentication (Stage 3)

3.a. Requirements:

• successfully passed the (stage 2), which means you were able to connect with username and password

3.b. Steps which you need to do:

• disable basic authentication

Since the LDAP authentication will use the same JMX groups you need to remove the basic authentication's password from the command line arguments. You can keep the file in case you want to switch back.

-Dcom.sun.management.jmxremote.password.file=/opt/jmx_access/jmxremote.password

• trust LDAP server's certificate (CA certificate) key

Since the LDAP authentication is happening on a lower application layer you need to add your certificate to the JRE/JDK's default trusted store. For some reasons if you have a trusted key store linked to the JVM (-Djavax.net.ssl.trustStore) the authentication will not work if you add there the certificate. For LDAP authentication adding to the default key store is the mandatory. The default key store is located "$JAVA_HOME/jre/lib/security/cacerts" here the trusted store is called "cacerts" and "$JAVA_HOME" is the location of your JAVA. The import will ask for a password the "cacerts" key store file's password is "changeit".

(Optional)

If your application needs to communicate with other environments you might need to create custom trusted key store and link it to the JVM. This is not mandatory for the JVM to authenticate the user on an SSL channel. The only reason why I think its important to mention this here, because as I mentioned earlier the host certificate's store type needs to be the same as the trusted custom key store type otherwise your application might not start (the advised type for both is PKCS12). You can create a trusted store with the JAVA "keytool" command line tool or use the same tool for appending an existing trusted store. (For more detials check the linked github repository)

You can link the trusted custom store to the JVM with the following JAVA command line arguments

# link the trusted store in the example my trust store called "trustedRemoteCertStore" and located in under the "/opt/cert" folder

-Djavax.net.ssl.trustStore=/opt/cert/trustedRemoteCertStore

# the password for the trusted store file is "changeit"
-Djavax.net.ssl.trustStorePassword=changeit

• create LDAP login module configuration

Create a "single_role_check.config" file with the following LDAP login module content. Here your $YourLDAPURL is your LDAP server, "$YourLDAPQuery" is the query which you want to execute,$YourUserDN is the DN of the loging required user and the role which you want to bind the user after successful login is the "monitorRole". So once you have authenticated successfully you will have the monitorRole rights within the JMX.

CustomLDAPLogin {
    com.sun.security.auth.module.LdapLoginModule REQUIRED
    userProvider="$YourLDAPURL"
    userFilter="$YourLDAPQuery"
    authIdentity="$YourUserDN"
    authzIdentity="monitorRole"
    tryFirstPass="true"
    useSSL="true"
    debug="true"
    ;
};

You can have multiple login checkings with different target roles. For this check the linked github repositository. In the "jmx-monitoring/jmx_access/multi_role_check.config" file you will find an example for that.

• update command line arguments to use LDAP authentication

# link the LDAP login module configuration here my file is located in the "/opt/jmx_access" folder

-Djava.security.auth.login.config=/opt/jmx_access/single_role_check.config

# the "single_role_check.config" can contain multiple login modules defined here you have to specify which one you want to use "CustomLDAPLogin" is the name of my login module in the file.

-Dcom.sun.management.jmxremote.login.config=CustomLDAPLogin

3.c. Achievements:

• After this configuration you are able to connect to the remote JMX client with LDAP authentication on SSL channel

3.d. Important note(s):

• the error messages are very generic. You will receive the same error message (authentication failed) if you have the following errors:

  • wrong LDAP query

  • wrong username or password

  • the authentication is disabled on the LDAP server just anonymous querying enabled

• If you have an LDAP service user in whose name you want to do the LDAP querying. You need to define it in the shared content object. If you don't have it you will see this warning in the log "[LdapLoginModule] tryFirstPass failed: javax.security.auth.login.FailedLoginException: No password was supplied". This means only that you don't have anything in the shared content and the login will proceed with the username and password which you have provided in the client. The service user is going to used to get only the user's DN name.

• The login process is the following query first the LDAP and then authenticate the user.