Configuring Technologies to Work with Java Mission Control

Introduction

Java Mission Control was introduced with JDK 7u40. With JDK 8u40 and later, you can get JMC to monitor any Java workload out of the box. However, in order to customise how JMC and JFR will act we need to configure the Java Virtual Machines and technologies running them ourselves.

This post will explain how to enable JFR, log-in details for remote monitoring, configure ports and pass any arguments you want to the JVM when using Tomcat and Docker with JMC. In this blog, our test environments will only be Tomcat and Docker.

Tomcat

Client Side – setenv.sh

Here, I will explain how to set up Tomcat in one particular way. There are however many ways to customise how Tomcat will act. To pass arguments to the JVM inside Tomcat, we can create a script called “setenv.sh”. This will be sourced when we run catalina.sh and must be placed in "bin" directory of your Tomcat installation. An example setenv.sh file is below. It declares the “JAVA_HOME” environment variable to be the directory where the JDK is installed respectively. Other environment variable can be declared. Finally, it passes arguments to the JVM by declaring them in “CATALINA_OPTS”.

JAVA_HOME="/usr/lib/jdk1.8.0_181"

CATALINA_OPTS="-XX:+UnlockCommercialFeatures -XX:+FlightRecorder -Doracle.net.tns_admin=/usr/project/oracle/admin -Dcom.sun.management.jmxremote

-Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false  -Dcom.sun.management.jmxremote.rmi.port=9010  \
-Djava.rmi.server.hostname=scottsvm.cern.ch -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.registry.ssl=false \
-Dcom.sun.management.jmxremote.password.file=/usr/lib/jdk1.8.0_181/jre/lib/management/jmxremote.password \
-Dcom.sun.management.jmxremote.access.file=/usr/lib/jdk1.8.0_181/jre/lib/management/jmxremote.access \
-Dcom.sun.management.config.file=/usr/lib/jdk1.8.0_181/jre/lib/management/management.properties"

 

  • The “UnlockCommercialFeatures” and “FlightRecorder” options enable JFR to record remote JVMs.
  • “tns_admin” and “jmxremote” enable the use of a TNS names file and remote monitoring by JMX respectively.
  • The tnsnames.ora file is a configuration file that contains network service names mapped to connect descriptors for the local naming method, or network service names mapped to listener protocol addresses. A network service name is an alias mapped to a database network address contained in a connector descriptor. If you are using a database (Oracle) in your Java application, you can reference the TSN name of the database you want in your applciation, specified in tsnnames.ora, and link the file instead of hard-coding the full url.
  • The “port” and “rmi.port” determine the port any remote JMXs will listen to. JMX over RMI uses two ports. One port is for the RMI registry and the second port is used to export JMX RMI connection objects and it is dynamically allotted. To connect to the JMX agent, ensure that the RMI registry port number is an unused port.
  • “local.only” determines whether the server binds only to localhost or not.
  • ”hostname” defines the name of the server; this is used when monitoring the JVM remotely.
  • “authenticate” decides whether authentication is required for remote monitoring.
  • “jmxremote.ssl” and “.jmxremote.registry.ssl” determine whether SSL will be used. The use of SSL with Tomcat and Docker will not be discussed in this post. “password.file”, “access.file” and “config.file” specify the paths to the password, permissions and configuration files used for authentication.

Client Side – Authentication Files

In order to create new log-in details we need to edit the “jmxremote.password” and “jmxremote.access” files, which store usernames and passwords and account permissions respectively. They are located in JAVA_HOME/jre/lib/management. To add new log in details, simply edit jmxremote.password and on a new line, type the desired username, followed by a space, then the password. Next, you must add the username and the permissions you want it to have to the jmxremote.access file. It uses the same format as before, username, space, permissions. If you want the user to only be able to monitor the JVM, not change it or record it, use “readonly” as the permission. To allow commands use “readwrite”. To allow the use of JFR, append this to the permission: “create com.sun.management.*,com.oracle.jrockit.* unregister”.

Remote Side

To connect to a remote server in Java Mission Control, select “File”, then “Connect…”, then “Create a new connection”. In the new menu, enter the name of the server, as specified by the “-Djava.rmi.server.hostname” option in setenv.sh, and the port to listen to, specified by” -Dcom.sun.management.jmxremote.port”. If authentication is enabled, enter the username and password you created in the “jmxremote.password” and “jmxremote.access” files in their respective fields. You can test the connection to ensure you will be able to monitor the remote JVM. If it fails, please ensure the details you entered are correct. If the connection test is successful, you can proceed to monitor the server.

Docker

Dockerfile

In order to create a Docker container running Tomcat, we need to create a Dockerfile in the highest level directory of the project. Below is an example Dockerfile.

FROM gitlab-registry.cern.ch/db/k8s-tomcat/tomcat:latest

COPY java_perf_issues-0.0.1-SNAPSHOT.war webapps/

RUN mkdir -p /gubbins && \
   yum update -y && \
   yum install -y wget \
   gettext && \
   wget -o /gubbins/tsnnames.ora \
   https://example.server.com/tnsnames.ora

COPY scripts/start.sh /start.sh

RUN ["chmod", "+x", "/start.sh"]

COPY config/jmxremote.access \
     config/jmxremote.password \
     config/management.properties \
     $JAVA_HOME/lib/management/

COPY config/setenv.sh bin/

EXPOSE 9011 8080

CMD ["/start.sh"]

 

This Dockerfile allows for the user to specify account details by passing them as environment labels at launch. First, it declares the parent image it will use. Here, we are using a Tomcat image. Next, it copies the war file Tomcat will run into its webapps folder, as one would normally do. The war file contains your Java web application. In this example it is called “java_perf_issues-0.0.1-SNAPSHOT.war”. This name denotes the title of the application and the version. You can find an example of a web application here: https://gitlab.cern.ch/db/java_perf_issues.

It then make a directory to store files in and installs the program that will insert the user specified account details into the relevant files. Then it fetches tsnnames.ora from the specified web address, copies a start-up script in to the container and makes it runnable. Finally, it copies the files necessary for authentication into the Java directory, the user-created setenv.sh script (discussed earlier) into Tomcat’s folder and exposes the ports 9011 and 8080.

If you wish to use GitLab CI/CD, your .gitlab-ci.yml doesn’t need to have any special modifications for JMC.

Custom Script: start.sh

As referenced earlier, a script can be created to automatically insert account details passed at start-up. This allows users to define accounts on the go. If you do not want this feature, you can ignore this part. Below is an example of a script that creates a user specified account.

envsubst '${JMX_REMOTE_USERNAME} ${JMX_REMOTE_PERMISSION}' < $JAVA_HOME/lib/management/jmxremote.access > $JAVA_HOME/lib/management/jmxremote.access

envsubst '${JMX_REMOTE_USERNAME} ${JMX_REMOTE_PASSWORD}' < $JAVA_HOME/lib/management/jmxremote.password > $JAVA_HOME/lib/management/jmxremote.password

$CATALINA_HOME/bin/catalina.sh run

 

This script replaces any occurrences of the strings “$JMX_REMOTE_USERNAME” and “$JMX_REMOTE_PERMISSION” in the file “jmxremote.access” with whatever value the user has given those variables at start up.  It then does the same for “$JMX_REMOTE_USERNAME” and “$JMX_REMOTE_PASSWORD” in the “jmxremote.password” password. Finally, it runs the catalina.sh script to start Tomcat.

Building and Running the Container

From with the directory of the project, build the container using this command:

sudo docker build -t java_perf_issues .

To run the container, execute this command, using the -p args to bind the container ports to external ones and the -e args to set the username, password and permissions as environment variables:

sudo docker run -t -p java_perf_issues \

-p 8112:8080 -p 9011:9011 \

-e JMX_REMOTE_PASSWORD='password' \

-e JMX_REMOTE_USERNAME='username' \

-e JMX_REMOTE_PERMISSION='readwrite' java_perf_issues

 

The -p arguments specify which container ports (the second port) are to be bound to which host ports (first port) e.g. 8112(host):8080(container). For the port that RMI uses, please bind it to the same host port e.g. 9011:9011 works because the likes of 9112:9011 won't.

The -e arguments specify environment variables. The variables specified here are for the username and password you will use to access the JVM through Java Mission Control. Again, you can ignore these if you aren’t using authentication. The JMX_REMOTE_PERMISSION variable specifies whether you can issues commands to the JVM or not. 'readwrite' allows you to issues commands, 'readonly' does not. Any other values will result in an error. You can give the username and password variables whatever values you want as long as they don’t include spaces.

Finally, in your browser go to: example.com:8112/java_perf_issues-0.0.1-SNAPSHOT

Connecting to the JVM via JMC works the same as before.

Add new comment

You are here