The Case

Oracle Grid infrastructure 12c comes with GIMR - Grid Infrastructure Management Repository, aka MGMTDB. This is intended to help in troubleshooting clusterware issues and has become mandatory starting from version 12.1.0.2.The tiny database instance in question runs in single instance mode and is managed by the clusterware. One of the perks is that it has its own listener (called MGMTLSNR), which listens on the private interconnect network interface and the default port 1521. Three services are typically registered on this special listener, the main one being the {DBNAME}_cluster service. This is used by the CHM (Cluster Health Monitor) daemons and clients. An example of the

$ lsnrctl status MGMTLSNR

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=MGMTLSNR)))
STATUS of the LISTENER
------------------------
Alias                     MGMTLSNR
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                04-MAR-2015 12:46:19
Uptime                    8 days 23 hr. 25 min. 55 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   {.....}/crs/network/admin/listener.ora
Listener Log File         {.....}/oracle/diag/tnslsnr/dbsrv1/mgmtlsnr/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=MGMTLSNR)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.0.0.1)(PORT=1521)))        <- Private interconnect network
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=169.254.165.41)(PORT=1521)))  <- HAIP network
Services Summary...
Service "-MGMTDBXDB" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
Service "_mgmtdb" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
Service "{DBNAME}_cluster" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
The command completed successfully

The Hardening

But here comes the main point of this blog post: by default the grid infrastructure remotely registers the MGMTDB services to the main database listeners: the local database listener of the node where it runs, as well as all the scan listeners. This creates an easy path to access to the MGMTDB from the user-facing network(s). We would rather restrict access to the MGMTDB to the local cluster nodes. Note: If you instead would like to keep the registration on SCAN and local LISTENER, see the recommendations of Support Note "_mgmtdb Service Registered with All Local Listeners in a Grid Infrastructure Environment (Doc ID 1945558.1)".

The grid infrastructure by default uses the local_listener and remote_listener parameters in the MGMTDB instance to register to the local and scan listener, while registration the private MGMTLSNR listener is performed using listener_networks parameter. However neither of those parameters is explicitly set in the MGMTDB parameter file, this functionality is implemented via the oraagent of the Grid infrastructure.

The Solution

As the registration in the user-facing listeners is unwanted for our configuration (for sure it is not used by the CHM functionality) and exposing the internal MGMTDB to the users network, let's get rid of it! As I've tested, it is sufficient for the MGMTDB to be registered only in its own MGMTLSNR listener as both ologgerd daemon and the client tools use just this listener for connecting to the MGMTDB. It is not clear why MGMTDB is by default registered in the main cluster's local and scan listeners, but it looks like it can be safely unregistered from them. To do it you can follow the steps below:

# Connect to the node where MGMTDB is running and run:
export ORACLE_SID=-MGMTDB
/CRS/dbs01/crs/bin/sqlplus "/ as sysdba"

# replace host names with the host names of your cluster
alter system set local_listener='(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbsrv1-priv)(PORT=1521))
      (ADDRESS=(PROTOCOL=TCP)(HOST=dbsrv2-priv)(PORT=1521)))' scope=both;
alter system set remote_listener='(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbsrv1-priv)(PORT=1521))
      (ADDRESS=(PROTOCOL=TCP)(HOST=dbsrv2-priv)(PORT=1521)))' scope=both;
alter system register;

When you set the local and remote listener parameters as above - pointing to the MGMTLSNR, the grid infrastructure will no longer register the DB in the cluster's local and scan listeners, it won't also try to set the MGMTLSNR listener again in listener_networks parameter (it is smart enough not to register it for the 2nd time).

Note: Because -MGMTDB instance can migrate between nodes of the cluster, always keeping the same name, it is necessary to include addresses of all cluster nodes in the address list (for both local and remote listener), so the definition can be pretty long for large clusters.

In principle it is also possible to change the MGMTLSNR port from the default 1521 to something else (it also requires ologgerd deamon to be restarted to pick up the new port); but in most configurations this is not really needed as this listener only listens on the private interconnect network, which should be, well... private(!) and thus not accessible from outside of the particular hardware RAC. However, just in case your network topology allows external access to the private network, or if you are really worried about using the default port, here is the procedure for changing it for the MGMTLSNR:

# as oracle
/CRS/dbs01/crs/crf/admin$ srvctl modify mgmtlsnr -endpoints "TCP:22333"
# as root
/CRS/dbs01/crs/bin/crsctl stop res ora.crf -init
/CRS/dbs01/crs/bin/crsctl start res ora.crf -init

Some Other Useful Tips

MGMTDB is a single instance database that can migrate between nodes of the cluster, but typically would run on instance 1. The spfile is located on the first CRS volume, together will all the MGMTDB files, for example:

/CRS/dbs00/DGTEST/COMPR-cluster/mgmtdb/_mgmtdb/spfile-MGMTDB.ora

The MGMTDB database and associated MGMTLSNR listener can be managed with SRVCTL using the following new commands:

srvctl modify mgmtlsnr -h
Modifies the configuration for the management listener.
Usage: srvctl modify mgmtlsnr -endpoints "[TCP:]<port>[,...][/IPC:<key>][/NMP:<pipe_name>][/TCPS:<s_port>] 
                                          [/SDP:<port>]"
    -endpoints           "[TCP:]<port>[, ...][/IPC:<key>][/NMP:<pipe_name>][/TCPS:<s_port>] [/SDP:<port>]"
                         Comma separated TCP ports or listener endpoints
    -help                Print usage

srvctl modify mgmtdb -h
Modifies the configuration for the management database.
Usage: srvctl modify mgmtdb [-pwfile <password_file_path>] [-spfile <server_parameter_file>]
    -pwfile <password_file_path>   Password file path
    -spfile <spfile>               Server parameter file path
    -help                          Print usage

Watch-out: MGMTDB instance uses huge pages by default (USE_LARGE_PAGES=TRUE) and it usually starts before the actual DB instances, together with CRS. It is important to remember that, when upgrading the clusterware to 12.1.0.2 the amount of huge pages configured in the OS should be increased, or database SGA decreased accordingly to leave enough huge pages for the MGMTDB instance. For now it requires around 750MB, so reserving 1GB of huge pages for this instance should be sufficient.

Default passwords

MGMTDB is a single tenant DB. In particular the CHM repository data is stored inside a PDB with a name of the form {DBNAME}_cluster. Most of the password for the root container and PDB are randomized by the installer or the accounts are locks, there are some notable exceptions: the PDBADMIN user has a default password. For security reasons we find advisable to change the PDBADMIN account password after installation or alternatively lock the account.

Acknowledgments

Many thanks to CERN DB Service colleagues for discussions and relevant work on this topic and in particular to Prasanth Kothuri, Andrei Dumitru and Luca Canali.