在ZooKeeper组件上停用JMX agent之后如何允许cm继续监控ZooKeeper的运行状态

摘要
In TSB 2019-310 the workaround we asked users to do was to disable Zookeeper monitoring. However turning off the monitoring of Zookeeper service can be risky. This article explain the workaround which would allow Cloudera Manager monitoring Zookeeper health using Safety Valve.

适用于
 Zookeeper
 Cloudera Manager 6.1.0 and lower, Cloudera Manager 5.16 and lower

说明
If you can not upgrade to Cloudera Manager 6.1, then use below workaround:

Step 1:
Confirm that Zookeeper service monitoring is turned off in Cloudera Manager by checking:
Cloudera Manager > Zookeeper > Configuration, locate the configuration field: "Enable JMX Agent", the check box should be already unchecked. If it is not unchecked, make sure to uncheck it and then save.

Step 2:
In Cloudera Manager > Zookeeper > Configuration, locate the following configuration field: Java Configuration Options for Zookeeper Server
You will need to add the following values into this filed separated either by spaces or lines:

-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.ssl=true
-Djavax.net.ssl.keyStore=/opt/cloudera/security/jks/bigdata-host-keystore.jks
-Djavax.net.ssl.keyStorePassword=xxxxxxxxxx
-Dcom.sun.management.jmxremote.ssl.need.client.auth=true
-Djavax.net.ssl.trustStore=/opt/cloudera/security/jks/bigdata-ca-truststore.jks
-Djavax.net.ssl.trustStorePassword=xxxxxxx

Note: Above would though expose plain text keystore and trustore passwords in the configuration, if you want to avoid that, then instead of this set of properties you can specify only the general properties and an additional file location like this:

-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.ssl=true
-Dcom.sun.management.jmxremote.ssl.need.client.auth=true
-Dcom.sun.management.jmxremote.ssl.config.file=/full/path/of/jmxremote.properties.key

and then in the jmxremote.properties.key file can contain the following values:

javax.net.ssl.keyStore=keystore.jks
javax.net.ssl.keyStorePassword=my_keystore_pw
javax.net.ssl.trustStore=truststore.jks
javax.net.ssl.trustStorePassword=my_truststore_pw

This jmxremote.properties.key file can be protected by file system permissions, they still contain plain text password, but unfortunately this is a limitation in the jmx framework in Java we can not overcome. If you choose to use the properties file, it has to be readable by the user who runs the Zookeeper process, usually zookeeper.

Step 3:

In addition to the Zookeeper setup, you will need to setup Service Monitor also to authenticate itself, for this you need to edit the following setting:
Cloudera Manager > Cloudera Management Services > Service Monitor > Configuration, locate the following configuration field: Java Configuration Options for Service Monitor
what you need to add here is the following (must be separated by spaces, using line break for easy reading only):

-Djavax.net.ssl.keyStore=/opt/cloudera/security/jks/bigdata-host-keystore.jks
-Djavax.net.ssl.keyStorePassword=xxxxxxxx
-Djavax.net.ssl.trustStore=/opt/cloudera/security/jks/bigdata-ca-truststore.jks
-Djavax.net.ssl.trustStorePassword=xxxxxxxx

Step 4:
Restart Zookeeper and Service Monitor from Cloudera Manager.