Fixed Log4j RCE Dec 10 2021 for Hadoop Cluster using CDH

cloudera_6.3.2_log4j-rce.md

Fixed Log4j RCE Dec 10 2021 for Hadoop Cluster using CDH

Info:

• Code: CVE-2021-44228
• https://nvd.nist.gov/vuln/detail/CVE-2021-44228
• https://bishopfox.com/blog/log4j-zero-day-cve-2021-44228-
• Cheatsheet: https://gist.github.com/SwitHak/b66db3a06c2955a9cb71a8718970c592

Show log4j version on Namenode, Datanode, Resource Manager, NodeManager

lsof | grep log4j

Result:

java       7277                  hdfs  mem       REG             252,16   228154   11799146 /data/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/log4j-api-2.8.2.jar

hdfs      7167 10857  0 Sep21 ?        00:00:01 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm proc_watcher 7277
hdfs      7277  7167  1 Sep21 ?        1-01:17:03 /usr/lib/jvm/java-8-oracle-cloudera/bin/java -Dproc_datanode -Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true -Xms1073741824 -Xmx1073741824 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/tmp/hdfs3_hdfs3-DATANODE-4adf850f1c117a1012a4eb81d336811f_pid7277.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh -Dyarn.log.dir=/data/var/log/hadoop-hdfs -Dyarn.log.file=hadoop-cmf-hdfs3-DATANODE-adt-sys-hbase-kylin-lab-test-94-149.log.out -Dyarn.home.dir=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-yarn -Dyarn.root.logger=INFO,console -Djava.library.path=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/lib/native -Dhadoop.log.dir=/data/var/log/hadoop-hdfs -Dhadoop.log.file=hadoop-cmf-hdfs3-DATANODE-adt-sys-hbase-kylin-lab-test-94-149.log.out -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.datanode.DataNode

Solution

Solution 1: Edit script start Hadoop and YARN:

File: /opt/cloudera/cm-agent/service/yarn/yarn.sh
103:export YARN_OPTS="-Dlog4j2.formatMsgNoLookups=true -Djava.net.preferIPv4Stack=true $YARN_OPTS"
-
File: /opt/cloudera/cm-agent/service/hdfs/hdfs.sh
711:export HADOOP_OPTS="-Dlog4j2.formatMsgNoLookups=true -Dsecurity.audit.logger=$HADOOP_SECURITY_LOGGER $HADOOP_OPTS"

Solution 2: Add a variable on Cloudera Manager

• Login to Cloudera Manager
• Search: Environment Advanced Configuration Snippet (Safety Valve)
• Add variable
  • Hadoop: HADOOP_OPTS="-Dlog4j2.formatMsgNoLookups=true"
  • YARN: YARN_OPTS="-Dlog4j2.formatMsgNoLookups=true"
  • HBase: HBASE_OPTS="-Dlog4j2.formatMsgNoLookups=true"