cdh集羣出現“無法發出查詢：Host Monitor 未運行”

cdh集羣出現“無法發出查詢：Host Monitor 未運行”

出現問題後，首先想到查看日誌：

/var/log/cloudera-scm-server/cloudera-scm-server.log

2018-06-01 11:58:00,001 INFO 618348787@agentServer-2579:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=1ms.
2018-06-01 11:58:24,208 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: (119 skipped) ScmActive completed successfully.
2018-06-01 11:58:38,291 ERROR DatabaseSizeGauge-0:com.cloudera.enterprise.DatabaseSizeGauge: Failed to execute db size query.
java.lang.NullPointerException
        at com.cloudera.enterprise.dbutil.DbUtil.getDatabaseSize(DbUtil.java:736)
        at com.cloudera.enterprise.DatabaseSizeGauge.run(DatabaseSizeGauge.java:75)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
2018-06-01 11:58:59,293 INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: Failed to send metrics.
java.lang.ArrayIndexOutOfBoundsException: 0

然後查看了一下磁盤使用情況：

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_bjesbpmaster-lv_root
                       50G   50G     0 100% /
tmpfs                  16G   68K   16G   1% /dev/shm
/dev/sda2             485M   69M  392M  15% /boot
/dev/sda1             200M  9.9M  190M   5% /boot/efi
/dev/mapper/vg_bjesbpmaster-lv_data
                      3.6T   33G  3.6T   1% /data
cm_processes           16G   38M   16G   1% /var/run/cloudera-scm-agent/process

根目錄被佔滿。哎~~~~

解決思路：

1、先解決根目錄磁盤滿的問題；

2、嘗試一次重啓Cloudera Management Service，發現到一個行重要日誌：

Removing any leveldbjni library files left over from previous runs

找下面兩個文件：

cloudera-host-monitor      
cloudera-service-monitor

我電腦上這兩個文件夾放到了/data/var/lib目錄下，然後將這兩個文件夾改名備份一下，

mv cloudera-host-monitor/ cloudera-host-monitor_BAK
mv  cloudera-service-monitor  cloudera-service-monitor_BAK

在次嘗試重啓Cloudera Management Service，等待幾秒鐘後，啓動成功：

到此，問題解決完成。