Flume (七) Monitoring

Flume中的監控仍在進行中。 變化可能經常發生。 幾個Flume組件向JMX平臺MBean服務器報告度量標準。 可以使用Jconsole查詢這些指標。

JMX Reporting

可以通過使用flume-env.sh在JAVA_OPTS環境變量中指定JMX參數來啓用JMX報告，如

export JAVA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5445 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

注意：上面的示例禁用安全性。 要啓用安全性，請參閱
http://docs.oracle.com/javase/6/docs/technotes/guides/management/agent.html

Ganglia Reporting

Flume can also report these metrics to Ganglia 3 or Ganglia 3.1 metanodes. To report metrics to Ganglia, a flume agent must be started with this support. The Flume agent has to be started by passing in the following parameters as system properties prefixed by flume.monitoring., and can be specified in the flume-env.sh:

Property Name	Default	Description
type	–	組件類型名稱必須是ganglia
hosts	–	以逗號分隔Ganglia服務器的hostname:port
pollFrequency	60	連續向Ganglia服務器報告之間的時間（以秒爲單位）
isGanglia3	false	Ganglia服務器版本爲3.默認情況下，Flume以Ganglia 3.1格式發送

我們可以使用Ganglia支持啓動Flume，如下所示：

$ bin/flume-ng agent --conf-file example.conf --name a1 -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=com.example:1234,com.example2:5455

JSON Reporting

Flume還可以以JSON格式報告指標。 爲了以JSON格式啓用報告，Flume在可配置端口上託管Web服務器。 Flume以以下JSON格式報告指標：

{
"typeName1.componentName1" : {"metric1" : "metricValue1", "metric2" : "metricValue2"},
"typeName2.componentName2" : {"metric3" : "metricValue3", "metric4" : "metricValue4"}
}

Here is an example:

{
"CHANNEL.fileChannel":{"EventPutSuccessCount":"468085",
                      "Type":"CHANNEL",
                      "StopTime":"0",
                      "EventPutAttemptCount":"468086",
                      "ChannelSize":"233428",
                      "StartTime":"1344882233070",
                      "EventTakeSuccessCount":"458200",
                      "ChannelCapacity":"600000",
                      "EventTakeAttemptCount":"458288"},
"CHANNEL.memChannel":{"EventPutSuccessCount":"22948908",
                   "Type":"CHANNEL",
                   "StopTime":"0",
                   "EventPutAttemptCount":"22948908",
                   "ChannelSize":"5",
                   "StartTime":"1344882209413",
                   "EventTakeSuccessCount":"22948900",
                   "ChannelCapacity":"100",
                   "EventTakeAttemptCount":"22948908"}
}

Property Name	Default	Description
type	–	組件類型名稱必須是http
port	41414	啓動服務器的端口。

We can start Flume with JSON Reporting support as follows:

$ bin/flume-ng agent --conf-file example.conf --name a1 -Dflume.monitoring.type=http -Dflume.monitoring.port=34545

然後，可以在http://<hostname>:<port>/metrics網頁上獲得度量標準。 自定義組件可以報告上面Ganglia部分中提到的指標。

Custom Reporting

可以通過編寫執行報告的服務器向其他系統報告度量標準。 任何報告類都必須實現org.apache.flume.instrumentation.MonitorService接口。 這樣的類可以與GangliaServer用於報告的方式相同。 他們可以輪詢平臺mbean服務器以輪詢mbeans以獲取指標。 例如，如果可以使用名爲HTTPReporting的HTTP監視服務，如下所示：

$ bin/flume-ng agent --conf-file example.conf --name a1 -Dflume.monitoring.type=com.example.reporting.HTTPReporting -Dflume.monitoring.node=com.example:332

Property Name	Default	Description
type	–	組件類型名稱必須是FQCN

Reporting metrics from custom components

任何自定義flume組件都應繼承自org.apache.flume.instrumentation.MonitoredCounterGroup類。 然後，該類應爲其公開的每個度量標準提供getter方法。 請參閱下面的代碼。 MonitoredCounterGroup需要一個屬性列表，其度量由此類公開。 截至目前，此類僅支持將指標公開爲long 。

public class SinkCounter extends MonitoredCounterGroup implements
    SinkCounterMBean {

  private static final String COUNTER_CONNECTION_CREATED =
    "sink.connection.creation.count";

  private static final String COUNTER_CONNECTION_CLOSED =
    "sink.connection.closed.count";

  private static final String COUNTER_CONNECTION_FAILED =
    "sink.connection.failed.count";

  private static final String COUNTER_BATCH_EMPTY =
    "sink.batch.empty";

  private static final String COUNTER_BATCH_UNDERFLOW =
      "sink.batch.underflow";

  private static final String COUNTER_BATCH_COMPLETE =
    "sink.batch.complete";

  private static final String COUNTER_EVENT_DRAIN_ATTEMPT =
    "sink.event.drain.attempt";

  private static final String COUNTER_EVENT_DRAIN_SUCCESS =
    "sink.event.drain.sucess";

  private static final String[] ATTRIBUTES = {
    COUNTER_CONNECTION_CREATED, COUNTER_CONNECTION_CLOSED,
    COUNTER_CONNECTION_FAILED, COUNTER_BATCH_EMPTY,
    COUNTER_BATCH_UNDERFLOW, COUNTER_BATCH_COMPLETE,
    COUNTER_EVENT_DRAIN_ATTEMPT, COUNTER_EVENT_DRAIN_SUCCESS
  };


  public SinkCounter(String name) {
    super(MonitoredCounterGroup.Type.SINK, name, ATTRIBUTES);
  }

  @Override
  public long getConnectionCreatedCount() {
    return get(COUNTER_CONNECTION_CREATED);
  }

  public long incrementConnectionCreatedCount() {
    return increment(COUNTER_CONNECTION_CREATED);
  }

}