使用Log4j將日誌實時寫入Flume

需求：使用Flume實時收集WebServer或者其它服務器上通過log4j產生的日誌

Agent選型：Log4jAppender的使用

創建一個maven項目，pom.xml文件中引入依賴

<dependency>
    <groupId>log4j</groupId>
    <artifactId>log4j</artifactId>
    <version>1.2.16</version>
</dependency>

<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-log4j12</artifactId>
    <version>1.7.21</version>
    <scope>test</scope>
</dependency>

在src/main/resource目錄下新建log4j.properties，配置如下

log4j.rootLogger=INFO,stdout
log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.target = System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} [%t] [%c] [%p] - %m%n

將resource目錄加載到classpath下

選擇左側Modules，找到resource目錄，點擊上面的Resources

新建LoggerGenerate類，用於產生日誌（模仿項目中的日誌輸出）

import org.apache.log4j.Logger;

/**
 * 循環打印值，模仿日誌輸出
 */
public class LoggerGenerate {

    private static Logger logger = Logger.getLogger(LoggerGenerate.class.getName());

    public static void main(String[] args) throws InterruptedException {
        int index = 0;
        while (true) {
            Thread.sleep(1000);
            logger.info("value is：" + index++);
        }
    }
}

運行測試（部分結果）：

2017-9-25 09:44:45,238 [main] [LoggerGenerate] [INFO] - value is：0
2017-9-25 09:44:46,241 [main] [LoggerGenerate] [INFO] - value is：1
2017-9-25 09:44:47,241 [main] [LoggerGenerate] [INFO] - value is：2
2017-9-25 09:44:48,241 [main] [LoggerGenerate] [INFO] - value is：3
2017-9-25 09:44:49,255 [main] [LoggerGenerate] [INFO] - value is：4
2017-9-25 09:44:50,258 [main] [LoggerGenerate] [INFO] - value is：5

目前這些日誌信息都打印在了控制檯，接下來我們想要把這些日誌收集到flume，首先先配置Agent

# Name the components on this agent
flume-log-agent.sources = arvo-source
flume-log-agent.sinks = logger-sink
flume-log-agent.channels = memory-channel

# Describe/configure the source
flume-log-agent.sources.arvo-source.type = avro
flume-log-agent.sources.arvo-source.bind = Master
flume-log-agent.sources.arvo-source.port = 44444

# Describe the sink
flume-log-agent.sinks.logger-sink.type = logger

# Use a channel which buffers events in memory
flume-log-agent.channels.memory-channel.type = memory

# Bind the source and sink to the channel
flume-log-agent.sources.arvo-source.channels = memory-channel
flume-log-agent.sinks.logger-sink.channel = memory-channel

啓動Agent

flume-ng agent \
--name flume-log-agent \
--conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/config/flume-log-agent.conf \
-Dflume.root.logger=INFO,console

想要將控制檯的日誌收集到flume，需要修改log4j.properties中的配置
參考官網：http://flume.apache.org/FlumeUserGuide.html

log4j.properties中新增內容如下配置

log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Hostname = Master
log4j.appender.flume.Port = 44444
log4j.appender.flume.UnsafeMode = true

在 log4j.rootLogger=INFO,stdout 後面加上flume

log4j.rootLogger=INFO,stdout,flume

運行測試，筆者的項目是在windows下創建的（你也可以將項目部署在linux服務器中），需要在windows的hosts文件中配置映射，否者在log4j中是不可以寫主機名的，只能寫ip地址，參照下圖配置即可

在hosts文件中添加一行內容（格式：IP 主機名）

192.168.242.150 Master

在運行期間，報了一個錯誤，如下
java.lang.ClassNotFoundException: org.apache.flume.clients.log4jappender.Log4jAppender

報錯的原因是類找不到，很明顯是項目中沒有添加依賴，在pom.xml文件中添加flume-ng-log4jappender的依賴

<dependency>
    <groupId>org.apache.flume.flume-ng-clients</groupId>
    <artifactId>flume-ng-log4jappender</artifactId>
    <version>1.6.0</version>
</dependency>

再次運行項目，查看Agent控制檯中打印的信息

2017-09-25 10:16:12,862 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926512143, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 30             value is...0 }
2017-09-25 10:16:12,863 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926513197, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 31             value is...1 }
2017-09-25 10:16:12,864 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926514203, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 32             value is...2 }
2017-09-25 10:16:13,082 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926515210, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 33             value is...3 }
2017-09-25 10:16:14,092 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926516221, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 34             value is...4 }
2017-09-25 10:16:15,103 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926517228, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 35             value is...5 }

總結：在Agent的配置文件編寫過程中，常用的參數設置，無需死記硬背（參照官網即可）