Flume 安裝與實戰

Flume 安裝與實戰

日誌採集框架Flume

一、安裝

• 說明：

首先要安裝 JDK 1.8+ 這裏就不多介紹了

本次安裝 Flume 版本爲 cdh5.7.0

下載地址：CDH5

flume官網：Flume 配置文檔

• 安裝步驟：

下載好後上傳到虛擬機CentOS6.7

解壓: tar -zxvf flume-ng-1.6.0-cdh5.7.0.tar.gz

配置環境變量:

#vi /etc/profile

增加Flume環境變量：

刷新環境變量：source /etc/profile

查看：Flume 版本：flume-ng version

二、Flume 實戰

實戰一：從指定網絡端口採集數據輸出到控制檯

進入 FLUME_HOME/conf/ 目錄添加配置文件 example.conf

參看官網示例：FlumeUserGuide

本次配置文件內容如下：

a1.sources.r1.bind 注意修改爲自己的主機名

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = hadoop1
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory


# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

啓動參數：

• 官網例子：$ bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console

• 按需要修改爲我執行的命令：注意：-Dflume.root.logger=INFO,console 前面只有一個 -

flume-ng agent --name a1 --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/example.conf -Dflume.root.logger=INFO,console

啓動完成：

這時候不要關閉該控制檯，在打開一個終端進行測試：

連接：telnet 主機名 44444

---

實戰二：監控一個文件實時採集新增的數據輸出到控制檯

參看官網Exec Source 的配置：

編寫配置文件：exec-memory-logger.conf

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /home/hadoop/data/data.log
a1.sources.r1.shell = /bin/bash -c

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory


# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

創建要監控的文件：/home/hadoop/data/data.log

啓動Flume: 終端不要關閉

flume-ng agent --name a1 --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/exec-memory-logger.conf -Dflume.root.logger=INFO,console

再開啓一個終端：往 /home/hadoop/data/data.log 文件追加內容：

實戰三：將服務器A上的日誌實時採集到服務器B

技術選型：

exec source + memory channel + avro sink

avro source + memory channel + logger sink

配置文件1: exec-memory-avro.conf Exec Source

exec-memory-avro.sources = exec-source
exec-memory-avro.sinks = avro-sink
exec-memory-avro.channels = memory-channel


exec-memory-avro.sources.exec-source.type = exec
exec-memory-avro.sources.exec-source.command = tail -F /home/hadoop/data/data.log
exec-memory-avro.sources.exec-source.shell = /bin/bash -c

exec-memory-avro.sinks.avro-sink.type = avro
exec-memory-avro.sinks.avro-sink.hostname = hadoop1
exec-memory-avro.sinks.avro-sink.port = 44444


exec-memory-avro.channels.memory-channel.type = memory

exec-memory-avro.sources.exec-source.channels = memory-channel
exec-memory-avro.sinks.avro-sink.channel = memory-channel

配置文件2：avro-memory-logger.conf Avro Source

avro-memory-logger.sources = avro-source
avro-memory-logger.sinks = logger-sink
avro-memory-logger.channels = memory-channel


avro-memory-logger.sources.avro-source.type = avro
avro-memory-logger.sources.avro-source.bind = hadoop1
avro-memory-logger.sources.avro-source.port = 44444

avro-memory-logger.sinks.logger-sink.type = logger


avro-memory-logger.channels.memory-channel.type = memory

avro-memory-logger.sources.avro-source.channels = memory-channel
avro-memory-logger.sinks.logger-sink.channel = memory-channel

先啓動：avro-memory-logger

flume-ng agent --name avro-memory-logger --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/avro-memory-logger.conf -Dflume.root.logger=INFO,console

再啓動：exec-memory-avro

flume-ng agent --name exec-memory-avro --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/exec-memory-avro.conf -Dflume.root.logger=INFO,console