Flume(分佈式日誌收集框架)
|
|-主要作用:把數據從各種各樣的server上移動到Kafka/HDFS/HIVE等分佈式存儲組件上;
|
|-組件示意圖
| \ _____________________________________________________
| | |
| | _____Agent_______________ |
| | | | |
| | |---------> Source Sink-------| |
| | / | \ /|\ | \|/ |
| | Web Server | \-->Channel--| | HDFS |
| | | | |
| | |_________________________| |
| |_____________________________________________________|
|
|-組件|--1.Source 收集
| |--2.Channel 聚集
| |--3.Sink 輸出
|
|-flume配置|--文件名:flume-env.sh
| |--添加內容：export JAVA_HOME = ~/jdk1.7(or later)
|
|-寫flume實例配置
| |
| |--關鍵點|---1.配置Source
| | |---2.配置Channel
| | |---3.配置Sink
| | |---4.把以上三個組件串起來
| |
| |--配置步驟 |---1.指定Agent(實例)名稱,如a1
| | |
| | |---2.指定組件名稱,如|---a1.sources = r1
| | | |---a1.sinks = k1
| | | |---a1.channels = c1
| | |
| | |---3.配置Source,如|---a1.sources.r1.type = netcat
| | | |---a1.sources.r1.bind = hadoop001
| | | |---a1.sources.r1.port = 44444
| | |
| | |---4.配置Channel,如|---a1.channels.c1.type = memory
| | | |---a1.channels.c1.capacity = 1000
| | | |---a1.channels.c1.transactionCapacity = 100
| | |
| | |---5.配置Sink,如: a1.sinks.k1.type = logger
| | |
| | |---6.綁定Channel到Source、Sink|---a1.sources.r1.channels = c1
| | | |---a1.sinks.k1.channel = c1
|
|-啓動flume agent |--~/bin> flume-ng agent \
| (flume 實例) | --name a1 \
| | --conf $FLUME_HOME/conf \
| | --conf-file $FLUME_HOME/conf/a1.conf \
| | --Dflume.root.logger=INFO,console
|
|-測試:登錄hadoop001,進入telnet(telnet hadoop001 44444)生產內容,觀察flume輸出
扇入扇出架構圖
官方用戶手冊
http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html
User Guide
|
|-Introduction
|
|-Setup
|
|-Config|--Flume Sources|---Avro Source
| | |---Thrift Source
| | |---Exec Source
| | |---JMS Source
| | |---Kafka Source
| | |---...
| |
| |--Flume Channels|---Memory Channel
| | |---JDBC Channel
| | |---Kafka Channel
| | |---FIle Channel
| |
| |--Flume Sinks|---HDFS Sink
| | |---Hive Sink
| | |---Logger Sink
| | |---Avro Sink
| | |---ElasticSearch Sink
| | |---...
| |
| |--...
|
|-...