Apache Flume 是一個分佈式、可靠的、可用的服務，應用於有效地收集、聚合以及移動大量的日誌文件。
接下來我們將來介紹配置Flume，使得SparkStreaming可以去接收來自Flume的數據。注意Spark Streaming 與Flume版本的兼容性。
一、Push方式整合
概念
Flume is designed to push data between Flume agents. In this approach, Spark Streaming essentially sets up a receiver
that acts an Avro agent for Flume, to which Flume can push the data. Here are the configuration steps.
這是官網上的一段解釋，我大概通俗翻譯下就是：
Flume的設計是爲了在Flume agents中進行推送數據，我們可以配置多個agent，可以是串聯的，也可以是並聯的，有了agent後，我們就可以在angent之間做數據推送。在Push方式的整合，Spark Streaming 必須建立一個receiver（一旦看到receiver就要注意在本地測試的時候，local一定要大於1，否則就沒有多餘的線程），這個receiver作用類似於Flume中的Avro agent，這樣的話Flume就可以對Spark Streaming進行推送數據。
General Requirements
Choose a machine in your cluster such that
-
When your Flume + Spark Streaming application is launched, one of the Spark workers must run on that machine.
-
Flume can be configured to push data to a port on that machine.
Due to the push model, the streaming application needs to be up, with the receiver scheduled and listening on the chosen port, for Flume to be able push data.
另外，這段話也是很重要的，大體意思是：首先從你的集羣中選出一臺機器，當你的Flume和Spark Streaming的應用程序啓動以後，你的Spark的worker中的一個work必須在選中的那臺機器上運行，即和Flume配置到同一個節點上面。
Flume可以被配置去推送數據到一個端口之上，因爲是用的Avro方式，所以需要指定一個端口。
在Push模式下，Spark Streaming需要先啓動，Spark Streaming中的receiver會定時地調度和監聽這個端口，然後Flume就可以推送數據了。
配置環境
Flume的配置文件：
Configure Flume agent to send data to an Avro sink by having the following in the configuration file.
agent.sinks = avroSink
agent.sinks.avroSink.type = avro
agent.sinks.avroSink.channel = memoryChannel
agent.sinks.avroSink.hostname = <chosen machine's hostname>
agent.sinks.avroSink.port = <chosen port on the machine>
Flume Agent的編寫： flume_push_streaming.conf
simple-agent.sources = netcat-source
simple-agent.sinks = avro-sink
simple-agent.channels = memory-channel
simple-agent.sources.netcat-source.type = netcat
simple-agent.sources.netcat-source.bind = hadoop000
simple-agent.sources.netcat-source.port = 44444
simple-agent.sinks.avro-sink.type = avro
simple-agent.sinks.avro-sink.hostname = 192.168.199.203
simple-agent.sinks.avro-sink.port = 41414
simple-agent.channels.memory-channel.type = memory
simple-agent.sources.netcat-source.channels = memory-channel
simple-agent.sinks.avro-sink.channel = memory-channel應用的開發與部署
配置完後，進行應用程序的開發：1.首先在Maven項目中引入如下的依賴
groupId = org.apache.spark
artifactId = spark-streaming-flume_2.11
version = 2.2.0
import org.apache.spark.streaming.flume._
val flumeStream = FlumeUtils.createStream(streamingContext, [chosen machine's hostname], [chosen port])
import org.apache.spark.SparkConf
import org.apache.spark.streaming.flume.FlumeUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}
/**
* Spark Streaming整合Flume的第一種方式
*/
object FlumePushWordCount {
def main(args: Array[String]): Unit = {
if(args.length != 2) {
System.err.println("Usage: FlumePushWordCount <hostname> <port>")
System.exit(1)
}
val Array(hostname, port) = args
val sparkConf = new SparkConf() //.setMaster("local[2]").setAppName("FlumePushWordCount")
val ssc = new StreamingContext(sparkConf, Seconds(5))
//TODO... 如何使用SparkStreaming整合Flume
val flumeStream = FlumeUtils.createStream(ssc, hostname, port.toInt)
flumeStream.map(x=> new String(x.event.getBody.array()).trim)
.flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).print()
ssc.start()
ssc.awaitTermination()
}
}接下來，用maven命令打成jar包，打包命令如下:
mvn clean package -DskipTests
3.對Spark應用程序進行部署
部署命令如下:
spark-submit \
--class com.imooc.spark.FlumePushWordCount \
--master local[2] \
--packages org.apache.spark:spark-streaming-flume_2.11:2.2.0 \
/home/hadoop/lib/sparktrain-1.0.jar \
hadoop000 41414
4.啓動Flume
啓動Flume的命令如下：
flume-ng agent \
--name simple-agent \
--conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/flume_push_streaming.conf \
-Dflume.root.logger=INFO,console5.測試
開啓個窗口，輸入 telnet hadoop 44444，然後隨意輸入些字母。
然後我們在sink配置的那臺服務器上的41414上可以觀察到如下：
本地測試總結
1）啓動sparkstreaming作業2 ) 啓動flume agent
3 ) 通過telnet輸入數據，觀察IDEA控制檯的輸出