Flink是什麼?
Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale. (官網原文)
Flink是一個框架、分佈式的處理引擎。主要用於對無限制和有限制的數據流進行有狀態的計算。在Flink中,一切都是由流組成:離線數據是有限制的流;實時數據時無限制的流。Flink最大的優點就是低延遲、高吞吐且可以保證精準一次性的狀態一致性,他是一個真真正正的流式處理框架。
下面使用scala編寫flink批處理版本和flink流式處理版本的WordCount程序:
1、新建maven工程,導入依賴
<dependencies>
<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-scala -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-scala_2.11</artifactId>
<version>1.7.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-scala -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-scala_2.11</artifactId>
<version>1.7.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-clients -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.11</artifactId>
<version>1.7.2</version>
</dependency>
</dependencies>
<build>
<plugins>
<!-- 該插件用於將Scala代碼編譯成class文件 -->
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.4.6</version>
<executions>
<execution>
<!-- 聲明綁定到maven的compile階段 -->
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
2、編寫批處理版本wordcount
package com.wei.scala
import org.apache.flink.api.scala._
object WordCount {
/**
* 批處理
* @param args
*/
def main(args: Array[String]): Unit = {
//創建執行環境
val env: ExecutionEnvironment = ExecutionEnvironment.getExecutionEnvironment
//讀取數據文件
val dataSet = env.readTextFile("C:\\idea\\Flink\\src\\main\\resources\\hello.txt")
//切分文本內容
val result =dataSet.flatMap(_.split(" "))
.map((_,1))
.groupBy(0)
.sum(1)
result.print()
}
}
3、編寫流式處理版本wordcount
package com.wei.scala
import org.apache.flink.streaming.api.scala._
object WordCount2 {
def main(args: Array[String]): Unit = {
//創建執行環境
val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment
//接收一個socket文本流
val dataStream: DataStream[String] = env.socketTextStream("hdp-1",9999)
//對流入的數據進行處理
val result: DataStream[(String, Int)] = dataStream.flatMap(_.split(" ")).filter(_.nonEmpty).map((_,1)).keyBy(0).sum(1)
//輸出結果
result.print()
//啓動executor
env.execute("my first wordCount job")
}
}
在虛擬機上啓動一個netcat端口,用於發送數據,程序利用socketTextStream接收一個文本流,經過處理並且輸出結果: