目錄
目錄
(2) 使用Manage State,Flink自動實現state保存和恢復
(3) 自定義state 自行實現實現checkpoint接口
背景
Flink相對於Storm和Spark Stream比較大的一個優勢就是State,pipline中可以保存狀態,這對於解決業務是有巨大的幫助,否則將需要藉助三方工具來時間狀態的存儲和訪問。
正文
State
什麼是State(狀態)?
- 某task/operator在某時刻的一箇中間結果
- 快照(shapshot)
- 在flink中狀態可以理解爲一種數據結構
- 舉例
對輸入源爲<key,value>的數據,計算其中某key的最大值,如果使用HashMap,也可以進行計算,但是每次都需要重新遍歷,使用狀態的話,可以獲取最近的一次計算結果,減少了系統的計算次數 - 程序一旦crash,恢復
- 程序擴容
State類型
總的來說,state分爲兩種,operator state和key state,key state專門對keystream使用,所包含的Sate種類也更多,可理解爲dataStream.keyBy()之後的Operator State,Operator State是對每一個Operator的狀態進行記錄,而key State則是在dataSteam進行keyBy()後,記錄相同keyId的keyStream上的狀態key State提供的數據類型:ValueState<T>、ListState<T>、ReducingState<T>、MapState<T>。
operator state種類只有一種就是ListState<T> ,flink官方文檔用kafka的消費者舉例,認爲kafka消費者的partitionId和offset類似flink的operator state
State理解
state分爲operator state和key state兩種,都屬於manage state,優勢是可以結合checkpoint,實現自動存儲狀態和異常恢復功能,但是state不一定要使用manage state,在source、windows和sink中自己聲明一個int都可以作爲狀態進行使用,只不過需要自己實現快照狀態保存和恢復。
State實戰
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.state.ValueState;
import org.apache.flink.api.common.state.ValueStateDescriptor;
import org.apache.flink.api.common.typeinfo.TypeHint;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;
/**
* Created by authoe on 2018/9/12.
*/
public class StateTest {
public static void main(String[] args) throws Exception{
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(4).fromElements(Tuple2.of(1L,1L),Tuple2.of(2L,2L),Tuple2.of(3L,2L),Tuple2.of(4L,3L))
.keyBy(0).flatMap(new RichFlatMapFunction<Tuple2<Long,Long>, Object>() {
private transient ValueState<Tuple2<Long,Long>> sum;
@Override
public void flatMap(Tuple2<Long, Long> longLongTuple2, Collector<Object> collector) throws Exception {
Tuple2<Long,Long> curSum = sum.value();
curSum.f0+=1;
curSum.f1+=longLongTuple2.f1;
System.out.println("+");
sum.update(curSum);
if(curSum.f0>0){
System.out.println("-");
collector.collect(Tuple2.of(curSum.f0,curSum.f1));
sum.clear();
}
}
@Override
public void open(Configuration parameters) throws Exception {
ValueStateDescriptor<Tuple2<Long,Long>> descriptor = new ValueStateDescriptor<Tuple2<Long, Long>>("avg", TypeInformation.of(
new TypeHint<Tuple2<Long, Long>>() {}),Tuple2.of(0L,0L));
sum = getRuntimeContext().getState(descriptor);
}
}).print();
env.execute();
}
}
CheckPointing
(1)介紹,實現方式分類
checkpoint可以保存窗口和算子的執行狀態,在出現異常之後重啓計算任務,並保證已經執行和不會再重複執行,檢查點可以分爲兩種,託管的和自定義的,託管檢查點會自動的進行存儲到指定位置:內存、磁盤和分佈式存儲中,自定義就需要自行實現保存相關,實現checkpoint有如下兩種方式:
- 使用託管State變量
- 使用自定義State變量實現CheckpointedFunction接口或者ListCheckpoint<T extends Serializable>接口
下面將會給出兩種方式的使用代碼
(2) 使用Manage State,Flink自動實現state保存和恢復
下面先給出託管狀態變量(manage stata)使用代碼,後面給出了代碼執行的打印日誌。
代碼分析:
- 代碼每隔2s發送10條記錄,所有數據key=1,會發送到統一窗口進行計數,發送超過100條是,拋出異常,模擬異常
- 窗口中統計收到的消息記錄數,當異常發生時,查看windows是否會從state中獲取歷史數據,即checkpoint是否生效
- 註釋已經添加進代碼中,有個問題有時,state.value()在open()方法中調用的時候,會拋出null異常,而在apply中 使用就不會拋出異常。
Console日誌輸出分析:
- 四個open調用,因爲我本地代碼默認並行度爲4,所以會有4個windows函數被實例化出來,調用各自open函數
- source發送記錄到達100拋出異常
- source拋出異常之後,count發送統計數丟失,重新從0開始
- windows函數,重啓後調用open函數,獲取state數據,處理記錄數從checkpoint中獲取恢復,所以從100開始
總結:
source沒有使用manage state狀態丟失,windows使用manage state,異常狀態不丟失
package per.test;
import org.apache.flink.api.common.state.ValueState;
import org.apache.flink.api.common.state.ValueStateDescriptor;
import org.apache.flink.api.common.typeinfo.TypeHint;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.java.tuple.Tuple;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.runtime.state.filesystem.FsStateBackend;
import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
import org.apache.flink.streaming.api.functions.windowing.RichWindowFunction;
import org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
import org.apache.flink.util.Collector;
/**
* Created by betree on 2018/9/14.
*/
public class StateCheckPoint {
public static void main(String[] args) throws Exception{
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
//打開並設置checkpoint
// 1.設置checkpoint目錄,這裏我用的是本地路徑,記得本地路徑要file開頭
// 2.設置checkpoint類型,at lease onece or EXACTLY_ONCE
// 3.設置間隔時間,同時打開checkpoint功能
//
env.setStateBackend(new FsStateBackend("file:///Users/username/Documents/程序數據/Flink/state_checkpoint/"));
env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
env.getCheckpointConfig().setCheckpointInterval(1000);
//添加source 每個2s 發送10條數據,key=1,達到100條時候拋出異常
env.addSource(new SourceFunction<Tuple3<Integer,String,Integer>>() {
private Boolean isRunning = true;
private int count = 0;
@Override
public void run(SourceContext<Tuple3<Integer, String, Integer>> sourceContext) throws Exception {
while(isRunning){
for (int i = 0; i < 10; i++) {
sourceContext.collect(Tuple3.of(1,"ahah",count));
count++;
}
if(count>100){
System.out.println("err_________________");
throw new Exception("123");
}
System.out.println("source:"+count);
Thread.sleep(2000);
}
}
@Override
public void cancel() {
}
}).keyBy(0)
.window(TumblingProcessingTimeWindows.of(Time.seconds(2)))
//窗口函數,比如是richwindowsfunction 否側無法使用manage state
.apply(new RichWindowFunction<Tuple3<Integer,String,Integer>, Integer, Tuple, TimeWindow>() {
private transient ValueState<Integer> state;
private int count = 0;
@Override
public void apply(Tuple tuple, TimeWindow timeWindow, Iterable<Tuple3<Integer, String, Integer>> iterable, Collector<Integer> collector) throws Exception {
//從state中獲取值
count=state.value();
for(Tuple3<Integer, String, Integer> item : iterable){
count++;
}
//更新state值
state.update(count);
System.out.println("windows:"+tuple.toString()+" "+count+" state count:"+state.value());
collector.collect(count);
}
//獲取state
@Override
public void open(Configuration parameters) throws Exception {
System.out.println("##open");
ValueStateDescriptor<Integer> descriptor =
new ValueStateDescriptor<Integer>(
"average", // the state name
TypeInformation.of(new TypeHint<Integer>() {}), // type information
0);
state = getRuntimeContext().getState(descriptor);
}
}).print();
env.execute();
}
}
給出日誌打印結果:
/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/bin/java "-javaagent:/Applications/IntelliJ IDEA CE.app/Contents/lib/idea_rt.jar=59422:/Applications/IntelliJ IDEA CE.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/lib/tools.jar:/Users/wangqi/Documents/code/workplace/flink_t/target/classes:/Users/wangqi/.m2/repository/org/apache/flink/flink-java/1.6.0/flink-java-1.6.0.jar:/Users/wangqi/.m2/repository/org/apache/flink/flink-core/1.6.0/flink-core-1.6.0.jar:/Users/wangqi/.m2/repository/org/apache/flink/flink-annotations/1.6.0/flink-annotations-1.6.0.jar:/Users/wangqi/.m2/repository/org/apache/flink/flink-metrics-core/1.6.0/flink-metrics-core-1.6.0.jar:/Users/wangqi/.m2/repository/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.jar:/Users/wangqi/.m2/repository/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar:/Users/wangqi/.m2/repository/org/objenesis/objenesis/2.1/objenesis-2.1.jar:/Users/wangqi/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/wangqi/.m2/repository/org/apache/commons/commons-compress/1.4.1/commons-compress-1.4.1.jar:/Users/wangqi/.m2/repository/org/tukaani/xz/1.0/xz-1.0.jar:/Users/wangqi/.m2/repository/org/apache/flink/flink-shaded-asm/5.0.4-4.0/flink-shaded-asm-5.0.4-4.0.jar:/Users/wangqi/.m2/repository/org/apache/commons/commons-lang3/3.3.2/commons-lang3-3.3.2.jar:/Users/wangqi/.m2/repository/org/apache/commons/commons-math3/3.5/commons-math3-3.5.jar:/Users/wangqi/.m2/repository/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.jar:/Users/wangqi/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/wangqi/.m2/repository/org/apache/flink/force-shading/1.6.0/force-shading-1.6.0.jar:/Users/wangqi/.m2/repository/org/apache/flink/flink-streaming-java_2.11/1.6.0/flink-streaming-java_2.11-1.6.0.jar:/Users/wangqi/.m2/repository/org/apache/flink/flink-runtime_2.11/1.6.0/flink-runtime_2.11-1.6.0.jar:/Users/wangqi/.m2/repository/org/apache/flink/flink-queryable-state-client-java_2.11/1.6.0/flink-queryable-state-client-java_2.11-1.6.0.jar:/Users/wangqi/.m2/repository/org/apache/flink/flink-hadoop-fs/1.6.0/flink-hadoop-fs-1.6.0.jar:/Users/wangqi/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/Users/wangqi/.m2/repository/org/apache/flink/flink-shaded-netty/4.1.24.Final-4.0/flink-shaded-netty-4.1.24.Final-4.0.jar:/Users/wangqi/.m2/repository/org/apache/flink/flink-shaded-jackson/2.7.9-4.0/flink-shaded-jackson-2.7.9-4.0.jar:/Users/wangqi/.m2/repository/org/javassist/javassist/3.19.0-GA/javassist-3.19.0-GA.jar:/Users/wangqi/.m2/repository/org/scala-lang/scala-library/2.11.12/scala-library-2.11.12.jar:/Users/wangqi/.m2/repository/com/typesafe/akka/akka-actor_2.11/2.4.20/akka-actor_2.11-2.4.20.jar:/Users/wangqi/.m2/repository/com/typesafe/config/1.3.0/config-1.3.0.jar:/Users/wangqi/.m2/repository/org/scala-lang/modules/scala-java8-compat_2.11/0.7.0/scala-java8-compat_2.11-0.7.0.jar:/Users/wangqi/.m2/repository/com/typesafe/akka/akka-stream_2.11/2.4.20/akka-stream_2.11-2.4.20.jar:/Users/wangqi/.m2/repository/org/reactivestreams/reactive-streams/1.0.0/reactive-streams-1.0.0.jar:/Users/wangqi/.m2/repository/com/typesafe/ssl-config-core_2.11/0.2.1/ssl-config-core_2.11-0.2.1.jar:/Users/wangqi/.m2/repository/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.4/scala-parser-combinators_2.11-1.0.4.jar:/Users/wangqi/.m2/repository/com/typesafe/akka/akka-protobuf_2.11/2.4.20/akka-protobuf_2.11-2.4.20.jar:/Users/wangqi/.m2/repository/com/typesafe/akka/akka-slf4j_2.11/2.4.20/akka-slf4j_2.11-2.4.20.jar:/Users/wangqi/.m2/repository/org/clapper/grizzled-slf4j_2.11/1.0.2/grizzled-slf4j_2.11-1.0.2.jar:/Users/wangqi/.m2/repository/com/github/scopt/scopt_2.11/3.5.0/scopt_2.11-3.5.0.jar:/Users/wangqi/.m2/repository/org/xerial/snappy/snappy-java/1.1.4/snappy-java-1.1.4.jar:/Users/wangqi/.m2/repository/com/twitter/chill_2.11/0.7.4/chill_2.11-0.7.4.jar:/Users/wangqi/.m2/repository/com/twitter/chill-java/0.7.4/chill-java-0.7.4.jar:/Users/wangqi/.m2/repository/org/apache/flink/flink-shaded-guava/18.0-4.0/flink-shaded-guava-18.0-4.0.jar:/Users/wangqi/.m2/repository/org/apache/flink/flink-clients_2.11/1.6.0/flink-clients_2.11-1.6.0.jar:/Users/wangqi/.m2/repository/org/apache/flink/flink-optimizer_2.11/1.6.0/flink-optimizer_2.11-1.6.0.jar:/Users/wangqi/.m2/repository/commons-cli/commons-cli/1.3.1/commons-cli-1.3.1.jar per.test.StateCheckPoint
objc[34882]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/bin/java (0x10b8bc4c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x10b9404e0). One of the two will be used. Which one is undefined.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
source:10
##open
##open
##open
##open
err##
err##
err##
err##
windows:(1) 10 state count:10
3> 10
source:20
source:30
windows:(1) 30 state count:30
3> 30
source:40
windows:(1) 40 state count:40
3> 40
source:50
windows:(1) 50 state count:50
3> 50
source:60
source:70
windows:(1) 70 state count:70
3> 70
source:80
windows:(1) 80 state count:80
3> 80
source:90
windows:(1) 90 state count:90
3> 90
source:100
err_________________
windows:(1) 110 state count:110
3> 110
source:10
##open
err##
##open
err##
##open
##open
err##
err##
windows:(1) 10 state count:10
3> 10
windows:(1) 20 state count:20
3> 20
source:20
Process finished with exit code 130 (interrupted by signal 2: SIGINT)
(3) 自定義state 自行實現實現checkpoint接口
實現CheckpointedFunction接口或者ListCheckpoint<T extends Serializable>接口
分析說明:
因爲需要實現ListCheckpoint接口,所以source和windows處理代碼,單獨寫成了JAVA類的形似,實現邏輯和驗證方法跟manage state相似,但是在如下代碼中,Source和Window都實現了ListCheckpoint接口,也就是說,Source拋出異常的時候,Source和Window都將可以從checkpoint中獲取歷史狀態,從而達到不丟失狀態的能力。
代碼列表:
AutoSourceWithCp.java Source代碼
WindowStatisticWithChk.java windows apply函數代碼
CheckPointMain.java 主程序,調用
代碼清單:
AutoSourceWithCp.java
package per.test.flink;
import org.apache.flink.api.common.state.ListState;
import org.apache.flink.api.java.tuple.Tuple4;
import org.apache.flink.streaming.api.checkpoint.ListCheckpointed;
import org.apache.flink.streaming.api.functions.source.RichSourceFunction;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
import per.test.bean.UserState;
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
/**
* Created by betree on 2018/9/13.
*/
public class AutoSourceWithCp extends RichSourceFunction<Tuple4<Integer,String,String,Integer>> implements ListCheckpointed<UserState> {
private int count = 0;
private boolean is_running = true;
@Override
public void run(SourceContext sourceContext) throws Exception {
Random random = new Random();
while(is_running){
for (int i = 0; i < 10; i++) {
sourceContext.collect(Tuple4.of(1, "hello-" + count, "alphabet", count));
count++;
}
System.out.println("source:"+count);
Thread.sleep(2000);
if(count>100){
throw new Exception("exception made by ourself!");
}
}
}
@Override
public void cancel() {
is_running = false;
}
@Override
public List<UserState> snapshotState(long l, long l1) throws Exception {
List<UserState> listState= new ArrayList<>();
UserState state = new UserState(count);
listState.add(state);
System.out.println("############# check point :"+listState.get(0).getCount());
return listState;
}
@Override
public void restoreState(List<UserState> list) throws Exception {
count = list.get(0).getCount();
System.out.println("AutoSourceWithCp restoreState:"+count);
}
}
WindowStatisticWithChk.java
package per.test.flink;
import org.apache.flink.api.java.tuple.Tuple;
import org.apache.flink.api.java.tuple.Tuple4;
import org.apache.flink.streaming.api.checkpoint.ListCheckpointed;
import org.apache.flink.streaming.api.functions.windowing.WindowFunction;
import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
import org.apache.flink.streaming.api.windowing.windows.Window;
import org.apache.flink.util.Collector;
import per.test.bean.UserState;
import java.util.ArrayList;
import java.util.List;
/**
* Created by betree on 2018/9/13.
*/
public class WindowStatisticWithChk implements WindowFunction<Tuple4<Integer,String,String,Integer>,Integer,Tuple,TimeWindow> ,ListCheckpointed<UserState> {
private int total = 0;
@Override
public List<UserState> snapshotState(long l, long l1) throws Exception {
List<UserState> listState= new ArrayList<>();
UserState state = new UserState(total);
listState.add(state);
return listState;
}
@Override
public void restoreState(List<UserState> list) throws Exception {
total = list.get(0).getCount();
}
@Override
public void apply(Tuple tuple, TimeWindow timeWindow, Iterable<Tuple4<Integer, String, String, Integer>> iterable, Collector<Integer> collector) throws Exception {
int count = 0;
for(Tuple4<Integer, String, String, Integer> data : iterable){
count++;
System.out.println("apply key"+tuple.toString()+" count:"+data.f3+" "+data.f0);
}
total = total+count;
System.out.println("windows total:"+total+" count:"+count+" ");
collector.collect(count);
}
}
CheckPointMain.java
package per.test.flink;
import org.apache.flink.api.java.tuple.Tuple4;
import org.apache.flink.runtime.state.filesystem.FsStateBackend;
import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.windowing.assigners.SlidingProcessingTimeWindows;
import org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;
/**
* Created by betree on 2018/9/13.
*/
public class CheckPointMain {
public static void main(String[] args) throws Exception{
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setStateBackend(new FsStateBackend("file:///Users/username/Documents/程序數據/Flink/checkpoint/"));
env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
env.getCheckpointConfig().setCheckpointInterval(1000);
DataStream<Tuple4<Integer,String,String,Integer>> data = env.setParallelism(4).addSource(new AutoSourceWithCp());
env.setParallelism(4);
data.keyBy(0)
.window(TumblingProcessingTimeWindows.of(Time.seconds(2)))
.apply(new WindowStatisticWithChk())
.print();
env.execute();
}
}
借鑑文章
https://forum.huawei.com/enterprise/en/thread-452547.html
http://www.zhanghs.com/2016/10/23/first-steps-of-apache-flink