Flink示例——State、Checkpoint、Savepoint

Flink示例——State、Checkpoint、Savepoint

版本信息

產品	版本
Flink	1.7.2
Java	1.8.0_231
Scala	2.11.12

Mavan依賴

• pom.xml 依賴部分

  <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-java</artifactId>
      <version>${flink.version}</version>
  </dependency>
  <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-streaming-java_2.11</artifactId>
      <version>${flink.version}</version>
  </dependency>
  <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-clients_2.11</artifactId>
      <version>${flink.version}</version>
  </dependency>
  

State Backend 狀態後端

• 用於管理State、Checkpoint
• State Backend分類
  • MemoryStateBackend
    • state數據 -> TaskManager內存
    • checkpoint數據 -> JobManager內存
  • FsStateBackend
    • state數據 -> TaskManager內存
    • checkpoint數據 -> 遠程文件系統(FileSystem)
  • RocksDBBackend
    • 所有數據 -> RocksDB -> 遠程文件系統(FileSystem)
• 代碼

  StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
  
  // 配置狀態後端 State Backend
  // MemoryStateBackend
  env.setStateBackend(new MemoryStateBackend());
  
  // FsStateBackend
  // env.setStateBackend(new FsStateBackend("hdfs://192.168.0.100:9000/flink/checkpoints"));
  
  // RocksDBStateBackend 需要導包
  //<dependency>
  //    <groupId>org.apache.flink</groupId>
  //    <artifactId>flink-statebackend-rocksdb_2.11</artifactId>
  //    <version>${flink.version}</version>
  //</dependency>
  // env.setStateBackend(new RocksDBStateBackend("hdfs://192.168.0.100:9000/flink/checkpoints"));
  

State 示例

• State分類
  • 算子狀態 OperatorState
    • 列表狀態 ListState
    • 聯合列表狀態 UnionListState
    • 廣播狀態 BroadcastState
  • 鍵控狀態 KeyedState
    • 值狀態 ValueState
    • 列表狀態 ListState
    • 映射狀態 MapState
    • 聚合狀態 ReducingState & AggregatingState
• 提供一個SourceFunction，方便後面測試

  public class CustomSourceFunction extends RichSourceFunction<Tuple2<String, Long>> {
  
      private boolean flag = true;
  
      @Override
      public void run(SourceContext<Tuple2<String, Long>> ctx) throws Exception {
          List<String> data = Arrays.asList("a", "b", "c", "d", "e", "f", "g");
          Random random = new Random();
          while (flag) {
              Thread.sleep(100);
              // 隨機取一個值
              String key = data.get(random.nextInt(data.size()));
              long value = System.currentTimeMillis();
              ctx.collect(Tuple2.of(key, value));
          }
      }
  
      @Override
      public void cancel() {
          flag = false;
      }
  
  }
  

• ValueState 示例

  public class ValueStateDemo {
  
      public static void main(String[] args) {
          StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
          env.setParallelism(3);
  
          // 自定義數據源
          CustomSourceFunction sourceFunction = new CustomSourceFunction();
          DataStreamSource<Tuple2<String, Long>> customDS = env.addSource(sourceFunction);
  
          // 處理
          customDS.keyBy(value -> value.f0)
                  .flatMap(flatMapWithState) // 帶狀態的Function
                  .print();
  
          try {
              env.execute();
          } catch (Exception e) {
              e.printStackTrace();
          }
      }
  
      // 定義一個帶狀態的FlatMapFunction
      // 該狀態用於保證一個key內每次只處理事務時間最新的數據
      private static RichFlatMapFunction<Tuple2<String, Long>, String> flatMapWithState = new RichFlatMapFunction<Tuple2<String, Long>, String>() {
  
          // 設置一個記錄時間的狀態
          private ValueState<Long> timeState;
  
          @Override
          public void open(Configuration parameters) throws Exception {
              // 初始化State
              timeState = getRuntimeContext().getState(new ValueStateDescriptor<>("maxTime", Long.class));
              // 不可在此賦值
              // timeState.update(Long.MIN_VALUE);
          }
  
          @Override
          public void flatMap(Tuple2<String, Long> value, Collector<String> out) throws Exception {
              Long maxTime = timeState.value();
              // 如果時間更大，則數據爲最新
              // maxTime == null 用於防止maxTime初始爲null的情況
              if (maxTime == null || value.f1 > maxTime) {
                  // 更新時間狀態
                  timeState.update(value.f1);
  
                  // 處理該數據
                  out.collect(value.f0 + "|" + value.f1);
              } else {
                  // 否則不處理，或者發送通知告警
                  System.out.println("---- Warning! ----");
              }
          }
      };
  
  }
  

• ListState 示例

  ListState<String> myListState = getRuntimeContext().getListState(new ListStateDescriptor<String>("my_liststate", String.class));
  myListState.add("state_1");
  Iterable<String> stateIter = myListState.get();
  for (String state : stateIter) {
      System.out.println("state = " + state);
  }
  

• MapState 示例

  MapState<String, Long> myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<String, Long>("my_mapstate", String.class, Long.class));
  myMapState.put("state_key_1", 1L);
  Long value = myMapState.get("state_key_1");
  

• ReducingState 示例

  // ReducingFunction(Math::max)控制每個傳入State的值的reduce計算
  ReducingStateDescriptor<Long> stateDescriptor = new ReducingStateDescriptor<>("my_reducingstate", Math::max, Long.class);
  ReducingState<Long> reducingState = getRuntimeContext().getReducingState(stateDescriptor);
  reducingState.add(100L);
  Long result = reducingState.get();
  

• AggregatingState與ReducingState同理，不做展示
• BroadcastState 示例

  public class BroadcastStateDemo {
  
      public static void main(String[] args) {
          StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
  
          // TagsSourceFunction 用於廣播，動態過濾事件數據
          TagsSourceFunction tagsSourceFunction = new TagsSourceFunction();
          DataStreamSource<String> tagDS = env.addSource(tagsSourceFunction);
          // 廣播流
          BroadcastStream<String> myBroadcast = tagDS.broadcast(new MapStateDescriptor<>("my_broadcast", String.class, String.class));
  
          // EventSourceFunction 實際處理的事件數據
          EventSourceFunction eventSourceFunction = new EventSourceFunction();
          DataStreamSource<Tuple2<String, Long>> eventDS = env.addSource(eventSourceFunction);
  
          // 連接廣播，並處理
          DataStream<Tuple2<String, Long>> resultDS = eventDS.connect(myBroadcast)
                  .process(new BroadcastProcessFunction<Tuple2<String, Long>, String, Tuple2<String, Long>>() {
                      // 廣播的描述符
                      private MapStateDescriptor<String, String> stateDescriptor = new MapStateDescriptor<>("my_broadcast", String.class, String.class);;
  
                      @Override
                      public void processElement(Tuple2<String, Long> value, ReadOnlyContext ctx, Collector<Tuple2<String, Long>> out) throws Exception {
                          ReadOnlyBroadcastState<String, String> broadcastState = ctx.getBroadcastState(stateDescriptor);
                          // 根據條件過濾
                          if (broadcastState.contains(value.f0)) {
                              out.collect(value);
                          }
                      }
  
                      @Override
                      public void processBroadcastElement(String value, Context ctx, Collector<Tuple2<String, Long>> out) throws Exception {
                          // 廣播流每來一條數據，就調用一次該方法
                          // 更新BroadcastState
                          BroadcastState<String, String> broadcastState = ctx.getBroadcastState(stateDescriptor);
                          broadcastState.put(value, "");
                      }
                  });
  
          resultDS.print();
  
          try {
              env.execute();
          } catch (Exception e) {
              e.printStackTrace();
          }
      }
  
      public static class TagsSourceFunction extends RichSourceFunction<String> {
  
          private boolean flag = true;
  
          @Override
          public void run(SourceContext<String> ctx) throws Exception {
              List<String> data = Arrays.asList("b", "e", "g");
              Random random = new Random();
              while (flag) {
                  // 隨機取一個值
                  ctx.collect(data.get(random.nextInt(data.size())));
  
                  Thread.sleep(5000);
              }
          }
  
          @Override
          public void cancel() {
              flag = false;
          }
  
      }
  
      public static class EventSourceFunction extends RichSourceFunction<Tuple2<String, Long>> {
  
          private boolean flag = true;
  
          @Override
          public void run(SourceContext<Tuple2<String, Long>> ctx) throws Exception {
              List<String> data = Arrays.asList("a", "b","c", "d", "e","f", "g", "h", "i", "j", "k");
              Random random = new Random();
              while (flag) {
                  Thread.sleep(100);
  
                  String key = data.get(random.nextInt(data.size()));
                  ctx.collect(Tuple2.of(key, System.currentTimeMillis()));
              }
          }
  
          @Override
          public void cancel() {
              flag = false;
          }
  
      }
  
  }
  

Checkpoint 示例

• 用於Flink自動存儲/恢復應用信息
• 啓用 Checkpoint

  // 每隔60秒做一次checkpoint
  env.enableCheckpointing(60 * 1000);
  // env.enableCheckpointing(60 * 1000, CheckpointingMode.EXACTLY_ONCE);
  // env.enableCheckpointing(60 * 1000, CheckpointingMode.AT_LEAST_ONCE);
  

• Timeout 超時

  env.getCheckpointConfig().setCheckpointTimeout(1000);
  

• FailOnCheckpointingErrors

  // 默認爲true，checkpoint失敗會導致Flink應用失敗
  // 設置爲false，checkpoint失敗時也會繼續運行Flink應用
  env.getCheckpointConfig().setFailOnCheckpointingErrors(false);
  

• 最大同時運行的Checkpoint個數

  // 如果前一個checkpoint太慢，當第二個checkpoint觸發時可能第一個還在
  // 就會出現多個checkpoint同時存在的情況
  env.getCheckpointConfig().setMaxConcurrentCheckpoints(3);
  

• Checkpoint之間的最小間隔時間

  // 設置之後，MaxConcurrentCheckpoints無效，同時只能有一個Checkpoint運行
  env.getCheckpointConfig().setMinPauseBetweenCheckpoints(1000);
  

• enableExternalizedCheckpoints

  // 默認情況下，Checkpoint用於自身應用異常時的恢復
  // 手動取消了應用後，會刪除Checkpoint（僅在應用失敗時，纔會保留Checkpoint）
  env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.DELETE_ON_CANCELLATION);
  // 手動取消了應用後，仍然保留Checkpoint
  // env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
  

• 保存多個Checkpoint，編輯conf/flink-conf.yaml

  // 默認一個最新的
  state.checkpoints.num-retained: 10
  

• 重啓策略

  // 重啓策略
  // 一共重試3次，每次重試間隔1秒
  env.setRestartStrategy(RestartStrategies.fixedDelayRestart(3, 1000));
  // 3min內最多重試3次，每次間隔5秒
  env.setRestartStrategy(RestartStrategies.failureRateRestart(3, Time.minutes(60), Time.seconds(10)));
  /*
  # 配置 conf/flink-conf.yaml
  # 無重啓策略
  restart-strategy: none
  # 策略 fixedDelayRestart
  restart-strategy: fixed-delay
  restart-strategy.fixed-delay.attempts: 3
  restart-strategy.fixed-delay.delay: 10 s
  # 策略 failureRateRestart
  restart-strategy: failure-rate
  restart-strategy.failure-rate.max-failures-per-interval: 3
  restart-strategy.failure-rate.failure-rate-interval: 3 min
  restart-strategy.failure-rate.delay: 10 s
  */
  

• 從Checkpoint恢復Job

  bin/flink run -s hdfs://skey_01:9000/flink-1.7.2/flink-checkpoints/19dd7c456b5507dc6b65cc836f319dd7/chk-30/_metadata flink-job.jar
  

Savepoint 示例

• 用於手動存儲/恢復應用信息
• 默認目錄，編輯conf/flink-conf.yaml

  state.savepoints.dir: hdfs://skey_01:9000/flink/savepoint
  

• 停止Job，並存儲Job狀態

  // 如果未指定路徑，則存入默認目錄
  bin/flink stop -p hdfs://skey_01:9000/flink/savepoint 19dd7c456b5507dc6b65cc836f319dd7
  

• 爲正在運行的Job設置狀態保存點

  // 如果未指定路徑，則存入默認目錄
  bin/flink savepoint -d hdfs://skey_01:9000/flink/savepoint 19dd7c456b5507dc6b65cc836f319dd7
  

• 從Savepoint恢復Job

  bin/flink run -s hdfs://skey_01:9000/flink/savepoint/savepoint-26dad6-4597317cb1b6 flink-job.jar