flink窗口聚合函數

flink的WindowFucntion按照計算原理的不同可以分爲兩大類:
一類是增量聚合函數:對應有ReduceFunction、AggregateFunction;
另一類是全量窗口函數,對應有ProcessWindowFunction(還有WindowFunction)。
代碼如下:

    //每隔5秒統計每個基站的日誌數量
    data.map(stationLog => ((stationLog.sid, 1)))
      .keyBy(_._1)
      .window(TumblingEventTimeWindows.of(Time.seconds(5)))
      .reduce((v1, v2) => (v1._1, v1._2 + v2._2))

    //每隔3秒計算最近5秒內,每個基站的日誌數量
    data.map(stationLog => ((stationLog.sid, 1)))
      .keyBy(_._1)
      .timeWindow(Time.seconds(5), Time.seconds(3))
      .aggregate(new AggregateFunction[(String, Int), (String, Long), (String, Long)] {

        override def createAccumulator() = ("", 0)

        override def add(in: (String, Int), acc: (String, Long)) = {
          (in._1, acc._2 + in._2)
        }
        override def getResult(acc: (String, Long)) = acc

        override def merge(acc: (String, Long), acc1: (String, Long)) = {
          (acc._1, acc1._2 + acc._2)
        }
      })

    //每隔5秒統計每個基站的日誌數量
    data.map(stationLog => ((stationLog.sid, 1)))
      .keyBy(_._1)
      .timeWindow(Time.seconds(5))
      .process(new
          ProcessWindowFunction[(String, Int), (String, Int), String, TimeWindow] {
        override def process(key: String, context: Context, elements: Iterable[(String,
          Int)], out: Collector[(String, Int)]): Unit = {
          println("-------")
          out.collect((key, elements.size))
        }
      })
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章