flink cep 案例之機架溫度監控報警

FlinkCEP是在Flink之上實現的複雜事件處理庫。它提供了豐富的API，允許您在不停止的事件流中檢測事件模式，並對複雜事件做相應處理。
模式匹配是複雜事件處理的一個有力的保障，應用場景包括受一系列事件驅動的各種業務流程，例如在正常的網略行爲中偵測異常行爲；在金融應用中查找價格、交易量和其他行爲的模式。

特點：

• 複雜性：多個流join，窗口聚合，事件序列或patterns檢測
• 低延遲：秒或毫秒級別，比如做信用卡盜刷檢測，或攻擊檢測
• 高吞吐：每秒上萬條消息

在這篇博客中，我們將通過一個案例來講解flink CEP的使用。
案例來源於官網博客：https://flink.apache.org/news/2016/04/06/cep-monitoring.html

輸入事件流由來自一組機架的溫度和功率事件組成。 目標是檢測
當機架過熱時我們需要發出警告和報警。

我們通過自定義的source來模擬生成機架的溫度，然後定義以下的規則來生成警告和報警

• 警告：某機架在10秒內連續兩次上報的溫度超過閾值；
• 報警：某機架在20秒內連續兩次匹配警告；

首先我們定義一個監控事件

注意要重寫裏面的hashcode方法和equal方法

來自官網： The events in the DataStream to which you want to apply pattern matching must implement proper equals() and hashCode() methods because FlinkCEP uses them for comparing and matching events.

public abstract class MonitoringEvent {
    private int rackID;

    public MonitoringEvent(int rackID) {
        this.rackID = rackID;
    }

    public int getRackID() {
        return rackID;
    }

    public void setRackID(int rackID) {
        this.rackID = rackID;
    }

    @Override
    public boolean equals(Object obj) {
        if (obj instanceof MonitoringEvent) {
            MonitoringEvent monitoringEvent = (MonitoringEvent) obj;
            return monitoringEvent.canEquals(this) && rackID == monitoringEvent.rackID;
        } else {
            return false;
        }
    }

    @Override
    public int hashCode() {
        return rackID;
    }

    public boolean canEquals(Object obj) {
        return obj instanceof MonitoringEvent;
    }
}




public class TemperatureEvent extends MonitoringEvent {
    private double temperature;
    ...
}

public class PowerEvent extends MonitoringEvent {
    private double voltage;
    ...
}

我們通過自定義的source來模擬生成MonitoringEvent數據。

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        // Use ingestion time => TimeCharacteristic == EventTime + IngestionTimeExtractor
        env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
        // Input stream of monitoring events
        DataStream<MonitoringEvent> inputEventStream = env
                .addSource(new MonitoringEventSource(
                        MAX_RACK_ID,
                        PAUSE,
                        TEMPERATURE_RATIO,
                        POWER_STD,
                        POWER_MEAN,
                        TEMP_STD,
                        TEMP_MEAN))
                .assignTimestampsAndWatermarks(new IngestionTimeExtractor<>());

接下來定義模式,在10秒鐘之內連續兩個event的溫度超過閾值

       // Warning pattern: Two consecutive temperature events whose temperature is higher than the given threshold
        // appearing within a time interval of 10 seconds
        Pattern<MonitoringEvent, ?> warningPattern = Pattern.<MonitoringEvent>begin("first")
                .subtype(TemperatureEvent.class)
                .where(new IterativeCondition<TemperatureEvent>() {
                    private static final long serialVersionUID = -6301755149429716724L;

                    @Override
                    public boolean filter(TemperatureEvent value, Context<TemperatureEvent> ctx) throws Exception {
                        return value.getTemperature() >= TEMPERATURE_THRESHOLD;
                    }
                })
                .next("second")  //緊接着上一個事件
                
                
                .subtype(TemperatureEvent.class)
                .where(new IterativeCondition<TemperatureEvent>() {
                    private static final long serialVersionUID = 2392863109523984059L;

                    @Override
                    public boolean filter(TemperatureEvent value, Context<TemperatureEvent> ctx) throws Exception {
                        return value.getTemperature() >= TEMPERATURE_THRESHOLD;
                    }
                })
                .within(Time.seconds(10));
                

使用報警模式和輸入流生成模式流

     // Create a pattern stream from our warning pattern
        PatternStream<MonitoringEvent> tempPatternStream = CEP.pattern(
                inputEventStream.keyBy("rackID"),
                warningPattern);

使用select方法爲每個匹配的報警模式生成相應的報警。其中返回值是一個map，key是我們定義的模式，value是匹配的事件列表。

    // Generate temperature warnings for each matched warning pattern
        DataStream<TemperatureWarning> warnings = tempPatternStream.select(
                (Map<String, List<MonitoringEvent>> pattern) -> {
                    TemperatureEvent first = (TemperatureEvent) pattern.get("first").get(0);
                    TemperatureEvent second = (TemperatureEvent) pattern.get("second").get(0);

                    return new TemperatureWarning(first.getRackID(), (first.getTemperature() + second.getTemperature()) / 2);
                }
        );

以上我們最後生成了相應的用於警告的DataStream類型的數據流warnings，接下來我們使用這個警告流來生成我們的報警流，即在20秒內連續兩次發生警告。

   // Alert pattern: Two consecutive temperature warnings appearing within a time interval of 20 seconds
        Pattern<TemperatureWarning, ?> alertPattern = Pattern.<TemperatureWarning>begin("first")
                .next("second")
                .within(Time.seconds(20));

然後通過上面的報警模式alertPattern和警告流warnings生成我們的報警流alertPatternStream。

   // Create a pattern stream from our alert pattern
        PatternStream<TemperatureWarning> alertPatternStream = CEP.pattern(
                warnings.keyBy("rackID"),
                alertPattern);

最後當收集到的兩次警告中，第一次警告的平均溫度小於第二次的時候，生成報警，封裝TemperatureAlert信息返回。

  // Generate a temperature alert only if the second temperature warning's average temperature is higher than
        // first warning's temperature
        DataStream<TemperatureAlert> alerts = alertPatternStream.flatSelect(
                (Map<String, List<TemperatureWarning>> pattern, Collector<TemperatureAlert> out) -> {
                    TemperatureWarning first = pattern.get("first").get(0);
                    TemperatureWarning second = pattern.get("second").get(0);

                    if (first.getAverageTemperature() < second.getAverageTemperature()) {
                        out.collect(new TemperatureAlert(first.getRackID()));
                    }
                },
                TypeInformation.of(TemperatureAlert.class));

最後我們將報警流和警告流輸出，當然我們也可以對這兩個流做其他的操作，比如發到報警系統等。

   // Print the warning and alert events to stdout
        warnings.print();
        alerts.print();

以上完整的代碼請參考：

https://github.com/zhangjun888/flink-demo/tree/master/src/main/java/streaming/cep

參考：
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/libs/cep.html
[2] https://flink.apache.org/news/2016/04/06/cep-monitoring.html

更多精彩內容，歡迎關注我的公衆號