一.項目概述
1.1 訂單支付實時監控
基本需求
- 用戶下單之後，應設置訂單失效時間，以提高用戶支付的意願，並降 低系統風險
- 用戶下單後15分鐘未支付，則輸出監控信息
解決思路
- 利用 CEP 庫進行事件流的模式匹配，並設定匹配的時間間隔
- 也可以利用狀態編程，用 process function 實現處理邏輯
1.2 訂單支付實時對賬
基本需求
- 用戶下單並支付後，應查詢到賬信息，進行實時對賬
- 如果有不匹配的支付信息或者到賬信息，輸出提示信息
解決思路
- 從兩條流中分別讀取訂單支付信息和到賬信息，合併處理
- 用 connect 連接合並兩條流，用 coProcessFunction 做匹配處理
二. 代碼
2.1 pom文件配置
pom文件配置如下:
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>1.10.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.11</artifactId>
<version>1.10.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka_2.11</artifactId>
<version>1.10.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-core</artifactId>
<version>1.10.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.11</artifactId>
<version>1.10.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-redis_2.11</artifactId>
<version>1.1.5</version>
</dependency>
<!-- https://mvnrepository.com/artifact/mysql/mysql-connector-java -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.19</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-statebackend-rocksdb_2.11</artifactId>
<version>1.10.1</version>
</dependency>
<!-- Table API 和 Flink SQL -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-planner-blink_2.11</artifactId>
<version>1.10.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-planner_2.11</artifactId>
<version>1.10.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-api-java-bridge_2.11</artifactId>
<version>1.10.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-scala_2.11</artifactId>
<version>1.10.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-common</artifactId>
<version>1.10.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-csv</artifactId>
<version>1.10.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-cep_2.11</artifactId>
<version>1.10.1</version>
</dependency>
2.2 POJO類
OrderEvent
private Long orderId;
private String eventType;
private String txId;
private Long timestamp;
ReceiptEvent
private String txId;
private String payChannel;
private Long timestamp;
OrderResult
private Long orderId;
private String resultState;
2.3 訂單支付超時監控-CEP
代碼:
package com.zqs.flink.project.orderpay_detect;
/**
* @author 只是甲
* @date 2021-10-20
* @remark 訂單支付超時監控
*/
import com.zqs.flink.project.orderpay_detect.beans.OrderEvent;
import com.zqs.flink.project.orderpay_detect.beans.OrderResult;
import org.apache.flink.cep.CEP;
import org.apache.flink.cep.PatternSelectFunction;
import org.apache.flink.cep.PatternStream;
import org.apache.flink.cep.PatternTimeoutFunction;
import org.apache.flink.cep.pattern.Pattern;
import org.apache.flink.cep.pattern.conditions.SimpleCondition;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.timestamps.AscendingTimestampExtractor;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.util.OutputTag;
import java.net.URL;
import java.util.List;
import java.util.Map;
public class OrderPayTimeout {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
env.setParallelism(1);
// 讀取數據並轉換成POJO類型
URL resource = OrderPayTimeout.class.getResource("/OrderLog.csv");
DataStream<OrderEvent> orderEventStream = env.readTextFile(resource.getPath())
.map( line -> {
String[] fields = line.split(",");
return new OrderEvent(new Long(fields[0]), fields[1], fields[2], new Long(fields[3]));
} )
.assignTimestampsAndWatermarks(new AscendingTimestampExtractor<OrderEvent>() {
@Override
public long extractAscendingTimestamp(OrderEvent element) {
return element.getTimestamp() * 1000L;
}
});
// 1. 定義一個帶時間限制的模式
Pattern<OrderEvent, OrderEvent> orderPayPattern = Pattern
.<OrderEvent>begin("create").where(new SimpleCondition<OrderEvent>() {
@Override
public boolean filter(OrderEvent value) throws Exception {
return "create".equals(value.getEventType());
}
})
.followedBy("pay").where(new SimpleCondition<OrderEvent>() {
@Override
public boolean filter(OrderEvent value) throws Exception {
return "pay".equals(value.getEventType());
}
})
.within(Time.minutes(15));
// 2. 定義側輸出流標籤， 用來表示超時事件
OutputTag<OrderResult> orderTimeoutTag = new OutputTag<OrderResult>("order-timeout"){};
// 3. 將pattern應用到輸入數據流上， 得到pattern stream
PatternStream<OrderEvent> patternStream = CEP.pattern(orderEventStream.keyBy(OrderEvent::getOrderId), orderPayPattern);
// 4. 調用select方法，實現對匹配複雜事件和超時複雜事件的提取和處理
SingleOutputStreamOperator<OrderResult> resultStream = patternStream
.select(orderTimeoutTag, new OrderTimeoutSelect(), new OrderPaySelect() );
resultStream.print("payed normally");
resultStream.getSideOutput(orderTimeoutTag).print("timeout");
env.execute("order timeout detect job");
}
// 實現自定義的超時事件處理函數
public static class OrderTimeoutSelect implements PatternTimeoutFunction<OrderEvent, OrderResult>{
@Override
public OrderResult timeout(Map<String, List<OrderEvent>> pattern, long timeoutTimestamp) throws Exception {
Long timeoutOrderId = pattern.get("create").iterator().next().getOrderId();
return new OrderResult(timeoutOrderId, "timeout " + timeoutTimestamp);
}
}
// 實現自定義的正常匹配事件處理函數
public static class OrderPaySelect implements PatternSelectFunction<OrderEvent, OrderResult>{
@Override
public OrderResult select(Map<String, List<OrderEvent>> pattern) throws Exception {
Long payedOrderId = pattern.get("pay").iterator().next().getOrderId();
return new OrderResult(payedOrderId, "payed");
}
}
}
測試記錄:
2.4 訂單支付超時監控-Without CEP
代碼:
package com.zqs.flink.project.orderpay_detect;
/**
* @author 只是甲
* @date 2021-10-20
* @remark 監控超時未支付訂單-不使用CEP
*/
import com.zqs.flink.project.orderpay_detect.beans.OrderEvent;
import com.zqs.flink.project.orderpay_detect.beans.OrderResult;
import org.apache.flink.api.common.state.ValueState;
import org.apache.flink.api.common.state.ValueStateDescriptor;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
import org.apache.flink.streaming.api.functions.timestamps.AscendingTimestampExtractor;
import org.apache.flink.util.Collector;
import org.apache.flink.util.OutputTag;
import java.net.URL;
public class OrderTimeoutWithoutCep {
// 定義超時時間的側輸出流標籤
private final static OutputTag<OrderResult> orderTimeoutTag = new OutputTag<OrderResult>("order-timeout"){};
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
env.setParallelism(1);
// 讀取數據並轉換成POJO類型
URL resource = OrderPayTimeout.class.getResource("/OrderLog.csv");
DataStream<OrderEvent> orderEventStream = env.readTextFile(resource.getPath())
.map(line -> {
String[] fields = line.split(",");
return new OrderEvent(new Long(fields[0]), fields[1], fields[2], new Long(fields[3]));
})
.assignTimestampsAndWatermarks(new AscendingTimestampExtractor<OrderEvent>() {
@Override
public long extractAscendingTimestamp(OrderEvent element) {
return element.getTimestamp() * 1000L;
}
});
// 自定義處理函數，主流輸出正常匹配訂單事件， 側輸出流輸出超時報警事件
SingleOutputStreamOperator<OrderResult> resultStream = orderEventStream
.keyBy(OrderEvent::getOrderId)
.process(new OrderPayMatchDetect());
resultStream.print("payed normally");
resultStream.getSideOutput(orderTimeoutTag).print("timeout");
env.execute("order timeout detect without cep job");
}
// 實現自定義KeyedProcessFunction
public static class OrderPayMatchDetect extends KeyedProcessFunction<Long, OrderEvent, OrderResult>{
// 定義狀態， 保存之前點單是否已經來過create、pay事件
ValueState<Boolean> isPayedState;
ValueState<Boolean> isCreatedState;
// 定義狀態，保存定時器時間戳
ValueState<Long> timerTsState;
@Override
public void open(Configuration parameters) throws Exception {
isPayedState = getRuntimeContext().getState(new ValueStateDescriptor<Boolean>("is-payed", Boolean.class, false));
isCreatedState = getRuntimeContext().getState(new ValueStateDescriptor<Boolean>("is-created", Boolean.class, false));
timerTsState = getRuntimeContext().getState(new ValueStateDescriptor<Long>("timer-ts", Long.class));
}
@Override
public void processElement(OrderEvent value, Context ctx, Collector<OrderResult> out) throws Exception {
// 先獲取當前裝填
Boolean isPayed = isPayedState.value();
Boolean isCreated = isCreatedState.value();
Long timerTs = timerTsState.value();
// 判斷當前事件類型
if( "create".equals(value.getEventType()) ){
// 1. 如果來的是create，要判斷是否支付過
if( isPayed ){
// 1.1 如果已經正常支付，輸出正常匹配結果
out.collect(new OrderResult(value.getOrderId(), "payed successfully"));
// 清空狀態，刪除定時器
isCreatedState.clear();
isPayedState.clear();
timerTsState.clear();
ctx.timerService().deleteEventTimeTimer(timerTs);
} else {
// 1.2 如果沒有支付過，註冊15分鐘後的定時器，開始等待支付事件
Long ts = ( value.getTimestamp() + 15 * 60 ) * 1000L;
ctx.timerService().registerEventTimeTimer(ts);
// 更新狀態
timerTsState.update(ts);
isCreatedState.update(true);
}
} else if( "pay".equals(value.getEventType()) ){
// 2. 如果來的是pay，要判斷是否有下單事件來過
if( isCreated ){
// 2.1 已經有過下單事件，要繼續判斷支付的時間戳是否超過15分鐘
if( value.getTimestamp() * 1000L < timerTs ){
// 2.1.1 在15分鐘內，沒有超時，正常匹配輸出
out.collect(new OrderResult(value.getOrderId(), "payed successfully"));
} else {
// 2.1.2 已經超時，輸出側輸出流報警
ctx.output(orderTimeoutTag, new OrderResult(value.getOrderId(), "payed but already timeout"));
}
// 統一清空狀態
isCreatedState.clear();
isPayedState.clear();
timerTsState.clear();
ctx.timerService().deleteEventTimeTimer(timerTs);
} else {
// 2.2 沒有下單事件，亂序，註冊一個定時器，等待下單事件
ctx.timerService().registerEventTimeTimer( value.getTimestamp() * 1000L);
// 更新狀態
timerTsState.update(value.getTimestamp() * 1000L);
isPayedState.update(true);
}
}
}
@Override
public void onTimer(long timestamp, OnTimerContext ctx, Collector<OrderResult> out) throws Exception {
// 定時器觸發， 說明一定有一個事件沒來
if ( isPayedState.value() ){
// 如果pay來了，說明create沒來
ctx.output(orderTimeoutTag, new OrderResult(ctx.getCurrentKey(), "payed but not found created log "));
} else {
// 如果pay沒來，支付超時
ctx.output(orderTimeoutTag, new OrderResult(ctx.getCurrentKey(), "timerout"));
}
// 清空狀態
isCreatedState.clear();
isPayedState.clear();
timerTsState.clear();
}
}
}
測試記錄:
2.5 支付賬單核對
代碼:
package com.zqs.flink.project.orderpay_detect;
/**
* @author 只是甲
* @Date 2021-10-20
* @remak 支付賬單核對
*/
import com.zqs.flink.project.orderpay_detect.beans.OrderEvent;
import com.zqs.flink.project.orderpay_detect.beans.ReceiptEvent;
import org.apache.flink.api.common.state.ValueState;
import org.apache.flink.api.common.state.ValueStateDescriptor;
import org.apache.flink.api.java.tuple.Tuple;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.co.CoProcessFunction;
import org.apache.flink.streaming.api.functions.timestamps.AscendingTimestampExtractor;
import org.apache.flink.util.Collector;
import org.apache.flink.util.OutputTag;
import java.net.URL;
public class TxPayMatch {
// 定義側輸出流標籤
private final static OutputTag<OrderEvent> unmatchedPays = new OutputTag<OrderEvent>("unmatched-pays"){};
private final static OutputTag<ReceiptEvent> unmatchedReceipts = new OutputTag<ReceiptEvent>("unmatched-receipts"){};
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
env.setParallelism(1);
// 讀取數據並轉換成POJO類型
// 讀取訂單支付事件數據
URL orderResource = TxPayMatch.class.getResource("/OrderLog.csv");
DataStream<OrderEvent> orderEventStream = env.readTextFile(orderResource.getPath())
.map( line -> {
String[] fields = line.split(",");
return new OrderEvent(new Long(fields[0]), fields[1], fields[2], new Long(fields[3]));
} )
.assignTimestampsAndWatermarks(new AscendingTimestampExtractor<OrderEvent>() {
@Override
public long extractAscendingTimestamp(OrderEvent element) {
return element.getTimestamp() * 1000L;
}
})
.filter( data -> !"".equals(data.getTxId()) ); // 交易id不爲空，必須是pay事件
// 讀取到賬事件數據
URL receiptResource = TxPayMatch.class.getResource("/ReceiptLog.csv");
SingleOutputStreamOperator<ReceiptEvent> receiptEventStream = env.readTextFile(receiptResource.getPath())
.map(line -> {
String[] fields = line.split(",");
return new ReceiptEvent(fields[0], fields[1], new Long(fields[2]));
})
.assignTimestampsAndWatermarks(new AscendingTimestampExtractor<ReceiptEvent>() {
@Override
public long extractAscendingTimestamp(ReceiptEvent element) {
return element.getTimestamp() * 1000L;
}
});
// 將兩條流進行連接合並，進行匹配處理，不匹配的事件輸出到側輸出流
SingleOutputStreamOperator<Tuple2<OrderEvent, ReceiptEvent>> resultStream = orderEventStream
.keyBy(OrderEvent::getTxId)
.connect(receiptEventStream.keyBy(ReceiptEvent::getTxId))
.process(new TxPayMatchDetect());
resultStream.print("matched-pays");
resultStream.getSideOutput(unmatchedPays).print("unmatched-pays");
resultStream.getSideOutput(unmatchedReceipts).print("unmathced-receipts");
env.execute("tx match detect job");
}
// 實現自定義CoProcessFunction
public static class TxPayMatchDetect extends CoProcessFunction<OrderEvent, ReceiptEvent, Tuple2<OrderEvent, ReceiptEvent>>{
// 定義狀態， 保存當前已經到來的訂單支付事件和到賬時間
ValueState<OrderEvent> payState;
ValueState<ReceiptEvent> receiptState;
@Override
public void open(Configuration parameters) throws Exception {
payState = getRuntimeContext().getState(new ValueStateDescriptor<OrderEvent>("pay", OrderEvent.class));
receiptState = getRuntimeContext().getState(new ValueStateDescriptor<ReceiptEvent>("receipt", ReceiptEvent.class));
}
@Override
public void processElement1(OrderEvent pay, Context ctx, Collector<Tuple2<OrderEvent, ReceiptEvent>> out) throws Exception {
// 訂單支付事件來了，判斷是否已經有對應的到賬事件
ReceiptEvent receipt = receiptState.value();
if ( receipt != null){
// 如果receipt不爲空， 說明到賬事件已經來過， 輸出匹配事件，清空狀態
out.collect( new Tuple2<>(pay, receipt));
payState.clear();
receiptState.clear();
} else {
// 如果receipt沒來， 註冊一個定時器，開始等待
ctx.timerService().registerEventTimeTimer((pay.getTimestamp() + 5) * 1000L); // 等待5秒鐘
// 更新狀態
payState.update(pay);
}
}
@Override
public void processElement2(ReceiptEvent receipt, Context ctx, Collector<Tuple2<OrderEvent, ReceiptEvent>> out) throws Exception {
// 到賬事件來了，判斷是否已經有對應的支付事件
OrderEvent pay = payState.value();
if ( pay != null ){
// 如果pay不爲空，說明支付事件已經來過，輸出匹配時間，清空狀態
out.collect( new Tuple2<>(pay, receipt));
payState.clear();
receiptState.clear();
} else {
// 如果pay沒來， 註冊一個定時器，開始等待
ctx.timerService().registerEventTimeTimer( (receipt.getTimestamp() + 3) * 1000L );
// 更新狀態
receiptState.update(receipt);
}
}
@Override
public void onTimer(long timestamp, OnTimerContext ctx, Collector<Tuple2<OrderEvent, ReceiptEvent>> out) throws Exception {
// 定時器觸發，有可能是有一個事件沒來，不匹配，也有可能是都來過了，已經輸出並清空狀態
// 判斷哪個不爲空，那麼另一個就沒來
if(payState.value() != null ){
ctx.output(unmatchedPays, payState.value());
}
if (receiptState.value() != null){
ctx.output(unmatchedReceipts, receiptState.value());
}
// 清空狀態
payState.clear();
receiptState.clear();
}
}
}
測試記錄:
2.6 賬單核對-使用join
代碼:
package com.zqs.flink.project.orderpay_detect;
/**
* @author 只是甲
* @date 2021-10-21
* @remark 賬單核對-使用join
*/
import com.zqs.flink.project.orderpay_detect.beans.OrderEvent;
import com.zqs.flink.project.orderpay_detect.beans.ReceiptEvent;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.co.ProcessJoinFunction;
import org.apache.flink.streaming.api.functions.timestamps.AscendingTimestampExtractor;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.util.Collector;
import java.net.URL;
public class TxPayMatchByJoin {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
env.setParallelism(1);
// 讀取數據並轉換成POJO類型
// 讀取訂單支付事件數據
URL orderResource = TxPayMatchByJoin.class.getResource("/OrderLog.csv");
DataStream<OrderEvent> orderEventStream = env.readTextFile(orderResource.getPath())
.map(line -> {
String[] fields = line.split(",");
return new OrderEvent(new Long(fields[0]), fields[1], fields[2], new Long(fields[3]));
})
.assignTimestampsAndWatermarks(new AscendingTimestampExtractor<OrderEvent>() {
@Override
public long extractAscendingTimestamp(OrderEvent element) {
return element.getTimestamp() * 1000L;
}
})
.filter(data -> !"".equals(data.getTxId())); // 交易id不爲空，必須是pay事件
// 讀取到賬事件數據
URL receiptResource = TxPayMatchByJoin.class.getResource("/ReceiptLog.csv");
SingleOutputStreamOperator<ReceiptEvent> receiptEventStream = env.readTextFile(receiptResource.getPath())
.map(line -> {
String[] fields = line.split(",");
return new ReceiptEvent(fields[0], fields[1], new Long(fields[2]));
})
.assignTimestampsAndWatermarks(new AscendingTimestampExtractor<ReceiptEvent>() {
@Override
public long extractAscendingTimestamp(ReceiptEvent element) {
return element.getTimestamp() * 1000L;
}
});
// 區間連接兩條流, 得到匹配的數據
SingleOutputStreamOperator<Tuple2<OrderEvent, ReceiptEvent>> resultStream = orderEventStream
.keyBy(OrderEvent::getTxId)
.intervalJoin(receiptEventStream.keyBy(ReceiptEvent::getTxId))
.between(Time.seconds(-3), Time.seconds(5)) // -3,5區間範圍
.process(new TxPayMatchDetectByJoin());
resultStream.print();
env.execute("tx pay match by join job");
}
// 實現自定義ProcessJoinFunction
public static class TxPayMatchDetectByJoin extends ProcessJoinFunction<OrderEvent, ReceiptEvent, Tuple2<OrderEvent, ReceiptEvent>>{
@Override
public void processElement(OrderEvent left, ReceiptEvent right, Context ctx, Collector<Tuple2<OrderEvent, ReceiptEvent>> out) throws Exception {
out.collect(new Tuple2<>(left, right));
}
}
}
測試記錄:
類似SQL的內連接，只能顯示對應上的數據。
