本文轉自個人微信公衆號，原文鏈接。
如 上篇 所述，Flink 裏時間包括Event Time、Processing Time 和 Ingestion Time 三種類型。
- Processing Time：Processing Time 是算子處理某個數據時到系統時間。Processing Time 是最簡單的時間，提供了最好的性能和最低的延遲，但是，在分佈式環境中，Processing Time具有不確定性，多次運行的結果可能出現不一致。
- Ingestion Time：Ingestion Time 是數據進入Flink 集羣的時間，Source Operator 給數據加上時間戳。
- Event Time：Event Time是數據在設備上產生時的時間，一般都嵌入到了數據記錄中，相比於其他兩種，Event Time 更具有業務意義， 取決於數據而不是系統。舉例來說，重跑歷史數據時，如果根據Processing Time 重跑，可能會造成結果不一致，而根據Event Time 重跑，結果是一致的。
由於Event Time 更能表達業務需求，所以，Event Time 應用更爲廣泛，但使用Event Time 也會存在一些問題。
1. 問題：亂序與延遲
亂序與延遲是實時系統中最常見的問題。比如說，在實時系統中廣泛使用的消息隊列，很難保證端到端的全局有序，從而導致進入 Flink 集羣的數據是無序的；然後，由於洪峯的存在，比如秒殺或者重跑歷史數據，很容易造成數據在消息隊列堆積，從而造成延遲。
2. 解決方案
採用Event Time的流計算處理器，需要評估Event Time進展，比如當窗口結束時，需要通知 Operator 關閉窗口並開始計算。
2.1 Watermark
Apache Flink 採用watermark來處理，watermark 帶有一個時間戳，作爲數據流的一部分隨數據流流動，Watermark(t) 表示event time 小於等於 t 的都已經到達，如下圖所示。
2.1.1 生成Watermark
2.1.1.1 方法1 Source 中生成
在source中，直接生成watermark，不過，source生成的watermark 優先級比較低，可以被方法2中的覆蓋掉。具體的定義在一篇講Source & Sink 時詳述。
2.1.1.2 方法2 Timestamp Assigner
Timestamp Assigner 輸入數據流，產生一個新的數據流，新數據流帶有產生的watermark，如果原數據流本身就有watermark，則覆蓋原watermark。Timestamp Assigner 一般緊跟在source後，但不是必須的，但是必須在第一個event time 操作前。
Timestamp Assigner 分兩種：
- Periodic： 週期性（一定時間間隔或一定數據量）產生watermark。
- Punctuated： 間斷的 watermark，一般根據event 決定是否產生新watermark。
Periodic
直接看源碼(註釋太明白，不捨得刪)。
/**
* A {@code TimestampAssigner} assigns event time timestamps to elements.
* These timestamps are used by all functions that operate on event time,
* for example event time windows.
*
* <p>Timestamps are represented in milliseconds since the Epoch
* (midnight, January 1, 1970 UTC).
*
* @param <T> The type of the elements to which this assigner assigns timestamps.
*/
public interface TimestampAssigner<T> extends Function {
/**
* Assigns a timestamp to an element, in milliseconds since the Epoch.
*
* <p>The method is passed the previously assigned timestamp of the element.
* That previous timestamp may have been assigned from a previous assigner,
* by ingestion time. If the element did not carry a timestamp before, this value is
* {@code Long.MIN_VALUE}.
*
* @param element The element that the timestamp will be assigned to.
* @param previousElementTimestamp The previous internal timestamp of the element,
* or a negative value, if no timestamp has been assigned yet.
* @return The new timestamp.
*/
long extractTimestamp(T element, long previousElementTimestamp);
}
/**
* The {@code AssignerWithPeriodicWatermarks} assigns event time timestamps to elements,
* and generates low watermarks that signal event time progress within the stream.
* These timestamps and watermarks are used by functions and operators that operate
* on event time, for example event time windows.
*
* <p>Use this class to generate watermarks in a periodical interval.
* At most every {@code i} milliseconds (configured via
* {@link ExecutionConfig#getAutoWatermarkInterval()}), the system will call the
* {@link #getCurrentWatermark()} method to probe for the next watermark value.
* The system will generate a new watermark, if the probed value is non-null
* and has a timestamp larger than that of the previous watermark (to preserve
* the contract of ascending watermarks).
*
* <p>The system may call the {@link #getCurrentWatermark()} method less often than every
* {@code i} milliseconds, if no new elements arrived since the last call to the
* method.
*
* <p>Timestamps and watermarks are defined as {@code longs} that represent the
* milliseconds since the Epoch (midnight, January 1, 1970 UTC).
* A watermark with a certain value {@code t} indicates that no elements with event
* timestamps {@code x}, where {@code x} is lower or equal to {@code t}, will occur any more.
*
* @param <T> The type of the elements to which this assigner assigns timestamps.
*
* @see org.apache.flink.streaming.api.watermark.Watermark
*/
public interface AssignerWithPeriodicWatermarks<T> extends TimestampAssigner<T> {
/**
* Returns the current watermark. This method is periodically called by the
* system to retrieve the current watermark. The method may return {@code null} to
* indicate that no new Watermark is available.
*
* <p>The returned watermark will be emitted only if it is non-null and its timestamp
* is larger than that of the previously emitted watermark (to preserve the contract of
* ascending watermarks). If the current watermark is still
* identical to the previous one, no progress in event time has happened since
* the previous call to this method. If a null value is returned, or the timestamp
* of the returned watermark is smaller than that of the last emitted one, then no
* new watermark will be generated.
*
* <p>The interval in which this method is called and Watermarks are generated
* depends on {@link ExecutionConfig#getAutoWatermarkInterval()}.
*
* @see org.apache.flink.streaming.api.watermark.Watermark
* @see ExecutionConfig#getAutoWatermarkInterval()
*
* @return {@code Null}, if no watermark should be emitted, or the next watermark to emit.
*/
@Nullable
Watermark getCurrentWatermark();
}
可以看出，自定義的Assigner 需要實現AssignerWithPeriodicWatermarks 接口，其中getCurrentWatermark 產生新的watermark，如果返回非空且大於原來的watermark，則生成了新的watermark；另外，extractTimestamp 用於給數據加上時間戳，這個時間戳在後續所有基於event time的計算中使用。以下面的代碼爲例，假設數據可能亂序，但最多延遲3.5秒。
/**
* This generator generates watermarks assuming that elements arrive out of order,
* but only to a certain degree. The latest elements for a certain timestamp t will arrive
* at most n milliseconds after the earliest elements for timestamp t.
*/
class BoundedOutOfOrdernessGenerator extends AssignerWithPeriodicWatermarks[MyEvent] {
val maxOutOfOrderness = 3500L // 3.5 seconds
var currentMaxTimestamp: Long = _
override def extractTimestamp(element: MyEvent, previousElementTimestamp: Long): Long = {
element.getCreationTime()
}
override def getCurrentWatermark(): Watermark = {
// return the watermark as current highest timestamp minus the out-of-orderness bound
new Watermark(currentMaxTimestamp - maxOutOfOrderness)
}
}
ExecutionConfig.setAutoWatermarkInterval(...) 定義了watermark產生的時間間隔，單位是毫秒。
Punctuated
根據event來確定是否需要產生新的watermark，定義Punctuated Assigner 需要實現AssignerWithPunctuatedWatermarks接口，包括函數extractTimestamp，checkAndGetNextWatermark，其中extractTimestamp 同Periodic Assigner，首先調用；然後調用checkAndGetNextWatermark ，用於確定是否需要產生新的watermark，當checkAndGetNextWatermark 產生一個非空且大於上一個watermark時就產生了新的watermark。舉個例子如下：
class PunctuatedAssigner extends AssignerWithPunctuatedWatermarks[MyEvent] {
override def extractTimestamp(element: MyEvent, previousElementTimestamp: Long): Long = {
element.getCreationTime
}
override def checkAndGetNextWatermark(lastElement: MyEvent, extractedTimestamp: Long): Watermark = {
if (lastElement.hasWatermarkMarker()) new Watermark(extractedTimestamp) else null
}
}
2.1.2 Flink 預定義Timestamp Assigner
爲了便於使用，Apache Flink 提供了兩種預定義的Timestamp Assigner：
-
AscendingTimestampExtractor： 這是
AssignerWithPeriodicWatermarks的最簡單的情況，數據流是按時間戳升序到達Flink的，這種情況下，數據裏的時間戳就可以作爲watermarkval withTimestampsAndWatermarks = stream.assignAscendingTimestamps( _.getCreationTime ) -
BoundedOutOfOrdernessTimestampExtractor: 這也是一個
AssignerWithPeriodicWatermarks的實現，表示已知數據的最大延遲。val withTimestampsAndWatermarks = stream.assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor[MyEvent](Time.seconds(10))( _.getCreationTime ))
這兩種Timestamp Assigner 一是可以直接使用，二是可以作爲學習的代碼示例。
Latency
即使採用watermark 技術，對於watermark(t) 也可能存在時間戳小於t卻沒有到達的數據，在現實中，延遲可能是無上限的，這種情況下，不可能無限等待下去；另外，即使延遲有限，但如果讓watermark 延遲太多也不好，因爲延遲太多可能就失去了實時的意義。所以，必須要作出選擇。
默認情況下，延遲超過watermark的數據會被丟棄，但 Flink 允許在窗口操作上指定最大延遲，我們用N表示支持的最大延遲（N默認爲0），對於窗口 [start_time, end_time)] ，數據遲於 watermark(t) 但先於end_time+N到達的，仍然會添加到窗口中再次觸發計算。爲了支持這種情況，Flink 需要保持這個窗口state 到時間戳 end_time + N ，當時間到達end_time+N後，Flink 刪除窗口和state。
stream
.keyBy(<key selector>)
.window(<window assigner>)
.allowedLateness(<time>)
.<windowed transformation>(<window function>)
3. 總結
本文主要介紹Flink 中Event Time 和Watermark。由於Event Time 具有業務意義，且具有確定性，所以Event Time 應用廣泛，但由於在現實中存在延遲和亂序問題，Flink 採用了 Watermark 來解決這個問題。
掃描下方二維碼關注我。
