袋鼠雲：基於Flink構建實時計算平臺的總體架構和關鍵技術點

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"平臺建設的背景","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"傳統離線數據開發時效性較差，無法滿足快速迭代的互聯網需求。伴隨着以 Flink 爲代表的實時技術的飛速發展，實時計算被越來越多的企業使用，但是在使用中，各種問題也隨之而來。比如開發者使用門檻高、產出的業務數據質量沒有保障、企業缺少統一平臺管理難以維護等。在諸多不利因素的影響下，我們決定利用現有的 Flink 技術構建一套完整的實時計算平臺。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"平臺總體架構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從總體架構來看，實時計算平臺大體可以分爲三層：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"計算平臺","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"調度平臺","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"資源平臺。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每層承擔着相應的功能，同時層與層之間又有交互，符合高內聚、低耦合的設計原子，架構圖如下：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a8/a8fb17681f36300bea72b31d37d3d1f8.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"img","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"計算平臺","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"直接面向開發人員使用，可以根據業務需求接入各種外部數據源，提供後續任務使用。數據源配置完成後，就可以在上面做基於 Flink 框架可視化的數據同步、SQL 化的數據計算的工作，並且可以對運行中的任務進行多維度的監控和告警。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"調度平臺","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該層接收到平臺傳過來的任務內容和配置後，接下來就是比較核心的工作，也是下文中重點展開的內容。這裏先做一個大體的介紹，根據任務類型的不同將使用不同的插件進行解析。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據同步任務：接收到上層傳過來的 json 後，進入到 FlinkX 框架中，根據數據源端和寫出目標端的不同生成對應的 DataStream，最後轉換成 JobGraph。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據計算任務：接收到上層傳過來的 SQL 後，進入到 FlinkStreamSQL 框架中，解析 SQL、註冊成表、生成 transformation，最後轉換成 JobGraph。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"調度平臺將得到的 JobGraph 提交到對應的資源平臺，完成任務的提交。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"資源平臺","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前可以對接多套不同的資源集羣，並且也可以對接不同的資源類型，如：yarn 和 k8s.","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"數據同步和數據計算","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在調度平臺中，接收到用戶的任務後就開始了後面的一系列的轉換操作，最終讓任務運行起來。我們從底層的技術細節來看如何基於 Flink 構建實時計算平臺，以及如何使用 FlinkX、FlinkStreamSQL 做一站式開發。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"FlinkX","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作爲數據處理的第一步，也是最基礎的一步，我們來看看 FlinkX 是如何在 Flink 的基礎上做二次開發。用戶只需要關注同步任務的 json 腳本和一些配置，無需關心調用 Flink 的細節，且 FlinkX 支持下圖中所展示的功能。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f9/f9b8e4aa8bed3314adae658b82b80b59.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"img","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們先看下 Flink 任務提交中涉及到的流程，其中的交互流程圖如下：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/69/69513e7790c88efcd8914a7259d70155.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"img","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那麼 FlinkX 又是如何在 Flink 的基礎對上述組件進行封裝和調用，使得 Flink 作爲數據同步工具使用更加簡單？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主要從 Client、JobManager、TaskManager 三個部分進行擴展，涉及到的內容如下圖：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c5/c553bd861d3024e016e4e63baea25cdc.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"img","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Client 端","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"FlinkX 對原生的 Client 做了部分定製化開發，在 ","attrs":{}},{"type":"link","attrs":{"href":"https://links.jianshu.com/go?to=http%3A%2F%2Fgitlab.prod.dtstack.cn%2Fdt-insight-engine%2Fflinkx%2Ftree%2Fmaster%2Fflinkx-launcher","title":null,"type":null},"content":[{"type":"text","text":"FlinkX-launcher","attrs":{}}]},{"type":"text","text":" 模塊下，主要有以下幾個步驟：","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"解析參數，如：並行度、savepoint 路徑、程序的入口 jar 包(平常寫的 Flink demo)、Flink-conf.yml 中的配置等;","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"通過程序的入口 jar 包、外部傳入參數、savepoint 參數生成 PackagedProgram;","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"通過反射調用 PackagedProgram 中指定的程序的入口 jar 包的 main 方法，在 main 方法中，通過用戶配置的 reader 和 writer 的不同，加載對應的插件;","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"生成 JobGraph，將其中需要的資源 (Flink 需要的 jar 包、reader 和 writer 的 jar 包、Flink 配置文件等) 加入到 YarnClusterDescriptor 的 shipFiles 中，最後 YarnClusterDescriptor 就可以和 yarn 交互啓動 JobManager；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"任務提交成功後，Client 端就可得到 yarn 返回的 applicationId，後續既可以通過 application 跟蹤任務的狀態。","attrs":{}}]}]}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"JobManager 端","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Client 端提交完後，隨後 yarn 啓動 jobmanager，jobmanager 會啓動一些自己的內部服務，並且會構建 ExecutionGraph。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在這個過程中，FlinkX 主要做了以下兩件事：","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"用不同插件重寫 InputFormat 接口中的 createInputSplits 的方法創建分片，在上游數據量較大或者需要多並行度讀取的時候，該方法就起到給每個並行度設置不同的分片的作用。比如：在兩個並行度讀取 MySQL 時，通過配置的分片字段 (比如自增主鍵 ID)。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"第一個並行度讀取 SQL 爲：select * from table where id mod 2=0;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"第二個並行度讀取 SQL 爲：select * from table where id mod 2=1;","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"分片創建完後，通過 getInputSplitAssigner 按順序返回分配給各個併發實例。","attrs":{}}]}]}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"TaskManager 端","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 TaskManager 端接收到 JobManager 調度過來的 task 之後，就開始了自己的生命週期的調用，主要包含以下幾個重要的階段：","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"initialize-operator-states()：","attrs":{}},{"type":"text","text":"循環遍歷該 task 所有的 operator，並調用實現了 CheckpointedFunction 接口的 initializeState 方法，在 FlinkX 中爲 DtInputFormatSourceFunction 和 DtOutputFormatSinkFunction，該方法在任務第一次啓動的時候會被調用，作用是恢復狀態，當任務失敗時可以從最近一次的 checkpoint 恢復讀取位置，從而達到可以續跑的目的，如下圖所示：","attrs":{}}]}]}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/50/50f1cc739f1651f11f706532c75bc4ce.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"img","attrs":{}}]},{"type":"numberedlist","attrs":{"start":"2","normalizeStart":"2"},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"open-operators()：","attrs":{}},{"type":"text","text":"該方法調用 OperatorChain 中所有 StreamOperator 的 open 方法，最後調用的是 BaseRichInputFormat 中的 open 方法。該方法主要做以下幾件事：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"初始化累加器，記錄讀入、寫出的條數、字節數；","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"初始化自定義的 Metric；","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"開啓限速器；","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":6,"align":null,"origin":null},"content":[{"type":"text","text":"初始化狀態；","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":7,"align":null,"origin":null},"content":[{"type":"text","text":"打開讀取數據源的連接 (根據數據源的不同，每個插件各自實現)。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":8,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"run()：","attrs":{}},{"type":"text","text":"調用 InputFormat 中的 nextRecord 方法、OutputFormat 中的 writeRecord 方法進行數據的處理。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":9,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"close-operators()：","attrs":{}},{"type":"text","text":"做一些關閉操作，例如調用 InputFormat、OutputFormat 的 close 方法等，並做一些清理工作。","attrs":{}}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上就是TaskManager 中 StreamTask 整體的生命流程，除了上面介紹的 FlinkX 如何調用 Flink 接口，FlinkX 還有如下一些特性。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"自定義累加器：","attrs":{}},{"type":"text","text":"累加器是從用戶函數和操作中，分佈式地統計或者聚合信息。每個並行實例創建並更新自己的 Accumulator 對象, 然後合併收集不同並行實例，在作業結束時由系統合併，並可將結果推動到普羅米修斯中，如圖：","attrs":{}}]}]}],"attrs":{}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a4/a41c3f733f65af07cfbdb19d0c93bbab.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"img","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"支持離線和實時同步：","attrs":{}},{"type":"text","text":"我們知道 FlinkX 是一個支持離線和實時同步的框架，這裏以 MySQL 數據源爲例，看看是如何實現的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"離線任務：在 DtInputFormatSourceFunction 的 run 方法中會調用 InputFormat 的 open 方法，讀取數據記錄到 resultSet 中，之後再調用 reachedEnd 方法，來判斷 resultSet 的數據是否讀取完。如果讀取完，就走後續的 close 流程。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"實時任務：open 方法和離線一致，在 reachedEnd 時判斷是否是輪詢任務，如果是，則會進入到間隔輪詢的分支中，將上一次輪詢讀取到的最大的一個增量字段值，作爲本次輪詢的開始位置，並進行下一次輪詢，輪詢流程圖如下：","attrs":{}}]}]}],"attrs":{}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c8/c8e6eec917db8d341ad710647ecdae1a.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"img","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"髒數據管理和錯誤控制：","attrs":{}},{"type":"text","text":"把寫入數據源時出錯的數據記錄下來，並把錯誤原因分類，然後寫入配置的髒數據表。錯誤原因目前有：類型轉換錯誤、空指針、主鍵衝突和其它錯誤四類。錯誤控制是基於 Flink 的累加器，在運行過程中記錄出錯的記錄數，然後在單獨的線程裏定時判斷錯誤的記錄數是否已經超出配置的最大值，如果超出，則拋出異常使任務失敗。這樣可以對數據精確度要求不同的任務，做不同的錯誤控制，控制流程圖如下：","attrs":{}}]}]}],"attrs":{}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/57/5745c891644a85225a593e1d18ecbf77.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"img","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"限速器：","attrs":{}},{"type":"text","text":"一些上游數據產生過快的任務，會對下游數據庫造成較大的壓力，故而需要在源端做一些速率控制，FlinkX 使用的是令牌桶限流的方式控制速率。如下圖，當源端產生數據的速率達到某個閾值時，就不會再讀取新的數據，在 BaseRichInputFormat的open 階段也初始化了限速器。","attrs":{}}]}]}],"attrs":{}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/03/03e6ff3252ca3cd3496d5e60c3abf568.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"img","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上就是 FlinkX 數據同步的基本原理，但是數據業務場景中數據同步只是第一步，由於 FlinkX 目前的版本中只有 ETL 中的 EL，並不具備對數據的轉換和計算的能力，故而需要將產生的數據流入到下游的 FlinkStreamSQL。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"FlinkStreamSQL","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於 Flink，對其實時 SQL 進行擴展，主要擴展了流與維表的 join，並支持原生 Flink SQL 所有的語法。目前 FlinkStreamSQL source 端只能對接 Kafka，所以默認上游數據來源都是 Kafka。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來我們看看 FlinkStreamSQL 如何在 Flink 基礎上做到，用戶只需要關注業務 SQL 代碼，如何調用 Flink api 來屏蔽底層。整體流程和上面介紹的 FlinkX 基本類似，不同點在 Client 端，這裏主要包括 SQL 解析、註冊表、執行 SQL 三個部分。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/40/40e0e1ffbfb609bce56f8518aee4e24d.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"img","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"解析 SQL","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏主要是解析用戶寫的 create function、create table、create view、insert into 四種 SQL 語句，封裝到結構化的 SQLTree 數據結構中。SQLTree 中包含了自定義函數集合、外部數據源表集合、視圖語句集合、寫數據語句集合。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"表註冊","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"得到了上面解析的 SQLTree 之後，就可以將 SQL中create table 語句對應的外部數據源集合作爲表註冊到 tableEnv 中，並且將用戶自定的 UDF 註冊進 tableEnv 中。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"執行 SQL","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將數據源註冊成表之後，就可以執行後面 insert into 的 SQL 語句了，執行 SQL 這裏會分兩種情況：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SQL 中沒有關聯維表，就直接執行 SQL；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SQL 中關聯了維表，由於在 Flink 早期版本中不支持維表 join 語法，我們在這塊做了擴展，不過在 FlinkStreamSQL v1.11 之後和社區保持了一致，支持了和維表 join 的語法。根據維表的類型不同，使用不同的關聯方式：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"全量維表：將上游數據作爲輸入，使用 RichFlatMapFunction 作爲查詢算子，初始化時將數據全表撈到內存中，然後和輸入數據組拼得到打寬後的數據，之後重新註冊一張大表，供後續 SQL 使用。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"異步維表：將上游數據作爲輸入，使用 RichAsyncFunction 作爲查詢算子，並將查詢得到的數據使用 LRU 緩存，然後和輸入數據組拼得到打寬後的數據，之後重新註冊一張大表，供後續SQL使用。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上面介紹的就是 FlinkX 和 FlinkStramSQL 在 Client 端的不同之處，由於 source 端只有 Kafka 且使用了社區原生的 Kafka-connector，所以在 jobmanager 端也沒有數據分片的邏輯，taskmanager 邏輯和 FlinkX 基本類似，這裏不再介紹。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"任務運維","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當使用 FlinkX 和 FlinkStreamSQL 開發完業務之後，接下來進入到了任務運維階段。在運維階段，我們主要在任務運行信息、數據進出指標 metrics、數據延遲、反壓、數據傾斜等維度做了監控。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"任務運行信息","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們知道 FlinkStreamSQL 是基於 FlinkSQL 封裝的，所以在提交任務運行時最終還是走的 FlinkSQL 的解析、驗證、邏輯計劃、邏輯計劃優化、物理計劃，最後將任務運行起來，也就得到了我們經常看見的 DAG 圖：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/6b/6b2fcda230652fc3a4cf58f06a339db5.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"img","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是由於 FlinkSQL 對任務做了很多優化，以至於我們只能看到如上圖的大體 DAG 圖，子 DAG 圖裏面的一些細節我們是沒法直觀的看到發生了什麼事情。所以我們在原來生成 DAG 圖的方式上進行了一定的改造，這樣就能直觀的看到子 DAG 圖中每個 Operator 和每個並行度裏面發生了什麼事情，有了詳細的 DAG 圖後，其他的一些監控維度就能直觀的展示，比如：數據輸入輸出、延時、反壓、數據傾斜，在出現問題時就能具體定位到，如下圖的反壓：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/33/335d156cb3a3a272dd1d919356bef690.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"img","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"瞭解了上面的結構後，我們來看看它是如何實現的。我們知道在 Client 提交任務時，會生成 JobGraph，JobGraph 中的 taskVertices 集合就封裝了上圖完整的信息，我們將 taskVertices 生成 json 後，再結合 LatencyMarker 和相關的 metrics，即可在前端生成上圖，並做相應的告警。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了上面的 DAG 以外，還有自定義 metrics、數據延時獲取等，這裏不具體介紹，有興趣的同學可以參考 FlinkStreamSQL 項目。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"使用案例：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過上面的介紹後，我們看下在平臺上使用的實際案例。下面展示了一個完整的案例：使用 FlinkX 將 MySQL 中新增用戶數據實時同步到 Kafka，然後使用 FlinkstreamSQL 消費 Kafka 實時計算每分鐘新增用戶數，產出結果落庫到下游 MySQL，供業務使用。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"實時同步 MySQL 新增數據","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/20/20a142a6b7fe400ef568cee33ba96b57.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"img","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"實時計算每分鐘新增用戶數","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5d/5d1d53c17b1e3814b8a43500e8497f3f.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"img","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"運行信息","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整體 DAG，可以直觀的顯示上面提到的多項指標","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b2/b26e5bdf5f89fecbf73327156795b570.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"img","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"解析後的詳細 DAG 圖，可以看到子 DAG 內部的多項指標","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d6/d6a92cab1ce9e4bc017865f3eb15fa9b.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"img","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ec/ecd656cb65a69616f6ae5b3078435061.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"img","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上就是 Flink 在袋鼠雲實時計算平臺的總體架構和一些關鍵的技術點，如有不足之處歡迎大家指出。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"0人點贊","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://www.jianshu.com/nb/37017347","title":null,"type":null},"content":[{"type":"text","text":"Apache Flink","attrs":{}}]}]}]}