5年迭代5次，抖音推薦特徵體系演進歷程

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2021年，字節跳動旗下產品總 MAU 已超過19億。在以抖音、今日頭條、西瓜視頻等爲代表的產品業務背景下，強大的推薦系統顯得尤爲重要。Flink 提供了非常強大的 SQL 模塊和有狀態計算模塊。目前在字節推薦場景，實時簡單計數特徵、窗口計數特徵、序列特徵已經完全遷移到 Flink SQL 方案上。結合 Flink SQL 和 Flink 有狀態計算能力，我們正在構建下一代通用的基礎特徵計算統一架構，期望可以高效支持常用有狀態、無狀態基礎特徵的生產。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"業務背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/54\/54e74d54df4a52d8b09d247246256b44.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"對於今日頭條、抖音、西瓜視頻等字節跳動旗下產品，基於 Feed 流和短時效的推薦是核心業務場景。而推薦系統最基礎的燃料是特徵，高效生產基礎特徵對業務推薦系統的迭代至關重要。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"主要業務場景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/83\/831d192142aa8810e8d1eff597720a02.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"抖音、火山短視頻等爲代表的短視頻應用推薦場景，例如 Feed 流推薦、關注、社交、同城等各個場景，整體在國內大概有6億+規模 DAU；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"頭條、西瓜等爲代表的 Feed 信息流推薦場景，例如 Feed 流、關注、子頻道等各個場景，整體在國內大概有1.5億+規模 DAU；"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"業務痛點和挑戰"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/25\/2550175b5c2e7057750a74f7ae7504e0.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"目前字節跳動推薦場景基礎特徵的生產現狀是“"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"百花齊放"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"”。離線特徵計算的基本模式都是通過消費 Kafka、BMQ、Hive、HDFS、Abase、RPC 等數據源，基於 Spark、Flink 計算引擎實現特徵的計算，而後把特徵的結果寫入在線、離線存儲。各種不同類型的基礎特徵計算散落在不同的服務中，缺乏業務抽象，帶來了較大的運維成本和穩定性問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"而更重要的是，缺乏統一的基礎特徵生產平臺，使業務特徵開發迭代速度和維護存在諸多不便。如業務方需自行維護大量離線任務、特徵生產鏈路缺乏監控、無法滿足不斷髮展的業務需求等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/6d\/6dd6cad8922304cfab6555531c87c3ee.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" 在字節的業務規模下，構建統一的實時特徵生產系統面臨着較大挑戰，主要來自四個方面："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"巨大的業務規模"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"：抖音、頭條、西瓜、火山等產品的數據規模可達到日均 PB 級別。例如在抖音場景下，晚高峯 Feed 播放量達數百萬 QPS，客戶端上報用戶行爲數據"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"高達數千萬 IOPS。"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"業務方期望在任何時候，特徵任務都可以做到不斷流、消費沒有 lag 等，這就要求特徵生產具備非常高的穩定性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"較高的特徵實時化要求"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"：在以直播、電商、短視頻爲代表的推薦場景下，爲保證推薦效果，實時特徵離線生產的時效性需實現常態穩定於分鐘級別。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"更好的擴展性和靈活性"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"：隨着業務場景不斷複雜，特徵需求更爲靈活多變。從統計、序列、屬性類型的特徵生產，到需要靈活支持窗口特徵、多維特徵等，業務方需要特徵中臺能夠支持逐漸衍生而來的新特徵類型和需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"業務迭代速度快"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"：特徵中臺提供的面向業務的DSL需要足夠場景，特徵生產鏈路儘量讓業務少寫代碼，底層的計算引擎、存儲引擎對業務完全透明，徹底釋放業務計算、存儲選型、調優的負擔，徹底實現實時基礎特徵的規模化生產，不斷提升特徵生產力；"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"迭代演進過程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在字節業務爆發式增長的過程中，爲了滿足各式各樣的業務特徵的需求，推薦場景衍生出了衆多特徵服務。這些服務在特定的業務場景和歷史條件下較好支持了業務快速發展，大體的歷程如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/aa\/aaaf625e86241ce935ea3b630544f6fb.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"推薦場景特徵服務演進歷程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"在這其中2020年初是一個重要節點，我們開始在特徵生產中引入 Flink SQL、Flink State 技術體系，逐步在計數特徵系統、模型訓練的樣本拼接、窗口特徵等場景進行落地，探索出新一代特徵生產方案的思路。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"新一代系統架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"結合上述業務背景，我們基於 Flink SQL 和 Flink 有狀態計算能力重新設計了新一代實時特徵計算方案。"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"新方案的定位是：解決基礎特徵的計算和在線 Serving，提供更加抽象的基礎特徵業務層 DSL。"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在計算層，我們基於 Flink SQL 靈活的數據處理表達能力，以及 Flink State 狀態存儲和計算能力等技術，支持各種複雜的窗口計算。極大地縮短業務基礎特徵的生產週期，提升特徵產出鏈路的穩定性。新的架構裏，我們將"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"特徵生產的鏈路分爲數據源抽取\/拼接、狀態存儲、計算三個階段，"},{"type":"text","marks":[{"type":"strong"}],"text":"Flink SQL完成特徵數據的抽取和流式拼接，Flink State 完成特徵計算的中間狀態存儲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"有狀態特徵是非常重要的一類特徵，其中最常用的就是帶有各種窗口的特徵，例如統計最近5分鐘視頻的播放 VV 等。對於窗口類型的特徵在字節內部有一些基於存儲引擎的方案，整體思路是“"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"輕離線重在線"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"”，即把窗口狀態存儲、特徵聚合計算全部放在存儲層和在線完成。離線數據流負責基本數據過濾和寫入，離線明細數據按照時間切分聚合存儲（類似於 micro batch），底層的存儲大部分是 KV 存儲、或者專門優化的存儲引擎，在線層完成複雜的窗口聚合計算邏輯，每個請求來了之後在線層拉取存儲層的明細數據做聚合計算。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我們新的解決思路是“"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"輕在線重離線"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"”，即把比較重的"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"時間切片明細數據"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"狀態存儲和窗口聚合計算全部放在離線層。窗口結果聚合通過"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"離線窗口觸發機制"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"完成，把特徵結果"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"推到"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在線 KV 存儲。在線模塊非常輕量級，只負責簡單的在線 serving，極大地簡化了在線層的架構複雜度。在離線狀態存儲層。我們主要依賴 Flink 提供的"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"原生狀態存儲引擎 RocksDB"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"，充分利用離線計算集羣本地的 SSD 磁盤資源，極大減輕在線 KV 存儲的資源壓力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"對於長窗口的特徵（7天以上窗口特徵），由於涉及 Flink 狀態層明細數據的回溯過程，Flink Embedded 狀態存儲引擎沒有提供特別好的外部數據回灌機制（或者說不適合做）。因此對於這種“"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"狀態冷啓動"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"”場景，我們引入了中心化存儲作爲底層狀態存儲層的存儲介質，整體是 "},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"Hybrid "},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"架構。例如7天以內的狀態存儲在本地 SSD，7～30天狀態存儲到中心化的存儲引擎，離線數據回溯可以非常方便的寫入中心化存儲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"除窗口特徵外，這套機制同樣適用於其他類型的有狀態特徵（如序列類型的特徵）。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"實時特徵分類體系"}]},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"

特徵類型

定義

特徵舉例

有狀態特徵

 

有狀態特徵是一類非常重要的特徵，我們對有狀態特徵的定義是：計算特徵需要緩存上下文數據。

 

• 帶有窗口的特徵，例如抖音視頻最近1h的點贊量（滑動窗口）、直播間用戶最近一個 session 的看播時長（session 窗口）等；
• 序列特徵，例如最近100個推薦展現視頻。

無狀態特徵

 

簡單的 ETL 特徵，通過簡單的數據過濾可以計算的特徵。

 

 

模型預估特徵

 

需要經過外部複雜模型預估的特徵

 

用戶的年齡、性別等特徵。

 

圖特徵

 

在直播和社交關係場景存在比較多的需要二跳關係的圖類型的特徵。

很多圖特徵同時也是有狀態類型的特徵。

 

• 禮物排序：用戶觀看最多的主播收到最多的禮物，首選需要找到用戶觀看最多的主播 ArchorId，然後通過 archon_id 獲取到主播收到最多的禮物 id；
• 社交關係：好友（可能是挖掘出來的關係）關注、看播、送禮、連麥的房間，社交關係天然是圖數據結構。"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"整體架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/72\/72f4d8a53303fa49b1c649d5eb7e0d41.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"數據源層"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在新的一體化特徵架構中，我們統一把各種類型數據源抽象爲 Schema Table，這是因爲底層依賴的 Flink SQL 計算引擎層對數據源提供了非常友好的 Table Format 抽象。在推薦場景，依賴的數據源非常多樣，每個特徵上游依賴一個或者多個數據源。數據源可以是 Kafka、RMQ、KV 存儲、RPC 服務。對於多個數據源，支持數據源流式、批式拼接，拼接類型包括 Window Join 和基於 key 粒度的 Window Union Join，維表 Join 支持 Abase、RPC、HIVE 等。具體每種類型的拼接邏輯如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"

  數據源類型

  Schema 解析

  Kafka、BMQ

   

  Kafka、BMQ 等 message 類型基本都是 JSON 和 PB，是自描述的數據類型。可以非常方便地映射成 SchemaTable 格式，其中對於 PB 類型，業務需要上傳 PB IDL 完成 Table Schema 定義。

  KV存儲

   

  KV 存儲裏的 Value 大部分爲 JSON、PB 格式，和 MQ 類似。業務方通過提供 PB IDL 完成 Table Schema 定義。我們通過 FlinkSQL 的維表 Join 能力，把普通的獲取外部存儲數據源過程抽象爲基本的維表 Join 操作，簡化業務開發週期。

  RPC

   

  FlinkSQL 提供了對 RPC 維表的 Join 能力，業務提供 RPC Thrift IDL 完整 rpc response Table Schema 定義。通過維表 Join，我們把普通的通過 RPC 獲取外部數據源的過程抽象爲了基本維表 Join 模型，簡化業務開發週期。

  Hive

   

  Hive 本身就是 SchemaTable 的存儲格式，對於在線 Join 數據量較小的離線 Hive 數據(其實就是 MapSide Join)，可通過 Hive 維表 Join 實現。"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"三種類型的 Join 和 Union 可以組合使用，實現複雜的多數據流拼接。例如"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"(A union B) Window Join (C Lookup Join D)。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"

  拼接類型

  拼接邏輯

  備註

  Window Join

   

  使用 Flink 原生 API 提供的 Join 算子，把多個數據流落入相同窗口的數據 Join 起來。

   

  直接在原始數據流上應用 TumblingWindow 進行切分，根據event_time 或 process_time 對齊兩個窗口後再關聯數據。

   

  基於 Key 粒度的 Interval State Join

   

  和樣本拼接邏輯類似。通過 Union 上游多個數據源，在每個關聯主鍵上面註冊 timer，等待一個固定的時間窗口完成多數據源的 Join 操作。

   

  Interval State Join 是利用 State 存儲數據再處理。上游兩個數據流經過 Union 後，同一個 uid 的 instance 數據和 label 數據落在同一個 operator 內，Joiner 中正負例樣本的產生就是通過這種 Join 方式。

   

  Lookup 維表 Join

   

  通過關聯主鍵，從 Abase、RPC、Hive 等服務查看需要關聯的數據，完成數據的 Join 操作。

   

   

  多數據源 Union

  多數據源 Union 起來

   

   "}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"另外，Flink SQL 支持複雜字段的計算能力，也就是業務方可以基於數據源定義的 TableSchema 基礎字段實現擴展字段的計算。業務計算邏輯本質是一個 UDF，我們會提供 UDF API 接口給業務方，然後上傳JAR 到特徵後臺加載。另外對於比較簡單的計算邏輯，後臺也支持通過提交簡單的 Python 代碼實現多語言計算。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"業務 DSL"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"從業務視角提供高度抽象的特徵生產 DSL 語言，屏蔽底層計算、存儲引擎細節，讓業務方聚焦於業務特徵定義。業務 DSL 層提供：數據來源、數據格式、數據抽取邏輯、數據生成特徵類型、數據輸出方式等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7a\/7a28ef7b9f0cd946fa057a95fe65c963.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"狀態存儲層"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ad\/add3b7c773c2a61bda2cf8ed19e4782b.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"如上文所述，新的特徵一體化方案解決的主要痛點是：如何應對各種類型（一般是滑動窗口）有狀態特徵的計算問題。對於這類特徵，在離線計算層架構裏會有一個狀態存儲層，把抽取層提取的 RawFeature 按照切片 Slot 存儲起來(切片可以是時間切片、也可以是 Session 切片等)。切片類型在內部是一個接口類型，在架構上可以根據業務需求自行擴展。狀態裏面其實存儲的不是原始 RawFeature（存儲原始的行爲數據太浪費存儲空間），而是轉化爲 FeaturePayload 的一種 POJO 結構，這個結構裏面支持了常見的各種數據結構類型："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Int：存儲簡單的計數值類型(多維度 counter)；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"HashMap：存儲二維計數值，例如 Action Counter，key 爲 target_id，value 爲計數值；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"SortedMap: 存儲 topk 二維計數 ；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"LinkedList： 存儲 id_list 類型數據；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"HashMap>：存儲二維 id_list；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"自定義類型，業務可以根據需求 FeaturePayload 裏面自定義數據類型"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"狀態層更新的業務接口：輸入是 SQL 抽取\/拼接層抽取出來的 RawFeature，業務方可以根據業務需求實現 updateFeatureInfo 接口對狀態層的更新。對於常用的特徵類型內置實現了 update 接口，業務方自定義特徵類型可以繼承 update 接口實現。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"\/**\n * 特徵狀態update接口\n *\/\npublic interface FeatureStateApi extends Serializable {\n \/**\n * 特徵更新接口, 上游每條日誌會提取必要字段轉換爲fields, 用來更新對應的特徵狀態\n *\n * @param fields\n * context: 保存特徵名稱、主鍵 和 一些配置參數;\n * oldFeature: 特徵之前的狀態\n * fields: 平臺\/配置文件 中的抽取字段\n * @return\n *\/\nFeaturePayLoad assign(Context context,FeaturePayLoad feature, Map rawFeature);\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"當然對於無狀態的 ETL 特徵是不需要狀態存儲層的。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"計算層"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"特徵計算層完成特徵計算聚合邏輯，有狀態特徵計算輸入的數據是狀態存儲層存儲的帶有切片的 FeaturePayload 對象。簡單的 ETL 特徵沒有狀態存儲層，輸入直接是 SQL 抽取層的數據 RawFeature 對象，具體的接口如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"\/**\n * 有狀態特徵計算接口\n *\/\npublic interface FeatureStateApi extends Serializable {\n\n\n \/**\n * 特徵聚合接口，會根據配置的特徵計算窗口, 讀取窗口內所有特徵狀態，排序後傳入該接口\n *\n * @param featureInfos, 包含2個field\n * timeslot: 特徵狀態對應的時間槽\n * Feature: 該時間槽的特徵狀態\n * @return\n *\/\n FeaturePayLoad aggregate(Context context, List> slotStates);\n\n\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"有狀態特徵聚合接口"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"\/**\n * 無狀態特徵計算接口\n *\/\npublic interface FeatureConvertApi extends Serializable {\n\n\n \/**\n * 轉換接口, 上游每條日誌會提取必要字段轉換爲fields, 無狀態計算時，轉換爲gauss內的feature類型;\n *\n * @param fields\n * fields: 平臺\/配置文件 中的抽取字段\n * @return\n *\/\n FeaturePayLoad convert(Context context, FeaturePayLoad featureSnapshot, Map rawFeatures);\n\n\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"無狀態特徵計算接口"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"另外通過觸發機制來觸發特徵計算層的執行，目前支持的觸發機制主要有："}]},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"

  策略	解釋
  OnTimerTrigger	週期性定時觸發特徵的計算邏輯
  OnUpdateTrigger	上游狀態層每次更新即觸發特徵計算
  CustomTrigger	自定義特徵計算的觸發時機  "}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"業務落地"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"目前在字節推薦場景，新一代特徵架構已經在"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"抖音直播、電商、推送、抖音推薦"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"等場景陸續上線了一些實時特徵。主要是有狀態類型的特徵，帶有窗口的一維統計類型、二維倒排拉鍊類型、二維 TOPK 類型、實時 CTR\/CVR Rate 類型特徵、序列類型特徵等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在業務核心指標達成方面成效顯著。在直播場景，依託新特徵架構強大的表達能力上線了一批特徵之後，業務看播核心指標、互動指標收益非常顯著。在電商場景，基於新特徵架構上線了400+實時特徵。其中在直播電商方面，業務核心 GMV、下單率指標收益顯著。在抖音推送場景，基於新特徵架構離線狀態的存儲能力，聚合用戶行爲數據然後寫入下游各路存儲，極大地緩解了業務下游數據庫的壓力，在一些場景中 QPS 可以下降到之前的10%左右。此外，抖音推薦 Feed、評論等業務都在基於新特徵架構重構原有的特徵體系。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"值得一提的是，在電商和抖音直播場景，Flink 流式任務狀態最大已經達到60T，而且這個量級還在不斷增大。"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"預計不久的將來，單任務的狀態有可能會突破100T"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"，這對架構的穩定性是一個不小的挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"性能優化"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Flink State Cache"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"目前 Flink 提供兩類 StateBackend：基於 Heap 的 FileSystemStateBackend 和基於 RocksDB 的 RocksDBStateBackend。對於 FileSystemStateBackend，由於數據都在內存中，訪問速率很快，沒有額外開銷。而 RocksDBStateBackend 存在查盤、序列化\/反序列化等額外開銷，CPU 使用量會有明顯上升。在字節內部有大量使用 State 的作業，對於大狀態作業，通常會使用 RocksDBStateBackend 來管理本地狀態數據。RocksDB 是一個 KV 數據庫，以 LSM 的形式組織數據，在實際使用的過程中，有以下特點"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"應用層和 RocksDB 的數據交互是以 Bytes數組的形式進行，應用層每次訪問都需要序列化\/反序列化；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"數據以追加的形式不斷寫入 RocksDB 中，RocksDB 後臺會不斷進行 compaction 來刪除無效數據。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"業務方使用 State 的場景多是 get-update，在使用 RocksDB 作爲本地狀態存儲的過程中，出現過以下問題："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"爬蟲數據導致熱 key，狀態會不斷進行更新(get-update)，單 KV 數據達到 5MB，而 RocksDB 追加更新的特點導致後臺在不斷進行 flush 和 compaction，單 task 出現慢節點（抖音直播場景）。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"電商場景作業多數爲大狀態作業(目前已上線作業狀態約60TB)，業務邏輯中會頻繁進行 State 操作。在融合 Flink State 過程中發現 CPU 的開銷和原有"},{"type":"text","marks":[{"type":"del"},{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"的"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"基於內存或 abase 的實現有40%~80%的升高。經優化後，CPU 開銷主要集中在序列化\/反序列化的過程中。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"針對上述問題，可以通過在內存維護一個對象 Cache，達到優化熱點數據訪問和降低 CPU 開銷的目的。通過上述背景介紹，我們希望能爲 StateBackend 提供一個通用的 Cache 功能，通過 Flink StateBackend Cache 功能設計方案達成以下目標："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"減少 CPU 開銷"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"："},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"通過對熱點數據進行緩存，減少和底層 StateBackend 的交互次數，達到減少序列化\/反序列化開銷的目的。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"提升 State 吞吐能力"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"："},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"通過增加 Cache 後，State 吞吐能力應比原有的 StateBackend 提供的吞吐能力更高。理論上在 Cache 足夠大的情況下，吞吐能力應和基於 Heap 的 StateBackend 近似。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Cache 功能通用化"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"："},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"不同的 StateBackend 可以直接適配該 Cache 功能。目前我們主要支持 RocksDB，未來希望可以直接提供給別的 StateBackend 使用，例如 RemoteStateBackend。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"經過和字節基礎架構 Flink 團隊的合作，在實時特徵生產升級"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"，上線 Cache 大部分場景的 CPU 使用率大概會有高達50%左右的收益；"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"PB IDL 裁剪"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在字節內部的實時特徵離線生成鏈路當中，我們主要依賴的數據流是 Kafka。這些 Kafka 都是通過 PB 定義的數據，字段繁多。公司級別的大 Topic 一般會有100+的字段，但大部分的特徵生產任務只使用了其中的部分字段。對於 Protobuf 格式的數據源，我們可以完全通過裁剪數據流，mask 一些非必要的字段來節省反序列化的開銷。PB 類型的日誌，可以直接裁剪 idl，保持必要字段的序號不變，在反序列化的時候會跳過 unknown field 的解析，這"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"對於 CPU 來說是更節省的，但是網絡帶寬不會有收益，"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"預計裁剪後能節省非常多的 CPU 資源。在上線了 PB IDL 裁剪之後，"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"大部分任務的 CPU 收益在30%左右。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"遇到的問題"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"新架構特徵生產任務本質就是一個有狀態的 Flink 任務，底層的狀態存儲 StateBackend 主要是本地的 RocksDB。主要面臨兩個比較難解的問題，一是任務 DAG 變化 Checkpoint 失效，二是本地存儲不能很好地支持特徵狀態歷史數據回溯。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"實時特徵任務不能動態添加新的特徵：對於一個線上的 Flink 實時特徵生產任務，我們不能隨意添加新的特徵。這是由於引入新的特徵會導致 Flink 任務計算的 DAG 發生改變，從而導致 Flink 任務的 Checkpoint 無法恢復，這對實時有狀態特徵生產任務來說是不能接受的。目前我們的解法是禁止更改線上部署的特徵任務配置，但這也就導致了線上生成的特徵是不能隨便下線的。對於這個問題暫時沒有找到更好的解決辦法，後期仍需不斷探索。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"特徵狀態冷啓動問題：目前主要的狀態存儲引擎是 RocksDB，不能很好地支持狀態數據的回溯。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"後續規劃"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"當前新一代架構還在字節推薦場景中快速演進，目前已較好解決了實時窗口特徵的生產問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"出於實現統一推薦場景下特徵生產的目的，我們後續會繼續基於 Flink SQL 流批一體能力，在批式特徵生產發力。此外也會基於 Hudi 數據湖技術，完成特徵的實時入湖，高效支持模型訓練場景離線特徵回溯痛點。規則引擎方向，計劃繼續探索 CEP，推動在電商場景有更多落地實踐。在實時窗口計算方向，將繼續深入調研 Flink 原生窗口機制，以期解決目前方案面臨的窗口特徵數據退場問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"支持批式特徵"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"：這套特徵生產方案主要是解決實時有狀態特徵的問題，而目前字節離線場景下還有大量批式特徵是通過 Spark SQL 任務生產的。後續我們也會基於 Flink SQL 流批一體的計算能力，提供對批式場景特徵的統一支持，目前也初步有了幾個場景的落地；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"特徵離線入湖"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"：基於 Hudi On Flink 支持實時特徵的離線數倉建設，主要是爲了支持模型訓練樣本拼接場景離線特徵回溯；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"Flink CEP 規則引擎支持"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"：Flink SQL 本質上就是一種規則引擎，目前在線上我們把 Flink SQL 作爲業務 DSL 過濾語義底層的執行引擎。但 Flink SQL 擅長表達的 ETL 類型的過濾規則，不能表達帶有時序類型的規則語義。在直播、電商場景的時序規則需要嘗試 Flink CEP 更加複雜的規則引擎。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"Flink Native Windowing 機制引入"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"：對於窗口類型的有狀態特徵，我們目前採用上文所述的抽象 SlotState 時間切片方案統一進行支持。另外 Flink 本身提供了非常完善的窗口機制，通過 Window Assigner、Window Trigger 等組件可以非常靈活地支持各種窗口語義。因此後續我們也會在窗口特徵計算場景引入 Flink 原生的 Windowing 機制，更加靈活地支持窗口特徵迭代。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"Flink HybridState Backend 架構"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"：目前在字節的線上場景中，Flink 底層的 StateBackend 默認都是使用 RocksDB 存儲引擎。這種內嵌的存儲引擎不能通過外部機制去提供狀態數據的回灌和多任務共享，因此我們需要支持 \bKV 中心化存儲方案，實現靈活的特徵狀態回溯。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"靜態屬性類型特徵統一管理"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"：通過特徵平臺提供統一的 DSL 語義，統一管理其他外部靜態類型的特徵服務。例如一些其他業務團隊維度的用戶分類、標籤服務等。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"作者介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"郭文飛，字節跳動推薦系統基礎服務方向負責人。2015年初加入字節，主要負責推薦系統基礎服務方向，例如消重、計數、特徵等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"字節跳動推薦架構團隊實時計算方向，負責抖音、今日頭條、西瓜視頻等超10億用戶產品推薦系統架構實時計算系統的設計和開發，保障系統穩定和高可用。抽象通用實時計算系統、構建統一的推薦特徵中臺，實現靈活可擴展的高性能存儲系統和計算模型，爲推薦業務實現先進的消重、計數、特徵服務等實時推薦數據流系統。目前非常缺人，歡迎對技術有追求的同學加入，一起構建世界級先進的實時推薦數據流系統，聯繫方式：[email protected]。"}]}]}