{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Uber 致力於爲全球客戶提供可靠的服務。要達到這個目標，我們很大程度上依靠機器學習來作出明智的決定，如預測和增益。所以，用來產生機器學習數據和特徵的實時流管道已經越來越受到重視。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Uber 公司使用了 Apache Flink 來建立實時流管道，並建立像"},{"type":"link","attrs":{"href":"https:\/\/eng.uber.com\/gairos-scalability\/","title":"xxx","type":null},"content":[{"type":"text","text":" Gairos"}]},{"type":"text","text":" 和"},{"type":"link","attrs":{"href":"https:\/\/eng.uber.com\/athenax\/","title":"xxx","type":null},"content":[{"type":"text","text":" AthenaX"}]},{"type":"text","text":" 這樣的平臺來簡化開發過程。但是，由於計算的複雜性或需要處理的實時數據量，仍有很多挑戰，如擴展性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文中，我們將以生產需求和供應特徵爲例，介紹我們所面臨的一些挑戰以及如何應對這些挑戰。尤其要說明的是，如何使用性能調整框架來優化實時管道。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖顯示了Apache Flink 中的流管道負責特徵計算和提取的架構。我們將在下文詳細討論這些管道。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/48\/486f6f86dfc251cefabb11e07ae00e4d.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 1：簡化的架構概述"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"特徵計算"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本節詳細介紹瞭如何通過地理空間和時間維度以及全局產品（UberX 等）對任何給定的六邊形（參見"},{"type":"link","attrs":{"href":"https:\/\/eng.uber.com\/h3\/","title":"xxx","type":null},"content":[{"type":"text","text":"此處"}]},{"type":"text","text":"）的原始事件，例如需求和供應事件進行聚合。以下是簡化的計算算法："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在一分鐘窗口內，按六邊形和全局產品類型計算出不同乘客和司機所發生的原始事件數量。在一分鐘窗口內，將 Kring Smooth 應用多個環，最多 20 個環（稍後進行討論）。將每一環的平滑值聚合在多個滑動窗口大小上，最長可達 32 分鐘。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總的來說，一條實時管道每分鐘爲一個六邊形生成 54 個特徵，使用 9 個環（0，1，2，3，4，5，10，15，20）和 6 個窗口大小（1，2，4，8，16，32）的組合。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來，我們討論算法的第二步。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Kring Smooth"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kring Smooth 過程通過向其 Kring 鄰居廣播一個六邊形的事件計數來計算地理空間聚合。換句話說，某一特定環的六邊形的特徵值考慮到了該環內所有六邊形的事件計數。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了計算給定六邊形 h 在環 r 上聚合的特徵值，公式爲："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"katexinline","attrs":{"mathString":" f(H, R)=\\frac{\\sum_{i=0}^{R} \\sum_{j=1}^{N u m(i)} f\\left(N_{i}^{j}, 0\\right)}{\\sum_{i=0}^{R} Num(i)} "}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中， "},{"type":"katexinline","attrs":{"mathString":" Num(i) "}},{"type":"text","text":" 是環 "},{"type":"katexinline","attrs":{"mathString":" i "}},{"type":"text","text":" 的六邊形數量；"},{"type":"katexinline","attrs":{"mathString":" N_{i}^{j} "}},{"type":"text","text":" 是環 "},{"type":"katexinline","attrs":{"mathString":" i "}},{"type":"text","text":" 的第 "},{"type":"katexinline","attrs":{"mathString":" j "}},{"type":"text","text":" 個六邊形； "},{"type":"katexinline","attrs":{"mathString":" f(H,0) "}},{"type":"text","text":" 是來自六邊形 "},{"type":"katexinline","attrs":{"mathString":" H "}},{"type":"text","text":" 的事件數。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，讓我們看看下面的例子，看看如何計算這三個特徵的值：六邊形 A 的第 0 環、第 1 環和第 2 環，公式如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"katexinline","attrs":{"mathString":" Num(0) = 1 "}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"katexinline","attrs":{"mathString":" Num(1) = 6 "}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "},{"type":"katexinline","attrs":{"mathString":" Num(2) = 12 "}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"katexinline","attrs":{"mathString":" f(A, 0) = 1 "}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"katexinline","attrs":{"mathString":" f(A, 1) = (f(A, 0) + f(B1, 0) + f(B2, 0)) \/ (Num(0) + Num(1)) = (1 + 2 + 1) \/ 7 = 4 \/ 7 "}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "},{"type":"katexinline","attrs":{"mathString":" f(A, 2) = (f(A, 0) + f(B1, 0) + f(B2, 0) + f(C1, 0) + f(C2, 0) + f(C3, 0)) \/ (Num(0) + Num(1) + Num(2)) = (1 + 2 + 1 + 3 + 2 + 1) \/ (1 + 6 + 12) = 10 \/ 19 "}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/1f\/1f5bdf3b2ad695c4010bef37f7899fd1.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 2：六邊形 A 的第 0 環、第 1 環、第 2 環"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該管道按照方程式計算出多個環形尺寸的特徵值，最多可達 20。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"時間聚合"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在一分鐘窗口的 Kring Smooth 完成後，算法的第 3 步是將平滑的事件計數在更大的窗口上聚合，最長可達 32 分鐘。要計算給定的六邊形 H 在更大窗口上的聚集，公式如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"katexinline","attrs":{"mathString":" q(H, T, W)=\\frac{\\sum_{i=0}^{W-1} q(H, T+i, 1)}{W} "}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中，"},{"type":"katexinline","attrs":{"mathString":" T "}},{"type":"text","text":" 是一個窗口的起始時間戳；"},{"type":"katexinline","attrs":{"mathString":" W "}},{"type":"text","text":" 是窗口的大小，以分鐘爲單位；"},{"type":"katexinline","attrs":{"mathString":" q(H,T,1) "}},{"type":"text","text":" 是來自 Kring Smooth 的平滑事件計數。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖 3 展示瞭如何計算 2 分鐘窗口的六邊形 A 的特徵值："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "},{"type":"katexinline","attrs":{"mathString":" W1 "}},{"type":"text","text":" 和 數學公式: "},{"type":"katexinline","attrs":{"mathString":" W2 "}},{"type":"text","text":" 窗口的 Kring Smooth 的平滑事件計數分別爲 1.0 和 3.0，分別在"},{"type":"katexinline","attrs":{"mathString":" T0+1 "}},{"type":"text","text":" 分鐘和"},{"type":"katexinline","attrs":{"mathString":" T0+2 "}},{"type":"text","text":" 分鐘發出；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2 分鐘窗口的特徵值爲 2.0，通過使用平滑的事件計數，按照上述方程計算，其時間範圍爲 ("},{"type":"katexinline","attrs":{"mathString":" T0 "}},{"type":"text","text":"， "},{"type":"katexinline","attrs":{"mathString":" T0 + 2 "}},{"type":"text","text":" 分鐘）。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/f2\/f27708203f3b35ed84498aeade95ca08.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 3：六邊形 A 的 2 分鐘窗口的聚合"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"流實現與優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本節以需求管道爲例，說明如何在 Apache Kafka 和 Apache Flink 中實現特徵計算算法，以及如何調整實時管道。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"邏輯作業拓撲"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖 4 說明了計算需求特徵的流管道的邏輯 DAG。對於所有尺寸大於 1 分鐘的窗口來說，它們是滑動窗口，這些窗口將以 1 分鐘爲單位滑動，這意味着一個輸入事件可能包含在 63 個窗口內：32 + 16 + 8 + 4 + 2 + 1。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/56\/56176c5769b26039c5ba30280a7c7fc6.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 4：需求管道的邏輯 DAG"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下表列出了邏輯 DAG 中主要運算符的功能："}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/af\/a3\/afda73d31d7c5d5fcd66e7b25e863ea3.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"表 1：需求管道的邏輯運算符"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"流管道的數據量"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本節列出了需求管道的數據量："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kafka 主題的平均輸入速率：120k\/s"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"六角形的計數：5M"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"城市的數量：1500"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每個城市的六邊形平均數和最大數：4000 和 76000"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1 分鐘內六邊形需求事件的平均計數：45"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"環 20 的六邊形計數：1261"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"顯然，該管道具有高容量、密集的計算和大的狀態需要管理。第一版實際上是按照邏輯 DAG 構建的，由於包括背壓和 OOM 等問題，無法穩定運行（如下圖儀表板所示）。由於我們的目標是接近實時的延遲（小於 5 分鐘）， 因此我們面臨的真正挑戰是如何建立穩定的工作通道。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"內存監視器："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/14\/141c73a9375372f65362d4bd61d383be.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 5：已用內存的儀表板"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"延遲監視器："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/24\/2411817a40e90424aa0e9b62eec45b09.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 6：延遲的儀表板"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"如何優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本節討論如何調整流管道。Uber 已開發出一種流程管道性能調優框架，並提供端到端集成測試框架。在啓動實際調整之前，Uber 就已經開發了專門的集成測試，使我們能夠重構或優化流管道，並確信管道仍將產生正確的結果，類似於單元測試，保護我們免受迴歸。在整個優化過程中，這些集成測試變得非常有價值。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面，我們將介紹性能調優框架。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"性能調優框架"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從下面的內三角可以看出，我們的框架集中在三個領域。通過 Uber uMonitor 系統提供的度量標準對網絡、 CPU 和內存進行測量和監控。外五邊形的頂點表示可以探索主要優化領域。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/61\/612601e552442b7bf47dfd14f362393f.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 7：性能調優框架"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下表簡要解釋了每個領域的技術和潛在影響："}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/b1\/ce\/b187b849217bd355d31bed3cca6b56ce.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"表 2：性能調優的領域"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來，我們討論如何優化管道。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們對流管道進行了許多優化，一些優化技術對上述多個領域都有影響。其中一項特別的技術：自定義滑動窗口，對所有三個領域都有重大影響，所以我們有一個專門的章節來討論它，還有一個章節討論存儲。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"網絡優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主要的優化技術列於下表："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/93\/75\/93607726df44d1107c7d1695de7c3d75.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"表 3：網絡優化技術"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"正如上文所詳述的，關鍵的改進是既提供較少的信息，也提供較小的信息。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"內存優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下表列出了內存技術："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/48\/22\/4890598877564de84685e600d6e27822.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"表 4：內存優化技術"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注：還包括一些用於網絡優化的技術。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"CPU 優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"應用於 CPU 優化的技術如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/7f\/1f\/7fdeb97b0924f678dcf443cdf561491f.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"自定義滑動窗口"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"僅通過上面的調整，管道仍無法順利運行，因爲它需要數個滑動窗口 (2、4、8、16、32）進行聚合。由於需要按一個鍵劃分事件，窗口聚合的開銷如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從上游向窗口運算符傳遞消息時的 De\/Ser；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過網絡傳輸消息；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"反序列化時正在創建的對象；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"窗口管理所需的狀態管理和元數據，如窗口觸發器。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣的開銷會對垃圾收集器、CPU 和網絡造成巨大壓力。更有甚者，滑動窗口比翻滾或固定尺寸的窗口需要更多的狀態，因爲一個事件需要保存在一系列滑動窗口中。就拿一個 4 分鐘的滑動窗口來說：給定一個事件發生在 2021-01-01 T1:15:01 Z，此事件保存在下面的 4 分鐘窗口中："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2021-01-01T01:12:00Z ~ 2021-01-01T01:16:00Z"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2021-01-01T01:13:00Z ~ 2021-01-01T01:17:00Z"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2021-01-01T01:14:00Z ~ 2021-01-01T01:18:00Z"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2021-01-01T01:15:00Z ~ 2021-01-01T01:19:00Z"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲滑動窗口的扇出效應，給管道的狀態管理帶來很大的壓力。針對這些問題，我們採用 FlatMap 運算符手工實現了滑動窗口邏輯，其特點如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果允許重用對象，則向上遊運算符傳遞並重用事件，從而避免分區和相關開銷。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該狀態在內存中被管理，因此每個事件實際上只能複製一份數據。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於在內存中保存狀態所需的最大內存，我們估計如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"Total Memory = Count(Hexagon) Count(Product) Max(window size) * sizeof(event)\n\n = 3M 6 32 * 237b\n\n = 136G\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於 128 的並行性，每個容器的內存約爲 1G，這是可管理的。在生產中，實際的內存遠遠低於最大值，因爲不是所有的六邊形都有時間範圍的事件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個自定義滑動窗口的效率非常顯著，所以我們已經成功地將這個運算符重新用於超過 5 個不同的用例，這些用例需要在多個大型滑動窗口上進行聚合。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"優化後的最終作業 DAG"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ae\/aefd4882e31ef02a998d2e40b75d6044.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 8：需求管道的最終 DAG"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過對其進行優化，最終得到了一個更簡單的作業 DAG，其中自定義滑動窗口代替了較大的窗口運算符。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下面的 24 小時儀表板所示，管道始終可靠地運行："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"延遲監視器："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/62\/62599547300941c05d507f9f5c87ba33.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 9：優化後顯示延遲的儀表板"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"容器內存監視器："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ac\/ac4f1288c169fed001d5c3af055f312c.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 10：優化後顯示內存使用情況的儀表板"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"存儲"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲簡化管道維護和重新使用 sink，我們對管道 DAG 進行了進一步重構，在 Flink 中將 sink 運算符分離爲專門的發佈器作業，並將計算和發佈器作業與 Kafka 連接起來。此部分主要關注此發佈器作業的細節："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而在服務模型中，它會根據地理、時間和產品的要求查詢供應信息。我們選擇了 "},{"type":"link","attrs":{"href":"https:\/\/eng.uber.com\/schemaless-sql-database\/","title":"xxx","type":null},"content":[{"type":"text","text":"Docstore"}]},{"type":"text","text":" (Uber 的內部 KV 商店解決方案）作爲存儲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們從一個 docstore 集羣開始，它由許多用例共享。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面是每個 API 調用插入一行的結果。寫入的 QPS 在 13000 左右達到峯值，但大多數時候都是幾百的數量級。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/69\/6989229f13e3c7b9c118896316ba198e.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 11：如果每個 API 調用只有一行，那麼編寫 QPS 就不穩定"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"批處理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們嘗試對這些行進行批處理寫入，看看能否增加吞吐量。爲使批處理更高效，我們基於 Docstore 中的分片號來劃分數據。但是，應用批處理後，寫入的 QPS 較低。經過深入的研究，我們發現這是因爲流作業中所發出的一種度量的一個維度基數過大。我們將這一維改爲常數字符串，而非隨機的 UUID。寫入的 QPS 可以達到 16000 左右。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在寫到 Docstore 之前，我們先把數據寫到 Kafka 主題。在禁用 Kafka sink 後，我們可以看到寫入 QPS 增加了 10% 左右。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在我們把每個分片的批處理量改爲 50 後，寫的 QPS 增加了一倍，達到 34000。我們還嘗試了批處理規模爲 100 和 200。對於批處理大小爲 100，寫 QPS 增加到 37000（大約增加 20%）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將批處理大小改爲 200 後，沒有發現有太大的差別。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在下表中，我們列出了不同配置下的 QPS："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/3d\/e2\/3d9edff72a52d319b154f27f0cef1ae2.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"表 6：不同批處理大小下的吞吐量"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"並行性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Flink 作業的並行性是我們爲提高 QPS 而調整的另一個參數。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在將發佈器作業的並行性更新爲 256 後，寫入的 QPS 約爲 75000，增加了一倍多。批處理小爲 200，在並行度爲 1024 時，我們看到 QPS 達到 112000。但是，我們發現存在大量的超時錯誤。將批處理改爲 50 後，寫 QPS 約爲 120000。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/67\/d0\/6789f60dfdd33e681c4a9359149f5ed0.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"表 7：不同作業並行性下的吞吐量"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"線程池"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於每個 Flink 作業，我們也嘗試使用線程池來提高寫 QPS，結果如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/82\/f2\/822ba9e0e5d45817ba37f8a5889a48f2.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"表 8：不同線程池大小下的吞吐量"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果我們使用線程池大小爲 16，峯值 QPS 約爲 120000，但是這並不太穩定。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經過對共享集羣所能想到的所有優化之後，它仍然不能達到寫 QPS 的要求。爲了進行測試，我們要求一個特殊的集羣。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"分區調優"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"移除 Docstoresink，僅保留 FlatMap。沒有對分區器的調用，那麼 64 個容器就能處理超過 200000 的輸入消息率，而不會延遲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 FlatMap 之前，我們添加了自定義分區策略。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/77\/77f086a91b0b3152d588d17b286fcf85.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於 384 個容器，延遲時間大約是 12 分鐘。分區器的延遲範圍爲 0.2~5 毫秒。當增加到 512 個容器時，延遲降低到 3 分鐘。隨後，我們發現每個分區器調用的 0.2 毫秒成爲瓶頸。在 flatmap 中，我們添加了本地分區器調用緩存。20 分鐘後，緩存的點擊率類似於輸入信息率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是，延遲性仍在增加："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b2\/b2b2d009807fc4447387f998f7794672.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 12：作業延遲現象持續增加。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"背壓處於自定義分區階段。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d3\/d3033794fe3eeca1a62cf4c5858b5802.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 13：作業和背壓的拓撲處於自定義分區階段"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將並行性更新爲 128，有效地消除了管道中的任何延遲性。每個 DC 都可以寫入 300000 QPS，沒有任何問題。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"數據大小"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們嘗試了 3 種不同的模式來觀察數據大小的差異。第一種模式爲每個（環的大小，時間桶，供應\/需求）元組使用一個列。第二種模式爲需求和供應各使用一張地圖。第三種是將顆粒度爲 9 級的 7 個六邊形分組爲一行。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/9f\/9f46003e06e4c80dd024da51324ce9ac.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過 6 天的數據，我們得到的數據大小如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/31\/c1\/3126a46be2948c504ccff5092f6fecc1.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"表 9：不同數據模式下的壓縮"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在啓用壓縮之後，我們可以看到 3 個表可以節省大約 60% 的磁盤。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"服務"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在測試過程中，我們發現了一些延遲問題。P99 大約有 150 毫秒的延遲。在我們的定價工作流程中，這是不能接受的。經過調試，我們發現每個分區鍵都有許多行——大約 6000。這就是說，數據庫引擎需要掃描至少 6000 行，然後在查詢中應用傳遞的過濾。當分區鍵大小增加時，就會週期性地出現 200 毫秒的峯值。但我們知道 TTL 也是爲這個表設置的，因此我們所做的就是在 Query 中部署一個熱補丁，將結果限制在只有未過期的行上，然後應用查詢中傳遞的過濾。這樣降低了對底層引擎的掃描，而 P99 延遲降低到 10 毫秒。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/86\/8617f55626efaacc89d74d5f0acf20ce.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 14：優化之後，服務延遲從 150 毫秒下降到 10 毫秒"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"結論"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"考慮到計算邏輯的複雜性、寫吞吐量、服務 SLA 等因素，爲具有近實時特性的機器學習模型提供性能非常具有挑戰性。通過本文，我們介紹了我們所面臨的問題和解決方法，希望對類似用例的同行們有所幫助。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作者介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Feng Xu，Uber 高級軟件工程師，領導 Gairos、uMetric 的流計算框架。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Gang Zhao，Uber 前高級軟件工程師，負責 Gairos 優化、UMetric 消費，同時專注於存儲層 Elasticsearch、Apache Pinot、Apache Cassandra、Docstore 和查詢層優化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/eng.uber.com\/building-scalable-streaming-pipelines\/","title":"xxx","type":null},"content":[{"type":"text","text":"https:\/\/eng.uber.com\/building-scalable-streaming-pipelines\/"}]}]}]}
Uber如何爲近實時特性構建可伸縮流管道？
{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Uber 致力於爲全球客戶提供可靠的服務。要達到這個目標，我們很大程度上依靠機器學習來作出明智的決定，如預測和增益。所以，用來產生機器學習數據和特徵的實時流管道已經越來越受到重視。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Uber 公司使用了 Apache Flink 來建立實時流管道，並建立像"},{"type":"link","attrs":{"href":"https:\/\/eng.uber.com\/gairos-scalability\/","title":"xxx","type":null},"content":[{"type":"text","text":" Gairos"}]},{"type":"text","text":" 和"},{"type":"link","attrs":{"href":"https:\/\/eng.uber.com\/athenax\/","title":"xxx","type":null},"content":[{"type":"text","text":" AthenaX"}]},{"type":"text","text":" 這樣的平臺來簡化開發過程。但是，由於計算的複雜性或需要處理的實時數據量，仍有很多挑戰，如擴展性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文中，我們將以生產需求和供應特徵爲例，介紹我們所面臨的一些挑戰以及如何應對這些挑戰。尤其要說明的是，如何使用性能調整框架來優化實時管道。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖顯示了Apache Flink 中的流管道負責特徵計算和提取的架構。我們將在下文詳細討論這些管道。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/48\/486f6f86dfc251cefabb11e07ae00e4d.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 1：簡化的架構概述"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"特徵計算"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本節詳細介紹瞭如何通過地理空間和時間維度以及全局產品（UberX 等）對任何給定的六邊形（參見"},{"type":"link","attrs":{"href":"https:\/\/eng.uber.com\/h3\/","title":"xxx","type":null},"content":[{"type":"text","text":"此處"}]},{"type":"text","text":"）的原始事件，例如需求和供應事件進行聚合。以下是簡化的計算算法："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在一分鐘窗口內，按六邊形和全局產品類型計算出不同乘客和司機所發生的原始事件數量。在一分鐘窗口內，將 Kring Smooth 應用多個環，最多 20 個環（稍後進行討論）。將每一環的平滑值聚合在多個滑動窗口大小上，最長可達 32 分鐘。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總的來說，一條實時管道每分鐘爲一個六邊形生成 54 個特徵，使用 9 個環（0，1，2，3，4，5，10，15，20）和 6 個窗口大小（1，2，4，8，16，32）的組合。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來，我們討論算法的第二步。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Kring Smooth"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kring Smooth 過程通過向其 Kring 鄰居廣播一個六邊形的事件計數來計算地理空間聚合。換句話說，某一特定環的六邊形的特徵值考慮到了該環內所有六邊形的事件計數。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了計算給定六邊形 h 在環 r 上聚合的特徵值，公式爲："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"katexinline","attrs":{"mathString":" f(H, R)=\\frac{\\sum_{i=0}^{R} \\sum_{j=1}^{N u m(i)} f\\left(N_{i}^{j}, 0\\right)}{\\sum_{i=0}^{R} Num(i)} "}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中， "},{"type":"katexinline","attrs":{"mathString":" Num(i) "}},{"type":"text","text":" 是環 "},{"type":"katexinline","attrs":{"mathString":" i "}},{"type":"text","text":" 的六邊形數量；"},{"type":"katexinline","attrs":{"mathString":" N_{i}^{j} "}},{"type":"text","text":" 是環 "},{"type":"katexinline","attrs":{"mathString":" i "}},{"type":"text","text":" 的第 "},{"type":"katexinline","attrs":{"mathString":" j "}},{"type":"text","text":" 個六邊形； "},{"type":"katexinline","attrs":{"mathString":" f(H,0) "}},{"type":"text","text":" 是來自六邊形 "},{"type":"katexinline","attrs":{"mathString":" H "}},{"type":"text","text":" 的事件數。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，讓我們看看下面的例子，看看如何計算這三個特徵的值：六邊形 A 的第 0 環、第 1 環和第 2 環，公式如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"katexinline","attrs":{"mathString":" Num(0) = 1 "}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"katexinline","attrs":{"mathString":" Num(1) = 6 "}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "},{"type":"katexinline","attrs":{"mathString":" Num(2) = 12 "}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"katexinline","attrs":{"mathString":" f(A, 0) = 1 "}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"katexinline","attrs":{"mathString":" f(A, 1) = (f(A, 0) + f(B1, 0) + f(B2, 0)) \/ (Num(0) + Num(1)) = (1 + 2 + 1) \/ 7 = 4 \/ 7 "}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "},{"type":"katexinline","attrs":{"mathString":" f(A, 2) = (f(A, 0) + f(B1, 0) + f(B2, 0) + f(C1, 0) + f(C2, 0) + f(C3, 0)) \/ (Num(0) + Num(1) + Num(2)) = (1 + 2 + 1 + 3 + 2 + 1) \/ (1 + 6 + 12) = 10 \/ 19 "}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/1f\/1f5bdf3b2ad695c4010bef37f7899fd1.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 2：六邊形 A 的第 0 環、第 1 環、第 2 環"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該管道按照方程式計算出多個環形尺寸的特徵值，最多可達 20。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"時間聚合"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在一分鐘窗口的 Kring Smooth 完成後，算法的第 3 步是將平滑的事件計數在更大的窗口上聚合，最長可達 32 分鐘。要計算給定的六邊形 H 在更大窗口上的聚集，公式如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"katexinline","attrs":{"mathString":" q(H, T, W)=\\frac{\\sum_{i=0}^{W-1} q(H, T+i, 1)}{W} "}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中，"},{"type":"katexinline","attrs":{"mathString":" T "}},{"type":"text","text":" 是一個窗口的起始時間戳；"},{"type":"katexinline","attrs":{"mathString":" W "}},{"type":"text","text":" 是窗口的大小，以分鐘爲單位；"},{"type":"katexinline","attrs":{"mathString":" q(H,T,1) "}},{"type":"text","text":" 是來自 Kring Smooth 的平滑事件計數。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖 3 展示瞭如何計算 2 分鐘窗口的六邊形 A 的特徵值："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "},{"type":"katexinline","attrs":{"mathString":" W1 "}},{"type":"text","text":" 和 數學公式: "},{"type":"katexinline","attrs":{"mathString":" W2 "}},{"type":"text","text":" 窗口的 Kring Smooth 的平滑事件計數分別爲 1.0 和 3.0，分別在"},{"type":"katexinline","attrs":{"mathString":" T0+1 "}},{"type":"text","text":" 分鐘和"},{"type":"katexinline","attrs":{"mathString":" T0+2 "}},{"type":"text","text":" 分鐘發出；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2 分鐘窗口的特徵值爲 2.0，通過使用平滑的事件計數，按照上述方程計算，其時間範圍爲 ("},{"type":"katexinline","attrs":{"mathString":" T0 "}},{"type":"text","text":"， "},{"type":"katexinline","attrs":{"mathString":" T0 + 2 "}},{"type":"text","text":" 分鐘）。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/f2\/f27708203f3b35ed84498aeade95ca08.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 3：六邊形 A 的 2 分鐘窗口的聚合"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"流實現與優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本節以需求管道爲例，說明如何在 Apache Kafka 和 Apache Flink 中實現特徵計算算法，以及如何調整實時管道。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"邏輯作業拓撲"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖 4 說明了計算需求特徵的流管道的邏輯 DAG。對於所有尺寸大於 1 分鐘的窗口來說，它們是滑動窗口，這些窗口將以 1 分鐘爲單位滑動，這意味着一個輸入事件可能包含在 63 個窗口內：32 + 16 + 8 + 4 + 2 + 1。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/56\/56176c5769b26039c5ba30280a7c7fc6.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 4：需求管道的邏輯 DAG"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下表列出了邏輯 DAG 中主要運算符的功能："}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/af\/a3\/afda73d31d7c5d5fcd66e7b25e863ea3.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"表 1：需求管道的邏輯運算符"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"流管道的數據量"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本節列出了需求管道的數據量："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kafka 主題的平均輸入速率：120k\/s"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"六角形的計數：5M"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"城市的數量：1500"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每個城市的六邊形平均數和最大數：4000 和 76000"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1 分鐘內六邊形需求事件的平均計數：45"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"環 20 的六邊形計數：1261"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"顯然，該管道具有高容量、密集的計算和大的狀態需要管理。第一版實際上是按照邏輯 DAG 構建的，由於包括背壓和 OOM 等問題，無法穩定運行（如下圖儀表板所示）。由於我們的目標是接近實時的延遲（小於 5 分鐘）， 因此我們面臨的真正挑戰是如何建立穩定的工作通道。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"內存監視器："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/14\/141c73a9375372f65362d4bd61d383be.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 5：已用內存的儀表板"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"延遲監視器："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/24\/2411817a40e90424aa0e9b62eec45b09.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 6：延遲的儀表板"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"如何優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本節討論如何調整流管道。Uber 已開發出一種流程管道性能調優框架，並提供端到端集成測試框架。在啓動實際調整之前，Uber 就已經開發了專門的集成測試，使我們能夠重構或優化流管道，並確信管道仍將產生正確的結果，類似於單元測試，保護我們免受迴歸。在整個優化過程中，這些集成測試變得非常有價值。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面，我們將介紹性能調優框架。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"性能調優框架"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從下面的內三角可以看出，我們的框架集中在三個領域。通過 Uber uMonitor 系統提供的度量標準對網絡、 CPU 和內存進行測量和監控。外五邊形的頂點表示可以探索主要優化領域。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/61\/612601e552442b7bf47dfd14f362393f.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 7：性能調優框架"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下表簡要解釋了每個領域的技術和潛在影響："}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/b1\/ce\/b187b849217bd355d31bed3cca6b56ce.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"表 2：性能調優的領域"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來，我們討論如何優化管道。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們對流管道進行了許多優化，一些優化技術對上述多個領域都有影響。其中一項特別的技術：自定義滑動窗口，對所有三個領域都有重大影響，所以我們有一個專門的章節來討論它，還有一個章節討論存儲。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"網絡優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主要的優化技術列於下表："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/93\/75\/93607726df44d1107c7d1695de7c3d75.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"表 3：網絡優化技術"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"正如上文所詳述的，關鍵的改進是既提供較少的信息，也提供較小的信息。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"內存優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下表列出了內存技術："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/48\/22\/4890598877564de84685e600d6e27822.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"表 4：內存優化技術"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注：還包括一些用於網絡優化的技術。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"CPU 優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"應用於 CPU 優化的技術如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/7f\/1f\/7fdeb97b0924f678dcf443cdf561491f.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"自定義滑動窗口"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"僅通過上面的調整，管道仍無法順利運行，因爲它需要數個滑動窗口 (2、4、8、16、32）進行聚合。由於需要按一個鍵劃分事件，窗口聚合的開銷如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從上游向窗口運算符傳遞消息時的 De\/Ser；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過網絡傳輸消息；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"反序列化時正在創建的對象；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"窗口管理所需的狀態管理和元數據，如窗口觸發器。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣的開銷會對垃圾收集器、CPU 和網絡造成巨大壓力。更有甚者，滑動窗口比翻滾或固定尺寸的窗口需要更多的狀態，因爲一個事件需要保存在一系列滑動窗口中。就拿一個 4 分鐘的滑動窗口來說：給定一個事件發生在 2021-01-01 T1:15:01 Z，此事件保存在下面的 4 分鐘窗口中："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2021-01-01T01:12:00Z ~ 2021-01-01T01:16:00Z"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2021-01-01T01:13:00Z ~ 2021-01-01T01:17:00Z"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2021-01-01T01:14:00Z ~ 2021-01-01T01:18:00Z"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2021-01-01T01:15:00Z ~ 2021-01-01T01:19:00Z"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲滑動窗口的扇出效應，給管道的狀態管理帶來很大的壓力。針對這些問題，我們採用 FlatMap 運算符手工實現了滑動窗口邏輯，其特點如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果允許重用對象，則向上遊運算符傳遞並重用事件，從而避免分區和相關開銷。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該狀態在內存中被管理，因此每個事件實際上只能複製一份數據。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於在內存中保存狀態所需的最大內存，我們估計如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"Total Memory = Count(Hexagon) Count(Product) Max(window size) * sizeof(event)\n\n = 3M 6 32 * 237b\n\n = 136G\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於 128 的並行性，每個容器的內存約爲 1G，這是可管理的。在生產中，實際的內存遠遠低於最大值，因爲不是所有的六邊形都有時間範圍的事件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個自定義滑動窗口的效率非常顯著，所以我們已經成功地將這個運算符重新用於超過 5 個不同的用例，這些用例需要在多個大型滑動窗口上進行聚合。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"優化後的最終作業 DAG"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ae\/aefd4882e31ef02a998d2e40b75d6044.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 8：需求管道的最終 DAG"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過對其進行優化，最終得到了一個更簡單的作業 DAG，其中自定義滑動窗口代替了較大的窗口運算符。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下面的 24 小時儀表板所示，管道始終可靠地運行："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"延遲監視器："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/62\/62599547300941c05d507f9f5c87ba33.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 9：優化後顯示延遲的儀表板"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"容器內存監視器："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ac\/ac4f1288c169fed001d5c3af055f312c.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 10：優化後顯示內存使用情況的儀表板"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"存儲"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲簡化管道維護和重新使用 sink，我們對管道 DAG 進行了進一步重構，在 Flink 中將 sink 運算符分離爲專門的發佈器作業，並將計算和發佈器作業與 Kafka 連接起來。此部分主要關注此發佈器作業的細節："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而在服務模型中，它會根據地理、時間和產品的要求查詢供應信息。我們選擇了 "},{"type":"link","attrs":{"href":"https:\/\/eng.uber.com\/schemaless-sql-database\/","title":"xxx","type":null},"content":[{"type":"text","text":"Docstore"}]},{"type":"text","text":" (Uber 的內部 KV 商店解決方案）作爲存儲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們從一個 docstore 集羣開始，它由許多用例共享。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面是每個 API 調用插入一行的結果。寫入的 QPS 在 13000 左右達到峯值，但大多數時候都是幾百的數量級。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/69\/6989229f13e3c7b9c118896316ba198e.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 11：如果每個 API 調用只有一行，那麼編寫 QPS 就不穩定"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"批處理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們嘗試對這些行進行批處理寫入，看看能否增加吞吐量。爲使批處理更高效，我們基於 Docstore 中的分片號來劃分數據。但是，應用批處理後，寫入的 QPS 較低。經過深入的研究，我們發現這是因爲流作業中所發出的一種度量的一個維度基數過大。我們將這一維改爲常數字符串，而非隨機的 UUID。寫入的 QPS 可以達到 16000 左右。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在寫到 Docstore 之前，我們先把數據寫到 Kafka 主題。在禁用 Kafka sink 後，我們可以看到寫入 QPS 增加了 10% 左右。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在我們把每個分片的批處理量改爲 50 後，寫的 QPS 增加了一倍，達到 34000。我們還嘗試了批處理規模爲 100 和 200。對於批處理大小爲 100，寫 QPS 增加到 37000（大約增加 20%）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將批處理大小改爲 200 後，沒有發現有太大的差別。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在下表中，我們列出了不同配置下的 QPS："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/3d\/e2\/3d9edff72a52d319b154f27f0cef1ae2.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"表 6：不同批處理大小下的吞吐量"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"並行性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Flink 作業的並行性是我們爲提高 QPS 而調整的另一個參數。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在將發佈器作業的並行性更新爲 256 後，寫入的 QPS 約爲 75000，增加了一倍多。批處理小爲 200，在並行度爲 1024 時，我們看到 QPS 達到 112000。但是，我們發現存在大量的超時錯誤。將批處理改爲 50 後，寫 QPS 約爲 120000。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/67\/d0\/6789f60dfdd33e681c4a9359149f5ed0.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"表 7：不同作業並行性下的吞吐量"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"線程池"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於每個 Flink 作業，我們也嘗試使用線程池來提高寫 QPS，結果如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/82\/f2\/822ba9e0e5d45817ba37f8a5889a48f2.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"表 8：不同線程池大小下的吞吐量"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果我們使用線程池大小爲 16，峯值 QPS 約爲 120000，但是這並不太穩定。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經過對共享集羣所能想到的所有優化之後，它仍然不能達到寫 QPS 的要求。爲了進行測試，我們要求一個特殊的集羣。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"分區調優"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"移除 Docstoresink，僅保留 FlatMap。沒有對分區器的調用，那麼 64 個容器就能處理超過 200000 的輸入消息率，而不會延遲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 FlatMap 之前，我們添加了自定義分區策略。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/77\/77f086a91b0b3152d588d17b286fcf85.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於 384 個容器，延遲時間大約是 12 分鐘。分區器的延遲範圍爲 0.2~5 毫秒。當增加到 512 個容器時，延遲降低到 3 分鐘。隨後，我們發現每個分區器調用的 0.2 毫秒成爲瓶頸。在 flatmap 中，我們添加了本地分區器調用緩存。20 分鐘後，緩存的點擊率類似於輸入信息率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是，延遲性仍在增加："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b2\/b2b2d009807fc4447387f998f7794672.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 12：作業延遲現象持續增加。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"背壓處於自定義分區階段。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d3\/d3033794fe3eeca1a62cf4c5858b5802.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 13：作業和背壓的拓撲處於自定義分區階段"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將並行性更新爲 128，有效地消除了管道中的任何延遲性。每個 DC 都可以寫入 300000 QPS，沒有任何問題。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"數據大小"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們嘗試了 3 種不同的模式來觀察數據大小的差異。第一種模式爲每個（環的大小，時間桶，供應\/需求）元組使用一個列。第二種模式爲需求和供應各使用一張地圖。第三種是將顆粒度爲 9 級的 7 個六邊形分組爲一行。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/9f\/9f46003e06e4c80dd024da51324ce9ac.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過 6 天的數據，我們得到的數據大小如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/31\/c1\/3126a46be2948c504ccff5092f6fecc1.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"表 9：不同數據模式下的壓縮"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在啓用壓縮之後，我們可以看到 3 個表可以節省大約 60% 的磁盤。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"服務"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在測試過程中，我們發現了一些延遲問題。P99 大約有 150 毫秒的延遲。在我們的定價工作流程中，這是不能接受的。經過調試，我們發現每個分區鍵都有許多行——大約 6000。這就是說，數據庫引擎需要掃描至少 6000 行，然後在查詢中應用傳遞的過濾。當分區鍵大小增加時，就會週期性地出現 200 毫秒的峯值。但我們知道 TTL 也是爲這個表設置的，因此我們所做的就是在 Query 中部署一個熱補丁，將結果限制在只有未過期的行上，然後應用查詢中傳遞的過濾。這樣降低了對底層引擎的掃描，而 P99 延遲降低到 10 毫秒。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/86\/8617f55626efaacc89d74d5f0acf20ce.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖 14：優化之後，服務延遲從 150 毫秒下降到 10 毫秒"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"結論"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"考慮到計算邏輯的複雜性、寫吞吐量、服務 SLA 等因素，爲具有近實時特性的機器學習模型提供性能非常具有挑戰性。通過本文，我們介紹了我們所面臨的問題和解決方法，希望對類似用例的同行們有所幫助。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作者介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Feng Xu，Uber 高級軟件工程師，領導 Gairos、uMetric 的流計算框架。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Gang Zhao，Uber 前高級軟件工程師，負責 Gairos 優化、UMetric 消費，同時專注於存儲層 Elasticsearch、Apache Pinot、Apache Cassandra、Docstore 和查詢層優化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/eng.uber.com\/building-scalable-streaming-pipelines\/","title":"xxx","type":null},"content":[{"type":"text","text":"https:\/\/eng.uber.com\/building-scalable-streaming-pipelines\/"}]}]}]}
發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章