百度信息流和搜索業務中的彈性近線計算探索與應用 | 文末送福利

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"導讀：","attrs":{}},{"type":"text","text":"在生產環境的PaaS平臺中，爲了應對流量增長、負載尖峯、軟硬件升級、大規模局部故障等，通常需要留有一定的資源冗餘。百度信息流和搜索服務在全網提供超過5個9的可用性，爲了應對極端情況，各個地域的PaaS集羣在設計上都有一定資源冗餘。此外，爲了控制成本，業務迭代需要取得合理的投入產出比之後，才能正式上線推全。因此，我們結合推薦系統和搜索系統的業務特點，設計並實現了一套介於在線和離線之間的彈性近線計算架構。相比於在線計算，突破計算速度限制，爲業務計算複雜度提供了更大空間；相比於離線批量計算，提供了時效性更強，穩定性更高的策略計算方式。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#40A9FF","name":"blue"}}],"text":"一、背景","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"推薦系統、搜索系統的在線服務這種計算場景，整個負載隨流量波動，而流量存在明顯的波峯波谷。大部分用戶會在什麼時候看新聞，刷信息流，搜索等存在一定的規律，例如上班通勤的路上刷信息流、睡前刷信息流等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但在線服務的資源備貨必須要按照流量最高峯期的負載來準備，另外爲了容災，設計上需要有一定的部署冗餘，實際部署的資源會比高峯期所需要的資源更多，因此在線集羣中存在可以挖掘的冗餘資源，在實際的生產環境中，集羣的利用率一般不會超過60%。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/63/63b10e11ebc5f9fbaa4bd085dbc2e93c.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"△在線系統CPU利用率波動及其資源冗餘示意圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，在線計算系統一般有嚴格的速度指標限制。以推薦系統爲例，用戶一次下拉刷新，需要在指定的時間內返推薦回結果給用戶。受限於速度指標的限制，一次在線計算的計算規模也受到了限制，包括一次推薦計算覆蓋的候選集受限以及召回和排序等計算的模型複雜程度也受限。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在這樣的背景之下，我們開始嘗試","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"在百度的信息流推薦系統和搜索系統中引入近線計算。","attrs":{}},{"type":"text","text":"一方面，通過架構和業務聯合設計，將複雜計算遷移到近線系統，從而突破在線計算的速度限制，進行更大規模的策略計算，提升業務效果；另一方面，通過彈性的設計，它能夠充分利用在線集羣的閒散資源進行策略計算，降低整個近線計算機制的成本。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#40A9FF","name":"blue"}}],"text":"二、整體思路","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f5/f5052abccb84757cbade13efdb10bb78.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"△彈性近線計算整體的思路","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"簡要來說，","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"彈性近線計算的整體思路是採用異步計算的方式，把複雜策略計算與在線計算解耦。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先在線計算系統緩存中間結果或者獨立建設複雜的業務近線計算流程，再通過一定的觸發信號，觸發近線計算，提前計算中間結果。在線服務響應請求時，直接使用提前算好的計算結果做低複雜度的在線處理之後響應請求，這樣就繞開了在線計算的速度限制，可以進行復雜度高、計算量大的業務計算。另外就是在系統存在冗餘資源的時候，可以結合歷史的觸發信號進行到訪預估，主動發起近線計算，優化資源利用提升業務效果。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整個系統最重要的部分在於爲了適配閒散資源，節約整個近線計算機制的成本。我們設計了一整套動態算力和動態參數的機制。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先根據PaaS集羣自身的負載情況，進行動態的超發，獲取超額算力。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二是設計跟負載相關的業務參數，根據動態算力的情況，決定整個近線計算的業務參數，釋放出和算力相匹配的負載，從而平穩&充分的可利用資源。平穩，指的是缺資源的時候，系統能自動降級，避免系統雪崩無法自愈，充分指的是沒有自然負載的時候，我們結合業務需要，主動計算，充分利用算力，最終提升業務效果。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三是爲了優化資源的利用，對資源使用情況進行預估和規劃。一方面錯峯調度資源，提高擴展資源調度的成功率；另一方面結合資源情況，分配計算負載，提升資源的利用效率。最後，針對非高峯期，資源冗餘的情況下，需要估算整個系統的容量，再根據剩餘的算力，發起主動近線計算，充分利用資源，提升業務效果。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，爲了爭取更多的算力，系統中同時支持CPU/GPU和崑崙芯片等類型的資源，因此需要考慮在異構算力模塊上調度計算負載。另外，計算集羣分佈在全國各地，所以系統設計上，也需要考慮如何利用跨地域的算力。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#40A9FF","name":"blue"}}],"text":"三、系統架構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/43/432ca5418a9d9f3c46cca7ded4e57fd1.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"△","attrs":{}},{"type":"text","text":"彈性近線計算系統","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"整體","attrs":{}},{"type":"text","text":"架構簡圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"彈性近線計算系統主要包括幾個子系統。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"· 觸發控制系統","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主要負責根據業務參數，控制近線計算的觸發，達到削峯填谷的目的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"· 動態算力和動態調參系統","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"它相當於彈性近線計算系統的大腦，根據集羣的資源情況，分配近線計算算力；再根據算力情況，計算控制參數，從而控制跟算力匹配的負載。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"· 歷史數據中心","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"保存近線計算曆史的計算記錄。可以根據資源的情況，複用歷史計算結果，來調節對算力的使用。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"· 業務近線計算&控制系統","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個主要是和業務接入近線計算相關的一些架構機制設計，比如說輸入輸出緩存的讀寫，計算的拆包/幷包等等，業務計算與失敗信號的反饋等等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"· 業務在線接入","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"部分主要是業務接入近線計算系統上的一些設計。這塊主要考慮的是如何高效的接入業務，避免業務接入過高的架構人力成本。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#40A9FF","name":"blue"}}],"text":"四、算力來源","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"1）擴展資源池","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"彈性近線計算使用的冗餘資源（下文稱擴展資源）來源主要包括兩個方面。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/33/3340b5678a27ab97671dc45d40d47099.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"△可超發資源示意圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先，集羣分配了但是沒使用的資源，爲什麼呢？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一，流量存在峯谷，但資源一般按照高峯期的負載準備，因此非高峯期存在資源空閒。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二，由於Feed和搜索在線服務對外提供超過5個9的穩定性保證，要應對的機房故障除了硬件層面的故障之外，更多的也包括頻繁的業務迭代，複雜的部署環境等造成的故障。因此即使在高峯期，機房的負載也是存在設計上的冗餘。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"舉個例子，任何時刻當一個機房出現故障的時候，其它機房能夠承接的這個故障機房的流量進行止損。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其次是碎片資源，PaaS集羣的分配率是不可能到100%的，總是存在一些無法滿足業務quota需求的資源碎片資源，這些也是近線計算可以利用的資源。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，通過監控PaaS集羣資源的使用情況，基於歷史資源使用數據情況進行預估，形成一個擴展資源池，這個過程可以用如下公示表示：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/df/df7c6457cb711b87c1ad240d90446aa4.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/84/84031dd1cc9c62e2f36177c9213d0ce9.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e4/e4cab9638796137a721ca356f6737d62.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"綜上所述，在線大規模混布背景下的擴展資源特性：1）不穩定；2）供給量大。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不穩定主要來自於混布環境下，資源熱點可能隨時隨機出現，再加上常態的機房間切流，因此擴展資源實例不穩定。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，由於是混布的集羣，因此所有PaaS上的模塊的閒置資源都可以統一利用，整體擴展資源的供給量很大。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"2）XPU擴展資源挖掘和利用","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f6/f6e0f2bc3afae9b7028928b4360be4d3.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"△XPU資源調度系統架構簡圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於XPU（這裏是對GPU和崑崙芯片的統稱）資源與CPU資源的差異，雖然XPU本身已經支持共享，但XPU共享本質上還是分時複用的共享方式，所以如何保證共享XPU不影響在線業務的速度就是一個問題。爲了利用冗餘的XPU資源，在PaaS之上設計了一套XPU上的模型調度系統，核心子系統包括：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先，在節點層面，設計一個模型容器，在實例內部進行模型調度，它根據遠程拓撲管理系統中的元數據，決定實例具體執行什麼模型。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二，通過擴展PaaS的K8S Operator，在一個APP中，按照資源需求將實例劃分給不同的業務，比如說在線和近線，使得每個APP可以承載多個模型的計算。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三，資源預估系統會根據實際各個模型的資源需求情況，調整不同模型的資源。另外資源調整之後，涉及異構算力的負載分配和和削峯填谷的參數調整等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第四，所有對業務模型的訪問都通過名字服務尋址，獲取具體的模型到計算實例的映射。通過獨立的調度，隔離了近線計算的XPU共享對在線業務的干擾。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#40A9FF","name":"blue"}}],"text":"五、在線近線混合計算架構","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5e/5ec60c4479c690737a2d0ef86d1fa841.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"△近線在線混合計算架構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"考慮到業務同學已經對原有在線業務的研發架構比較熟悉，爲了降低業務接入的學習成本，我們設計了一種通用的在線服務和近線服務混合計算架構，通過微服務進程間的上下文數據交換機制，對業務同學屏蔽近線計算開發細節，讓業務可以用開發線邏輯的形式來開發近線邏輯，從而通過一套代碼，兩套部署，實現任意中間數據的在線和近線混合計算。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，業務計算模塊爲了適配擴展資源，需要做一些特殊的設計。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"· 服務分片","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果一個模塊需要加載太大量的數據，那麼需要將該模塊進行分片化處理，不同的分片計算不同的數據，限制模塊的quota粒度在100個邏輯核，15G內存以下。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"· 計算拆包","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果模型計算複雜，單次計算的粒度大影響成功率。通過計算拆包，即使部分計算結果失敗，不會導致全部的計算結果不可用。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":"center","level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#40A9FF","name":"blue"}}],"text":"六、計算觸發控制","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/94/9409dc3dedfaa32ecb966353bf25c9be.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"△計算觸發控制示意圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"觸發控制系統根據資源情況，以及不同觸發源的優先級，來觸發近線計算。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"系統在收到觸發信號之後，需要處理削峯和填谷兩個方面的工作。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一方面，在資源不足時，通過控制業務計算參數來對流量進行削峯。如果通過預先設計的計算參數的削峯不一定能控制負載以匹配更低的算力，尤其是搜索這種檢索詞分佈特別散的業務場景，在資源不足時，對觸發信號按需進行保存，延遲觸發。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另一方面，在資源一旦出現冗餘，就可以把主動觸發的信號提取出來觸發近線計算，包括：1）削峯機制保存的信號。2）系統預估的會到訪的流量；3）以及歷史全量的觸發信號索引，按照優先級進行主動觸發。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/da/dad5eaa29fd057fa7dff6fa3bbd85e80.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"△實際計算qps","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ee/ee5a3f4ad76b3dae57f4e235bd8f8a8d.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"△","attrs":{}},{"type":"text","text":"填谷信號qps","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此這裏還涉及到對整個近線計算系統的容量進行預估，預估完成之後，根據系統自然負載控制主動觸發的負載的水位，保持系統整體計算的負載不要超過系統的預估容量，達到平穩和均衡使用資源的目的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#40A9FF","name":"blue"}}],"text":"七、負載&算力智能調度","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於近線計算大量使用擴展資源，算力上難免存在波動。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原因主要有幾個方面，從大的層面說，由於局部系統故障，或者流量冗餘度壓測，業務經常在機房之間做流量遷移，就是說把A機房的流量，打到B機房，來驗證機房的冗餘度或者容災。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從小的層面來說，混布環境下，調度不均衡或者有意或者無意的資源超用，導致局部資源熱點，造成擴展資源退場。因此需要根據實時的資源情況，在機房之間調度負載，另外就是XPU與CPU資源由於計算能力不同，需要進行資源係數的學習和調度。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0c/0c8f6a6136cf09ee10b7caaf5a34d9fa.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"△算力調度與計算量調度系統示意圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從資源層面來說，一個近線計算任務往往涉及上下游多個模塊的交互。由於上下游各模塊都使用的動態擴展資源，就涉及到因爲局部資源熱點造成上下游模塊利用率不均衡的情況，因此需要不斷動態的在全鏈路的模塊之間做資源均衡。第二個是隨着負載波動，以及算力的波動，計算任務之間需要做資源均衡。第三個是由於資源調度的高峯往往和流量的高峯重疊，因此需要在流量高峯到來之前，預估近線計算所需要的資源，提前分配，錯峯調度。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"實踐下來，通過錯峯調度，預估資源需求並提前分配計算資源是比較有效的提升算力的辦法，可以理解是如果在資源已經緊張的時候，再進行近線計算模塊的調度，新的近線計算模塊的算力消耗有概率本身就造成局部熱點，導致擴展資源被回收，造成調度失敗。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/73/733135144db14dba914056d11409454d.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"△負載調度效果示意圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#40A9FF","name":"blue"}}],"text":"八、基於歷史計算結果優化資源利用","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在系統中我們會把歷史計算結果存下來，結合近線計算算力不穩定的特點，來優化資源利用。在系統計算資源充足時，用歷史上當前用戶偏好較高的文章取出來重複計算，擴大計算候選集，提升業務效果。在資源不足時，將歷史上認爲跟當前用戶不太相關的資源進行過濾，提高資源使用效率。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/38/38272d64c88878b263a794441be629f8.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"△近線計算曆史結果複用","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二個是到訪預估。系統會跟進歷史上用戶的到訪記錄，預估用戶的到訪時間。主要是提高用戶計算結果的時效性。因爲不斷的會有新內容會出現，策略模型也在不斷的更新，我們希望新的內容與新的模型儘快的出現在近線計算的結果中。另外，像Feed這種場景第一刷的質量非常重要，從我們的實驗結果也能看出來，通過主動計算，提升了DAU。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三個是對歷史觸發信號進行保存和索引。按照不同的優先級記錄歷史觸發信號，從而可以按照業務自定義的優先級進行刷庫，針對推送這種業務場景，就非常的有用，通過全庫刷新，及時的把新模型和新資源能夠推送到用戶。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#40A9FF","name":"blue"}}],"text":"九、典型應用案例","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以下介紹典型的應用場景。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"典型應用一：Feed在線&近線混合計算架構","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ca/ca5c9b8623d8dd9347d4b29edec0cc9d.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"△Feed混合計算-近線召回隊列示意圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"圖上是我們最早在Feed中的近線和在線混合召回計算落地的一種場景。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在推薦算法服務這一層，每個推薦算法對應召回的是數千或者數萬條的計算結果，由於資源和速度的限制，我們不能對這數千或者數萬條結果都進行粗排，並且粗排的模型也需要控制複雜程度。因此我們考慮將多個推薦算法服務的召回結果進行緩存，再放到近線系統來進行綜合打分之後生成一個用戶對應的近線計算候選集，作爲一路新的召回。之後在線請求來了之後，使用這路新的召回結果，進行輕量的在線計算之後就可以返回給用戶。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣就把召回層初步排序打分的計算規模提升了1個數量級。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏主要涉及到的調參場景包括：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）根據算力情況，決定一個用戶的計算頻率，舉例來說，就是資源充足時，每個用戶在一定時間內只算一次，資源不足時，逐漸提高計算間隔，降低計算頻率。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）預估用戶的到訪時間，根據系統容量的冗餘程度，提前更新用戶的近線計算結果。 ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"典型應用二：搜索結果排序","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8a/8a11f6c6553d5a39f5c81287e479d6d8.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"△搜索結果相關性近線計算示意圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外一個典型的應用是彈性近線計算在搜索產品的落地。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個系統中由於涉及基於Transformer的複雜模型，用到了XPU資源和CPU資源的混合計算。業務上跟推薦的排序類似，我們把搜索的相關性計算放在近線計算系統進行排序，計算檢索詞跟對應的搜索結果的相關性排序。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"跟推薦場景不同的是，搜索的場景下，檢索詞的全集更大，因此我們針對這種場景，用上了更激進的削峯填谷機制，如果沒有足夠的算力，直接將觸發信號緩存到消息隊列，在低谷期再把計算完成。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#40A9FF","name":"blue"}}],"text":"總結","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2018年落地以來，彈性近線計算已經支持了CPU、GPU、崑崙芯片等異構算力，在百度Feed推薦業務和搜索相關性業務落地。在Feed推薦業務落地超過10個業務場景，已經累計帶來大盤時長和分發帶來超過2位數的增長。在搜索業務落地3個業務場景，並支持複雜語義模型，在搜索業務落地，顯著提升搜索視頻搜索、問答搜索的相關性和用戶體驗。隨着更復雜的模型的落地以及更多目標的模型疊加，彈性近線計算系統將發揮更大的作用，低成本地助力業務增長。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d2/d245ddea709a515191a63125b34ddf03.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"△搜索產品用戶體驗評估","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本期作者 | 盧嘉龍，百度資深研發工程師","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"               呂輝明，百度高級研發工程師","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"                  聶勇，百度高級研發工程師","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#40A9FF","name":"blue"}},{"type":"strong","attrs":{}}],"text":"本期福利","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"本期百度架構師送出","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"“城市剪影·變色馬克杯”20份！","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/55/556db47da904b5f7069a464514957c27.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"注入90-100℃熱水","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"杯體自動呈現城市剪影圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"科技感與設計感結合","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"爲這個冬日留守在城市的你提供一份溫暖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#40A9FF","name":"blue"}},{"type":"strong","attrs":{}}],"text":"參與方式","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"公衆號後臺回覆","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"【抽獎】","attrs":{}},{"type":"text","text":"即可參與抽獎","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"還可邀請好友助力，大幅提升中獎機會","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"快來參與吧！","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接：","attrs":{}},{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/s/0xkeTfXFWWclet5qp3JjvQ","title":""},"content":[{"type":"text","text":"https://mp.weixin.qq.com/s/0xkeTfXFWWclet5qp3JjvQ","attrs":{}}]}]},{"type":"horizontalrule","attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"百度架構師","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"百度官方技術公衆號上線啦！","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"技術乾貨 · 行業資訊 · 線上沙龍 · 行業大會","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"招聘信息 · 內推信息 · 技術書籍 · 百度周邊","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"歡迎各位同學關注！","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2e/2ecfd10ad2477e2e4460efe0cba10be7.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}