雲原生架構下的持續交付實踐

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"導讀：","attrs":{}},{"type":"text","text":"隨着虛擬化技術的成熟和分佈式框架的普及，在容器技術、可持續交付、編排系統等開源社區的推動下，以及微服務等開發理念的帶動下，應用上雲已經是不可逆轉的趨勢。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"雲原生帶來了標準化、松耦合、易觀測、易擴展的特性，爲交付基建與業務解耦、更靈活的環境管理和無損發佈帶來新機遇。同時，微服務架構下服務數量爆炸式增長，對應的交付基建工作量暴增，且服務間拓撲復雜，又導致了升級影響難評估、問題定位困難、單獨測試環境成本極高等問題給高效能交付帶來了極大挑戰。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"全文6228字，預計閱讀時間17分鐘。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"愛番番產品從20年4月全面雲化，在雲化時代，通過與業務解耦的DevOps基建，讓業務團隊專注與業務開發本身，大幅度提升業務效能，通過智能生成契約測試 case保障服務間調用的可靠性，通過全鏈路灰度能力爲線下測試和無損發佈賦能，實現產品的高效能交付。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"一、業務背景","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"愛番番是典型的toB型業務，具有以下特點：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從產品形態上，產品戰線長，涵蓋(拓、聊、追、洞察)等核心產品能力；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從市場環境上，市場環境競爭異常激烈，對產研的效率與質量提出更高的要求；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從研發模式上，產品與研發採用敏捷思維研發，需要不斷的創新與試錯，快速完成PoC及MVP產品的研發上線；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從部署形態上，除了提供SaaS服務外，同時具有多樣化售賣的訴求；","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"團隊以業務領域劃分的多個scrumTeam，如下圖:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/bc/bc748b901af4208cc6e11ebb93ae1de8.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"二、效能體系面臨的挑戰","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.1 服務爆炸導致的基礎設施成本劇增","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"活躍模塊數200+，月均新增模塊8個，流水線、監控等基礎設施接入管理維護成本劇增。每個模塊需要接入的基礎設施如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/bf/bf9597d44a645bdb4a49649c36e8a02e.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.2 複雜拓撲導致的問題定位困難和迴歸範圍難以評估","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/eb/eb26c4e2f80e0a728451f0f51d01e51d.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務間拓撲復雜，如上圖，複雜拓撲帶來下列問題：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、升級影響難評估，迴歸漏測多；","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、線上問題定位困難；","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3、環境規模龐大，聯調測試成本高；","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.3 越來越高頻的發佈需求和隨拓撲復雜度提升的發佈成本的矛盾","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"模塊衆多且複雜拓撲，而且模塊間上線有依賴關係，每次上線100+模塊，人工控制流程，風險高而且效率越發低下。但是業務上發佈的需求愈發頻繁，在高頻次的發佈下，如何保障發佈過程的高效、安全也是一項極大的挑戰。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"三、整體的效能改進思路","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"流程機制層面： ","attrs":{}},{"type":"text","text":"用戶價值，流動效率提升爲核心的敏捷體系建設，包含以下幾個方面","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"敏捷迭代機制：","attrs":{}},{"type":"text","text":"以用戶價值流動效率爲核心理念，保障團隊目標一致，信息透明；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"需求拆分管理：","attrs":{}},{"type":"text","text":"標準化、可視化、自動化的管理機制，在成本可控的前提下達成小批量需求加速流動，快速驗證價值；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"分支模式和環境管理：","attrs":{}},{"type":"text","text":"基於雲原生強大的流量管控能力，實現基於istio的全鏈路灰度環境能力，實現簡潔、靈活、低風險的分支模式；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"全流程的數據度量體系：","attrs":{}},{"type":"text","text":"通過目標指標度量了解現狀，過程指標度量挖掘問題，問題自動創建任務，協同 peer推動問題閉環；","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"技術層面：","attrs":{}},{"type":"text","text":"全流程環節自動化智能化提升，包含以下幾個方面：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"基礎設施：","attrs":{}},{"type":"text","text":"建設與業務解耦的基礎設施服務；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"自動化：","attrs":{}},{"type":"text","text":"微服務下合理分層自動化體系，可控投入下保障有效質量召回；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"發佈能力：","attrs":{}},{"type":"text","text":"一鍵操作高效執行、過程可視、可感知、可控的極致發佈體驗；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"工具賦能：","attrs":{}},{"type":"text","text":"豐富的工具能力賦能研發測試各效能痛點環節，爲人員賦能（建設中，本文暫不詳細介紹）；","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面主要從技術層面的4個方向逐一進行方案說明：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f2/f2298654ccd9089646af6afc3fa7dedd.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"四、與業務解耦的Devops基礎設施服務","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上文已經提到，基礎設施面臨的最大問題是，由於爆炸的服務個數帶來的暴增的DEVOPS基礎設施接入和維護成本問題。在這個問題的處理上，我們借鑑了 serverless 的思路，把基礎設施服務化，與業務解耦，獨立運維。以前，我們的業務團隊研發和QA，除了需要進行業務的開發和測試工作之外，有大量的時間都花費在了新應用、日誌、配置的接入以及環境、流水線、監控維護等等和核心業務無關的事項上，就像下面這個圖的左邊，而且，任意基礎設施服務要升級，比如日誌平臺SDK升級、流水線需要統一增加一項安全檢測環節等等，都需要各個業務團隊配合升級，落地推廣困難。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果我們把這些基建內容通過服務化的形式提供給業務團隊使用，就能讓業務研發和QA聚焦於業務的關鍵事項，從而大幅度提升團隊效能。就像下面的右邊這個圖。同時基礎設施的升級業務無感知，再也不會有基礎設施能力落地推廣困難的問題。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d4/d4cea04bebce635841d8e02ca79bea87.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"如何打造與業務解耦，服務化的基礎設施？","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4.1 基礎設施標準化","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與業務解耦的第一步是基礎設施的標準化，只有標準化的過程纔有可能規模化，從而實現技術設施服務化。我們主要針對以下幾部分內容進行了標準化改造：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"1.模塊標準化：","attrs":{}},{"type":"text","text":"代碼結構、打包流程、標準容器、鏡像管理、部署過程","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"2.標準流水線","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"3.標準的基礎服務：","attrs":{}},{"type":"text","text":"APM組件、配置中心、發佈平臺、資源管理","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"4.研發模式：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/00/0041b38a4f510f3c93709f5c99fd0616.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4.2 聲明式基礎設施","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與業務解耦的第二步，是基於標準化的基礎上，建立聲明式的基礎設施能力。這裏的聲明式是指給業務團隊聲明式的基礎設施體驗。業務團隊只需要在標準配置中聲明一些基礎屬性，即可自動完成所有基礎設施的接入，且後續維護上業務0成本。主要分爲兩個環節的建設：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"接入時：分鐘級的一鍵接入","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們的做法是通過腳手架爲抓手來構建基礎設施的一鍵接入能力。如下圖所示：腳手架是我們這邊新模塊創建的入口。所有新代碼庫都是通過腳手架創建，他會幫助開發自動生成一整套集成了標準組件的代碼框架。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在腳手架創建新模塊的時候，根據業務聲明的模塊屬性，如是否接入apm、模塊代碼類型、模塊服務類型等等自動完流水線創建、基礎組件接入、集羣環境申請、配置文件生成等操作。一個新的服務，從創建代碼庫到服務全套基礎設施接入完成，服務可直接部署到測試集羣的時間<10分鐘。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/40/404ea068f2825e9151a00ff81c9d7035.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"腳手架：自動生成框架代碼，包含基礎apm組件、api管理平臺等的接入；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"configMap:自動生成應用標準配置並基於配置新增/變更主動觸發接入服務；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接入服務：拉取configMap配置並解析，根據配置內容調度不同的基礎設施服務完成接入初始化；","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"運行時：根據服務聲明內容動態運行，實現業務升級維護0成本","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基礎組件部分，因爲都是以sidecar模式提供服務，所以運行時天然與業務解耦，因此重點在於如何實現流水線在運行時與業務解耦。我們針對流水線進行了模板化、參數化改造，並和業務的聲明屬性結合。就像下面這張圖，流水線每次都是動態運行的，運行的內容是依賴左側5部分聲明數據實時生成，包括cicd通用配置、流水線模板、任務腳本、任務策略、業務聲明屬性。除了業務自己的聲明文件，其餘部分都是基礎設施組獨立運維，故對應任務優化、添加、統一配置修改等均對業務透明。就像右圖，如果要針對流水線上的某個環節進行優化，或者增加一些環節，僅需基礎設施組修改流水線模板或者任務腳本即可。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/1a/1af530c2de22b0722a5ae7c9724100b0.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4.3 智能化基礎設施","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲服務化之後，基礎設施作爲獨立運維的服務，所有的問題都需要設施團隊獨立維護排查，所以與業務解耦的第三步就是建立高穩定高效低運維成本的基礎設施能力。我們的思路是通過智能化的策略，來保障高效和穩定。在流水線運行的前中後通過策略給流水線增加一個”監工”，模擬人工判斷任務是否應該執行，模擬人工分析跟進、修復問題等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分析常見的流水線穩定和效率問題比如環境不穩定、底層資源不穩定、網絡異常等等，大體可分爲 偶發問題重試可恢復、問題相對複雜需人工排查、阻塞問題需人工修復三類。而效率方面大量重複、無效任務比如只加了個log也要跑全套測試流程，導致了資源浪費，也導致了執行效率低下。如下圖左側所示：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/6b/6b2d94bcc102f9df8b5015449b59f673.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對這些場景，我們在流水線運行前後都添加了可配置的策略判斷過程，判斷任務是否需要跳過、排隊、重試等等，從而提升穩定性和效率。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"典型場景：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"自動紅燈分析：任務失敗後可自動根據日誌錯誤碼分析問題原因並給出標註，方面後續根據統計數據更有效的優化。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d3/d34e34d2716860520bbd5341fe7d96a7.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"排隊策略：","attrs":{}},{"type":"text","text":"在自動化等任務執行之前，自動檢測依賴環境是否正常，從而降低運行失敗導致的紅燈。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8b/8b27b4e5a05a60756d185680477b5870.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"五、分層自動化體系","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"要想實現持續的交付，自動化是一個繞不開的話題，在雲原生微服務的背景下，自動化層級會發生怎樣的變化呢？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"和傳統3層金字塔自動化不一樣，雲原生架構下的自動化，由於服務內部相對簡單，而服務拓撲復雜，所以測試的重點是在系統端到端測試，實際的分層測試的比重更像一個倒過來的金字塔。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/58/580996171104dc7b1adf301a132767bf.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而由於端到端成本過高,考慮到投入產出比，愛番番的分層自動化是按照右下角這個結構來建設的，其中接口DIFF測試、契約測試、純前端DIFF測試是無人工介入，最核心的三個部分。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/26/260539f702cca3250474d534abe6c1b3.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"5.1 基於全鏈路灰度環境的接口DIFF自動化","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5.1.1 全鏈路灰度方案","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們接口的DIFF 測試是基於強大的全鏈路灰度環境能力來建設的，這是雲原生架構給我們帶來的紅利。先介紹下我們的全鏈路灰度方案。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們是基於istio的靈活的路由能力，通過同構底層「分組多維路由」的架構設計， 自研CRD Operator 構建愛番番的「全鏈路灰度發佈」平臺。該方案支持了我們的線下多路複用環境、線上安全的容量評估以及金絲雀發佈等多個場景。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c8/c8f297182014219c6c935abcdd188f18.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5.1.2 測試環境多路複用","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試環境多路複用是指，使用有限的資源，在一套基礎環境上邏輯隔離出多套環境，支持並行開發、聯調的需求。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下圖所示，不同的分支對應着不同的feature，通過流量染色+流量規則路由的方式，使得不同分支擁有邏輯上隔離的環境，支持並行開發。在前端給流量打上橘色標記之後，全鏈路的請求會走橘色的鏈路進行訪問。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/41/4102e2384758a713936ab9430a07b330.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5.1.3 基於多路複用的DIFF測試","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有了如上所述的多套邏輯隔離的測試環境之後，每當有新的分支環境拉出並有代碼更新時，即可通過將流量在base環境（部署最後一次上線的代碼）和新分支環境進行回放，並對比兩者的返回是否存在差異來進行迴歸測試。我們的diff方案如下：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/bc/bce7ae9880f549ac55035034a26507e0.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"該方案具備如下幾個優點：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於流量回放的接口diff，最大限度的覆蓋線上用戶真實場景；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"全流程自動化，無人工參與；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置化的流量篩選策略和diff策略接入，便於擴展優化；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式任務運行，支持大批量併發；","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"5.2 保障召回服務間調用問題的契約測試","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5.2.1 什麼是契約測試","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"微服務的架構，服務之間依賴複雜，而且通常每個服務都是獨立的團隊維護，服務和服務之間，前後端之間大多通過API調用。那麼這種情況下可能就會出現如下場景：A 團隊開發的 API 同時服務於 B\\C 團隊。最開始測試的時候都是通過的。但是後續迭代中，B 團隊對字段 A 有一些調整的需求，A 團隊根據需求改了，也測試通過了，但是上線後發現 C 團隊功能異常了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"以上問題的本質原因爲：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務提供方服務的消費者越來越多的情況下，服務的變更影響難以評估，服務的變更也不能及時同步到所有消費者，所以往往是消費方發現問題了反饋，導致損失。爲了避免上述問題，我們引入了契約測試。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"契約測試的核心思路是通過消費者驅動的方式，建立服務端和各個消費端之前的契約，在服務端有修改之後，通過測試和所有消費方之前的契約是否被毀壞來保障服務升級的安全性。同時，契約也可以作爲雙方測試解耦的手段。通過契約測試，團隊能以一種離線的方式(不需要消費者、提供者同時在線)，通過契約作爲中間的標準，驗證提供者提供的內容是否滿足消費者的期望","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5.2.2 常見的契約測試方案","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"常見的契約測試方案有真正實踐消費者驅動的如pact，契約由消費端生成並維護，提供方代碼更新之後，拉取所有消費方契約進行測試，即解決了集成測試解耦問題，又保障了服務方能滿足所有消費方需求。（下左圖）","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"也有非消費者驅動，提供方生產契約，並提供mock服務，消費方可以基於契約文件測試，如Spring Cloud Contract。只能解決集成測試解耦的問題（下右圖）","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/1d/1d9350619dcc2c21efb42ea6f8d693ed.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5.2.3 愛番番的契約測試方案","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"愛番番的方案則是取了折中。一方面由於團隊習慣，契約一直是服務提供方給出，另一方面又希望保留消費者驅動特性，從而保障服務方能滿足所有消費方需求。我們選擇了在提供方生成契約，但是通過線上日誌和調用鏈解析的方式來補充模擬消費端契約case。且整個過程全自動化。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/19/19629cf7ff4f41b189d9eb1a8b4fb299.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5.2.4 契約測試技術實現","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一步：引入swagger推動全接口接入，保障接口管理平臺的接口文檔信息與實際代碼達到準實時同步，詳細的實現步驟如下；","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a4/a44a595f291433b1286dcb0a152eb653.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二步： 根據接口文檔自動生成契約case","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有了和代碼同步的接口信息之後，則根據接口文檔信息自動生成基礎的契約測試case。在每次接口信息上傳平臺的時候，會檢測本次上傳內容，根據內容自動觸發新case的生成和老case的驗證。驗證會運行修改了的接口之前關聯的契約測試case來檢測本次接口更新是否破壞原有契約，運行結果通過報表記錄，並推送到對應團隊標註，根據標註結果判斷是否更新case。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/fd/fdb9a6eff00309a1fca4c0b8266eae5d.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三步： 依賴調用鏈、日誌信息智能分析消費端特徵，生成模擬消費端的case","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下圖，通過調用鏈信息，提取出各個服務的消費方有哪些，通過各消費方的日誌分析，獲取模擬各消費方契約，並自動生成case和接口進行關聯；","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f1/f1bc1c2806624488041be5f324a72c0f.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"5.3 問題智能定位降低自動化維護成本","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"自動化雖然是效能提升的好手段，但是長期以來，自動化的穩定性問題、問題跟進排查成本的居高不下都是阻止大家開展自動化建設或者自動化建設半途而廢的重要原因。針對自動化的穩定性提升和跟進成本降低，我們建設了case失敗自動定位和修復能力，讓智能化的小助手幫助大家輕輕鬆鬆維護case運行。下面是我們自動定位的一個效果實例：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2e/2edfa9fc809daf35509781c899335316.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們會在自動化case運行失敗後，調用自動定位服務，自動對失敗的case進行標註，根據標註結果會對失敗case進行分類處理。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"比如，環境問題會自動重試，批量未知會發送到自動化小組進行排查，元素找不到會發送到業務QA排查。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以下是實現的方案。包含基礎定位能力和基礎數據獲取。在這些基礎能力之上，建設了配置層，實現配置解析和調度能力，讓我們可以通過配置的方式，靈活組合不同的定位策略快速支持不同場景的問題定位。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0c/0c54a4f4fb140324f6f3c7e2358a678b.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"六、高效安全的持續發佈","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"6.1 發佈困境","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不同類型模塊對接了不同的不發平臺和流程，統一發布困難，底層發佈方式的變更需要各模塊升級，遷移成本高","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於模塊衆多且複雜拓撲，而且模塊間上線有依賴關係，每次上線100+模塊，人工控制流程，風險高而且效率低。上線過程的的記錄和分析人耗也很高。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整體上線過程不可見，風險感知滯後","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"如何解決以上問題？","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"6.2 多平臺部署引擎","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於雲原生構建多平臺統一的部署與發佈引擎，無縫集成CICD，實現發佈過程的高度標準化，同時支持多種發佈策略。如下圖：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過CD發佈平臺的統一，實現各類型模塊統一發布，且底層部署遷移業務無感知。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/36/36ecb00e17024761fc3bdc22597a0faa.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"6.3 發佈劇本設計","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有了統一的發佈平臺之後，爲了解決上線過程複雜低效的問題，我們希望實現完全自動化的發佈過程。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分析發佈前後需要進行的事項，如下做圖所示。基於這些事項，梳理了要自動完成整個發佈過程需要收集的數據，如右圖所示，包含發佈模塊封板信息、依賴信息、配置信息等等。基於這些數據，根據固定的編排邏輯，自動生成服務發佈拓撲以及本次上線步驟。生成的上線拓撲和步驟信息經人工確認之後，自動調用對應上線發佈服務進行發佈，並針對發佈過程數據自動統計，生成發佈過程總結。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/38/3895048e4e4df5e233016cc32bba2e6a.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"6.4 過程可視、可感知、可控的一鍵發佈","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有了自動化分發布過程之後，爲了能夠及時感知發佈過程中的問題，降低發佈風險，進行了發佈過程可視化建設，並與APM、金絲雀發佈等策略結合，保障發佈的安全。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"發佈過程可視：","attrs":{}},{"type":"text","text":"服務粒度的依賴拓撲已經實時上線進度展現、過程可視可感知；","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/df/df36117cd23995e57d8390fe1c1cfc7e.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"金絲雀發佈策略：","attrs":{}},{"type":"text","text":"發佈無損、風險及時感知並召回","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a9/a9285b12095b80ab19bbd5424d75b4ff.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"七、整體收益","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"迭代 story 量增長85.8%，發佈週期穩定，研發測試周期下降30%，千行 bug 率從1.5降低到0.5。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"八、未來展望","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、通過IDE本地插件工具，賦能開發編碼測試過程，提升研發環節效能；","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、通過白盒能力，構建質量風險識別體系，應用於准入、準出、灰度等場景；","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"推薦閱讀：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http://mp.weixin.qq.com/s?__biz=Mzg5MjU0NTI5OQ==&mid=2247498936&idx=1&sn=777b75ff2b3f85a66240762dcbba1aca&chksm=c03eccc4f74945d299c32433e10055d50821dd74874ba66722a6c9b15b914316298300f1020b&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"｜百度短視頻推薦系統的目標設計","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http://mp.weixin.qq.com/s?__biz=Mzg5MjU0NTI5OQ==&mid=2247497498&idx=1&sn=76aec4723a8ace1c62f84fa69ebd5865&chksm=c03ec766f7494e7018d15106466f3476ce992cdf87de7c063627762598c59b77337a5d3f6e48&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"｜","attrs":{}}]},{"type":"link","attrs":{"href":"http://mp.weixin.qq.com/s?__biz=Mzg5MjU0NTI5OQ==&mid=2247498828&idx=1&sn=70422ee59299dbe640bb9d192579fb43&chksm=c03ecc30f74945262a80f2f04873ea759d56b780a9bff6fcf9ff22880bb8929481fc85589681&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"百度信譽認證中臺架構解析","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http://mp.weixin.qq.com/s?__biz=Mzg5MjU0NTI5OQ==&mid=2247498745&idx=1&sn=88ab93caf2e8a3662b307206954bbcdb&chksm=c03ecb85f749429346766d992c69ffbe58037d33ae55642693efe497a5da722b5ae484b38d11&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"｜","attrs":{}}]},{"type":"link","attrs":{"href":"http://mp.weixin.qq.com/s?__biz=Mzg5MjU0NTI5OQ==&mid=2247498782&idx=1&sn=20b01f2eccf6ccd827fba5a97f1f333c&chksm=c03ecc62f74945747ad8e8ef207f7a1a140bafe66c93119e5e11889d14f3edfe601b8f8afe0c&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"圖數據庫在百度漢語中的應用","attrs":{}}],"marks":[{"type":"underline"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http://mp.weixin.qq.com/s?__biz=Mzg5MjU0NTI5OQ==&mid=2247498745&idx=1&sn=88ab93caf2e8a3662b307206954bbcdb&chksm=c03ecb85f749429346766d992c69ffbe58037d33ae55642693efe497a5da722b5ae484b38d11&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"｜","attrs":{}}]},{"type":"link","attrs":{"href":"http://mp.weixin.qq.com/s?__biz=Mzg5MjU0NTI5OQ==&mid=2247499032&idx=1&sn=4ae30ea9c39f5d91f174aa4145daeae1&chksm=c03ecd64f7494472abe0790107bf9984f836ed590e6d2c124c09a0121807ad3a74f955ee9696&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"一年數十萬次實驗背後的架構與數據科學","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"---------- END ----------","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"百度Geek說","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"百度官方技術公衆號上線啦！","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"技術乾貨 · 行業資訊 · 線上沙龍 · 行業大會","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"招聘信息 · 內推信息 · 技術書籍 · 百度周邊","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"歡迎各位同學關注","attrs":{}}]}]}