伴魚發佈系統實踐：集泳道、灰度、四端和多區域於一體的設計與權衡

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在伴魚發展早期，四端的系統發佈實踐比較簡單、直接：web 端直接登錄線上機器替換靜態文件；服務端所有項目共用一個倉庫，直接利用腳本部署不同倉庫子目錄到固定機器，並且在發佈的時候才嘗試解決代碼衝突；移動端從打包到發版流程基本靠手動保證。儘管比較粗糙，但是在公司早期人員少、產品迭代頻繁的大背景下，這些實踐方案的確起到很大的作用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着公司業務逐步穩定，技術團隊擴張，早期的發佈實踐方案逐漸暴露出問題："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"直接登錄線上機器替換文件是高危操作，運維團隊也開始逐步收回所有主機訪問權限"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務端共享一個倉庫，一個 go.mod，隨着人員增加，代碼衝突的概率也呈指數增加"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"各端都有公共基礎代碼，這些基礎庫的版本控制、灰度控制問題需要各自解決"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試環境需要支持不同泳道，生產環境中需要新增預發佈環境，方便測試人員迴歸、冒煙"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"生產環境除了全量發佈，還需支持灰度發佈，以減少新版本的影響範圍，保證服務質量"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"..."}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一個統一各端的發佈系統呼之欲出，這便是技術中臺各組通力協作，共同打造的"},{"type":"text","marks":[{"type":"strong"}],"text":"東風發布系統"},{"type":"text","text":"。目前東風發布系統已經逐步接管各端的日常發佈需求，成爲伴魚全體研發每天都需要訪問的系統之一。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"系統設計"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"基本流程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"東風發布系統負責將各項目的倉庫拉取到編譯機器上，編譯打包後發佈到相應的環境，基本流程如下圖所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/27/270c16e263713ef4f5b7c842a26a4be7.jpeg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於歷史原因，許多老項目都託管在 gerrit 上。出於統一管理和優化團隊合作開發體驗的目的，東風發布系統要求所有項目都託管在私有部署的 Gitlab 上，順便依託這次遷移梳理、解決以前代碼倉庫管理混亂的問題。由於發佈操作耗時較長，在發佈結束後，發佈系統會將相關消息同步給研發人員，方便他們及時瞭解發佈結果。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"邏輯分層"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"東風發布系統內部分爲三個邏輯抽象層，自底向上依次是資源層 (Resource Layer)、基礎層 (Base Layer)、服務層 (Service Layer)："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f2/f2bb243b7770858e661a5150da516c5a.jpeg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"資源層"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"資源層向上提供"},{"type":"text","marks":[{"type":"strong"}],"text":"存儲資源"},{"type":"text","text":"和*"},{"type":"text","marks":[{"type":"italic"}],"text":"計算資源"},{"type":"text","text":"*。伴魚在較早技術選型階段時就選中 TiDB 作爲核心持久化存儲，因此發佈系統內部的各種關係型數據存放在 TiDB 中；發佈系統的計算資源主要用於編譯項目，生成產物，由於需要支持四端項目，需要使用擁有不同編譯環境的機器。構建項目時，發佈系統利用 ansible 遠程驅動構建主機拉取倉庫、編譯項目、上傳產物，做到了 agentless。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"基礎層"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基礎層向上提供服務打包和部署的基礎模塊，包括環境 (Environment)、區域 (Region)、泳道 (Lane)、主機 (HostManager)、運行時參數 (RuntimeArg) 以及鑑權 (Authorization) 等。"},{"type":"text","marks":[{"type":"strong"}],"text":"一次發佈的最小單元由區域、環境、泳道以及被髮布服務共同確定"},{"type":"text","text":"，如：在日本區域的測試環境的 stable 泳道發佈 report 服務。在一些比較複雜的發佈任務中，不同環境、不同區域的發佈參數不同，因此需要運行時參數模塊對此進行集中管理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在發佈系統上的操作通常都有風險，因此發佈系統也遵從"},{"type":"text","marks":[{"type":"strong"}],"text":"最小權限原則"},{"type":"text","text":"，藉助內部統一鑑權服務，僅賦予需要的人操作權限。我們從兩個維度爲研發人員賦權，"},{"type":"text","marks":[{"type":"strong"}],"text":"端"},{"type":"text","text":"和*"},{"type":"text","marks":[{"type":"italic"}],"text":"角色"},{"type":"text","text":"*。其中端就是四端，用於端與端之間的權限隔離；角色分爲普通研發和管理員，如服務端的管理員、web 端的普通研發等，普通研發只需要關心服務層，對基礎層與資源層無感知，而管理員可以配置運行時參數等，需要了解基礎層。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"服務層"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務層直接對用戶提供"},{"type":"text","marks":[{"type":"strong"}],"text":"任務編排、流水線執行、觀察、取消、重試、通知，服務重啓、回滾"},{"type":"text","text":"等功能，滿足研發人員日常發佈需求。"}]},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","text":"任務編排與流水線執行"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"考慮到各端、各服務發佈的流程既有共同部分，也有不同部分，因此發佈系統需要提供靈活的流程編排功能。在發佈系統中，一個完整的發佈流程定義在一條發佈流水線 (Pipeline) 上，每次發佈執行就是流水線運行的過程。一條完整的流水線由多個階段 (Stage) 構成："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2a/2a1f8b1650cc2baec288cd71de22e1dd.jpeg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中開始階段 (Start Stage) 與結束階段 (End Stage) 是系統的固定階段，中間若干階段爲用戶自定義階段，不同發佈流程可以按需編排，如項目編譯階段通常屬於自定義階段，因爲各端的編譯流程大相庭徑。流水線中的每個階段又可以進一步劃分爲多個任務："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2a/2a482947317a9fd7767670372b9a7c40.jpeg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"流水線執行時，不同階段串行執行，相同階段內的任務並行"},{"type":"text","text":"。例如，開始階段包含兩個任務 clone"},{"type":"text","marks":[{"type":"italic"}],"text":"code 和 send"},{"type":"text","text":"msg，前者負責從 Gitlab 拉取項目代碼；後者負責發送即時消息到項目羣組中通知發佈啓動。因爲流水線中的最小執行單位是任務，因此我們稱定義流水線的過程爲"},{"type":"text","marks":[{"type":"strong"}],"text":"任務編排"},{"type":"text","text":"。"}]},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","text":"流水線觀察"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"流水線執行過程中會產生兩種日誌，任務的運行日誌及流水線的執行歷史，二者由日誌 (Log) 模塊負責管理。任務的運行日誌會實時地從打包機器、運行腳本採集，通過 websocket 暴露給用戶；流水線的執行歷史會被持久化，以便日後覆盤、審計時有據可查。"}]},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","text":"流水線取消、重試"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果開發者意外啓動某發佈流水線，他可以在階段級別上取消流水線，即：取消流水線時將停止當前階段之後的所有階段的任務運行。如果某階段因爲一些不穩定因素執行失敗，如網絡抖動，開發者可以在界面上重試該階段。由於我們完整保存了流水線的執行歷史，流水線在階段級別上的停止、重啓都能夠方便地實現，這也爲後續的灰度發佈奠定基礎。"}]},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","text":"重啓、回滾"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每次發佈流水線執行成功後會生成服務的一個新版本 (多區域發佈時會產生多個新版本)，版本模塊就負責管理各個服務的不同版本，研發人員可以在當前版本上重啓服務，也可以回滾到服務的歷史版本。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"核心功能"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"泳道發佈"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在服務迭代的過程中，常常有多個功能在同時開發，如果測試環境只有一套，就意味着要麼團隊內部商量好時間，大家輪流使用測試環境，要麼就先把不同功能的代碼合到某個分支再統一解衝突。無論使用哪種方式，都令人感覺十分別扭，於是我們做了泳道發佈功能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當用戶指定某泳道發佈服務時，發佈系統會爲該服務的實例打上相應的泳道標記，服務註冊和發現模塊就能知道同一服務的不同實例所屬的泳道。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5a/5a0a47425da0a008e191dff245203645.jpeg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上圖所示：當服務 A 要訪問服務 B 時，服務 A 會先從服務發現模塊中找到服務 B 上某實例的地址，由於服務 A 在請求中打上了 t2 泳道的標記，這時服務發現模塊就會從服務 B 在 t2 泳道上的實例中選擇一個返回，接着服務 A 就能直接訪問相應實例。如果請求中沒有打上任何泳道標記，服務發現模塊就會從服務 B 的默認 (default) 泳道上的實例中選擇一個返回，通過默認泳道可以實現單泳道測試環境到多泳道測試環境的平滑過渡，也符合研發人員的使用預期。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了更方便地使用泳道環境，我們的移動端和 web 端團隊都配合做了相應的改造。測試時，移動端和 web 端的測試人員能夠自行切換至指定泳道，切換完畢後，所有發送至服務端的請求都會帶上相應的泳道標記，這樣泳道測試就實現了端到端統一。"}]},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","text":"預演環境"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在特定場景下，測試團隊需要在線上環境才能測試產品功能，這時如果直接將服務發佈到線上環境再測就等於同時把未確認的功能暴露到用戶面前。爲此，發佈系統在線上環境提供了類似泳道的功能，即預演環境。預演環境中，所有服務、數據、中間件都與用戶使用的一樣，只不過由於線上流量不帶預演環境的泳道標籤，請求不會進入到預演環境，這樣測試團隊就能夠在用戶無感知的前提下完成測試。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"灰度發佈"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務功能的快速迭代和穩定運行是一堆矛盾體，我們既不希望步子邁得太快摔跟頭，也不希望過於謹慎錯過機會。發佈系統也需要在二者中尋求平衡，即便新功能可能存在問題，也要做到影響範圍可控，爲此東風發布系統引入了灰度發佈功能。服務端的研發人員在發佈時可以指定流量灰度比例，就能將新版本功能以指定的比例部署到生產環境中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於目前業務發展階段和微服務基礎設施，服務端暫不支持精準控制流量的能力。我們假設負載均衡是完美的，就可以通過控制不同版本服務的實例數量間接控制服務的灰度。目前我們在 k8s 中使用 StatefulSet 管理 Pod，保證實例的唯一標識不會因爲發佈而產生變化："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"灰度發佈時：容器編排服務會依據服務現有實例數，根據流量灰度比例，計算出灰度版本實例數，向上取整；然後重建一個 Statefulset，其中 Pod 鏡像即爲灰度版本，副本數即爲計算出灰度實例數；灰度實例啓動成功後，最後對主版本 StatefulSet 進行縮容，確保最終服務實例數不變。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"灰度回滾時：容器編排服務先將主版本 StatefulSet 中 Pod 鏡像回滾到指定版本，並且擴容到目前服務總實例數，成功後再將灰度版本 StatefulSet 刪除。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用戶在使用灰度發佈時，可以選擇自動或手動的方式將灰度版本進行全量部署："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"自動：用戶指定一段校驗時長後，發佈系統就會生成一個延時任務，到期後，系統就會自動將灰度版本全量部署到線上環境。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"手動：當灰度版本經過線上流量驗證，沒有問題後，用戶就可手動觸發全量部署。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"得益於靈活的流水線設計，實現自動和手動部署並非難事。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"多區域發佈"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了配合公司業務發展，今年我們也開始在海外逐漸部署核心服務。原本發佈系統只需要關心一個區域的發佈，現在則需要支持多區域發佈。考慮到在不同區域發佈服務的過程大體一致，只是具體到階段任務中的運行時參數有所差異，我們可以爲單條流水線的運行時新增一個參數，"},{"type":"text","marks":[{"type":"strong"}],"text":"部署區域"},{"type":"text","text":"，這個參數可能會影響到每個階段任務"},{"type":"text","marks":[{"type":"strong"}],"text":"在哪執行"},{"type":"text","text":"，"},{"type":"text","marks":[{"type":"strong"}],"text":"使用哪些運行時參數"},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經過在用戶體驗與實現複雜度二者間反覆權衡權衡，我們從使用者角度出發，在啓動發佈時能選定多個區域同時發佈；從實現複雜度角度出發，多區域發佈會生成多個相互獨立的流水線的運行時同時執行，這樣做的好處在於："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"減少使用者操作次數；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"避免漏發某個區域；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"壞處在於："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"單條流水線的多個運行時日誌分開展示，不方便使用者實時觀察；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果不同區域的發佈流程中，希望某些階段任務共用，如編譯得到產物，很難支持；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"多區域的回滾不能優雅地處理，需要分別操作；"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"思考與權衡"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"解耦"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"發佈系統是所有伴魚服務的造物者，因此發佈系統不能強依賴於需要它部署的服務，否則當它依賴的服務不可用時，發佈系統將無法發佈任何服務。但發佈系統自身又不能做得大而全，任何輪子都親自造一遍。爲了平衡這兩種對立的設計要求，我們梳理了發佈系統對其它服務的依賴特點。從依賴內容上分爲"},{"type":"text","marks":[{"type":"strong"}],"text":"數據性依賴"},{"type":"text","text":"和"},{"type":"text","marks":[{"type":"strong"}],"text":"邏輯性依賴"},{"type":"text","text":"，從依賴方式上分爲"},{"type":"text","marks":[{"type":"strong"}],"text":"強依賴"},{"type":"text","text":"和"},{"type":"text","marks":[{"type":"strong"}],"text":"弱依賴"},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當弱依賴不可用時，我們可以直接使用降級數據或降級邏輯；面對數據性強依賴，發佈系統的基本思路是在本地緩存一份數據，當相應服務不可用時則使用緩存數據，另外本地緩存的數據也會有一個硬編碼的兜底版本，以應對相應服務長時間不可用的情況；對於邏輯性強依賴，發佈系統只能自行實現，如判斷是否爲法定節假日的邏輯。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"資源節省"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於一些不合理實踐，有些項目的倉庫大小已經達到 GB 級別，這既增加了代碼拉取的耗時，又佔用了網絡帶寬資源。尤其是遇到海外部署的場景，從國內 Gitlab 拉取較大的倉庫十分昂貴，也可能造成專線阻塞，影響線上服務跨區域調用。採用增量拉取的方式，可以有效減少帶寬資源消耗，但需要依賴編譯環境本來的狀態，這樣編譯的過程就不再是純函數式的，不利於後期問題復現與排查。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經過權衡，發佈系統採用"},{"type":"text","marks":[{"type":"strong"}],"text":"增量拉取代碼，同時儘可能保存必要的運行時環境和產物"},{"type":"text","text":"的折衷方案。另外發布系統的流水線如果在某個階段執行失敗，使用者可以直接從失敗的階段繼續運行，而無需從頭開始執行，這種設計也間接避免了重複消耗計算、存儲和網絡資源。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"控制力"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在各端項目接入東風發布系統後，我們就能夠對項目的發佈流程擁有更強的控制力，如："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對各端的基礎庫進行管理，制定嚴格的上線流程。必要時甚至可以封禁問題版本、強行升級到某最低版本，達到修補漏洞、推廣功能等目的；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對異常時間點的發佈行爲進行監控和通知，避免緊急發佈與日常發佈之間的界限變模糊；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲後續統一支持代碼靜態檢查、迴歸測試等奠定基礎；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"強大的控制力也得益於靈活的流水線設計，使得我們能夠在系統固定階段和內置任務腳本中精準控制各端服務的發佈流程。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"未來工作"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"東風發布系統在推廣過程中，也發現一些不足："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"流水線只支持不同階段串行和同階段內部任務並行的運行方式，一些複雜的場景需要階段級別的分支邏輯；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前編譯任務都是在指定編譯機器的本地執行，當同時發佈的服務數量增多後，這種方式可能會拖慢服務和產品的持續交付速度，需要考慮引進編譯集羣來分攤計算消耗資源；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這也是未來東風發布系統的前進方向。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"參考"}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://www.infoq.cn/article/xCz*MzpguS5lH9816sxU","title":""},"content":[{"type":"text","text":"知乎部署系統演進"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://github.com/Tencent/bk-ci","title":""},"content":[{"type":"text","text":"Tencent/bk-ci"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://github.com/ctripcorp/tars","title":""},"content":[{"type":"text","text":"ctripcorp/tars"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://docs.gitlab.com/ee/ci/README.html","title":""},"content":[{"type":"text","text":"Gitlab CI/CD"}]}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://jenkins.io/doc/","title":""},"content":[{"type":"text","text":"Jenkins Pipeline"}]}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}