集羣管理系統 Mesos 的設計原理 · NSDI '11

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文要介紹的是 2011 年 NSDI 期刊中的論文 —— "},{"type":"link","attrs":{"href":"https:\/\/www.usenix.org\/legacy\/events\/nsdi11\/tech\/full_papers\/Hindman.pdf","title":null,"type":null},"content":[{"type":"text","text":"Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center"}]},{"type":"sup","content":[{"type":"text","text":"1"}]},{"type":"text","text":"，該論文實現的 Mesos 能夠在集羣中管理不同的計算框架，例如 Hadoop 和 MPI 等。雖然 Mesos 集羣管理系統是 10 多年前發佈的技術，今天已經逐漸被更主流、更通用的容器編排系統 Kubernetes 取代，但是它確實可以解決集羣管理上的部分問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Apache Mesos 和 Kubernetes 都是優秀的開源框架，也都支持大規模的集羣管理，但是它們兩個管理的集羣規模仍然差一個數量級，單個 Mesos 集羣可以管理 50,000 節點，而 Kubernetes 集羣卻只能管理 5,000 節點，需要做很多優化和限制，才能達到相同的數量級。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/22\/2215b3c6f2551a6d51392c521a07e920.png","alt":"apache-mesos-and-kubernetes","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"圖 1 - Kubernetes 和 Mesos 集羣"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"雖然 Kubernetes 是今天集羣管理的主流技術，但是 Mesos 在剛剛出現時也是很先進的集羣管理系統，它想要取代的是當時更爲常見的靜態分片集羣。靜態分片集羣雖然可以同時運行屬於不同框架的工作負載（例如：Hadoop、MPI），但是因爲框架的異構性，使用靜態分片技術會將集羣中的機器預先分配給不同的框架，再由這些框架分配和管理資源。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/bc\/bc6ce752cb434cd8bb35fabd7df977bc.png","alt":"static-partitioning","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"圖 2 - 靜態分片"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Mesos 在最初設計時並不會直接管理和調度開發者提交的工作負載，而是提供一組接口暴露集羣的資源，並通過這組輕量級的接口同時對接 Hadoop、MPI 等框架。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下圖所示的 Mesos 集羣同時運行了 Hadoop 和 Mesos 兩個框架，如果忽略圖中與 Hadoop、MPI 框架的相關模塊，我們會發現架構會變得非常簡單，它僅由 Zookeeper 集羣、Mesos 主節點和工作節點組成。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/9d\/9d5e61704be60846cf799982a2c03002.png","alt":"mesos-architecture-diagram","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"圖 3 - Mesos 架構圖"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Zookeeper 集羣提供了高可用的數據存儲和選舉等功能；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Mesos 主節點收集工作節點上報的數據並向框架的調度器提供資源；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Mesos 工作節點上報數據並通過框架的執行者在本地啓動任務；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每個 Mesos 集羣中運行的框架都由調度器和執行者兩部分組成，調度器會處理主節點提供的資源，與 Kubernetes 的調度器有着相同的作用，當調度器接受主節點提供的資源後，它會返回待運行任務的相關信息；而執行者會在工作節點上運行框架創建的任務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Mesos 爲了保證更好的可擴展性，它定義了一套能夠滿足資源共享的最小接口，將任務調度和執行的控制權都通過如下所示的接口交給框架，其本身僅保留較粗粒度的調度和資源管理功能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7c\/7cea8757b5df64791b3b80f6d5813fc4.png","alt":"mesos-api","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"圖 4 - Mesos 接口"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲 Mesos 中的任務調度是分佈式的過程，所以爲了保證該過程的效率和可靠性，它引入了下面的這三種機制："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"節點過濾器：框架使用過濾器剔除集羣中不滿足自身調度條件的節點；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"資源主動分配：爲了提高框架的調度速度，會將預先提供給框架的資源計入框架的總分配資源，直到框架完成調度，這能激勵框架實現更快的調度器；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"資源撤回：如果框架在一段時間內沒有處理主節點提供的資源，Mesos 會撤回資源並提供給其他框架；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了提供良好的擴展性和性能之外，作爲集羣調度管理系統，Mesos 也面臨着隔離不同任務資源的問題。在 Mesos 剛剛發佈時，容器技術還沒有像今天這麼普及，但是它也利用操作系統的容器隔離不同工作負載的影響"},{"type":"sup","content":[{"type":"text","text":"2"}]},{"type":"text","text":"，並利用可插拔式的隔離模塊支持多種隔離機制。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"調度模型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們在文章開篇就已經介紹過 Mesos 和 Kubernetes 能夠管理的集羣規模有數量級的差距，這裏簡單對比分析下兩者在調度器上的差異，這能幫助我們理解 Kubernetes 調度器在設計時做出的決策，以及這些決策是如何影響它的可擴展性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"需要注意的是，提升系統可擴展性往往都是複雜的問題，而在 Kubernetes 這樣龐大的系統中會顯得更加複雜，Kubernetes 的調度器不是影響其可擴展性的唯一因素，想要提升單個集羣的規模要從多個方面入手。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Mesos 的調度器選擇了兩層的調度設計，其中頂層調度器僅會根據底層框架調度器的需求"},{"type":"text","marks":[{"type":"strong"}],"text":"粗粒度地過濾集羣中的節點"},{"type":"text","text":"，而框架調度器會"},{"type":"text","marks":[{"type":"strong"}],"text":"執行真正的任務調度"},{"type":"text","text":"，將任務綁定到相應的節點上。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ee\/ee0f2a72184843a3d580be23c0c9b28f.png","alt":"two-level-scheduler","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"圖 5 - 兩層調度器"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這種兩層的調度器設計看起來雖然很複雜，但是實際上它能夠降低 Mesos 調度器的複雜度並提高了它的可擴展性："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"降低複雜度：頂層調度器不需要處理真正的調度過程，它僅通過資源提供（Resource Offer）機制將一組節點交給底層調度器控制；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"提高可擴展性：兩層調度器設計可以更方便地接入新的框架調度器，兼容不同複雜的調度策略，不同框架調度器內部可以串行爲任務選擇節點，提高整體調度的吞吐量；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"雖然 Mesos 通過兩層調度器設計提供了很強的擴展性，但是它卻不能爲調度決策提供"},{"type":"text","marks":[{"type":"strong"}],"text":"全局最優解"},{"type":"text","text":"。這是因爲所有的調度決策都是在整個集羣中的一部分節點中做出的，所有的調度決策都只是局部最優的，而這也是多調度器中的常見問題"},{"type":"sup","content":[{"type":"text","text":"3"}]},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在調度系統中，想要實現更好的擴展性就一定面臨着分片，分片必然導致調度器無法提供全局最優解並且顯著地增加系統的複雜性。我們從 Linux、Go 語言等 CPU 調度器的演進可以觀察到這點，最初的調度器大多數都是單線程的，爲了提高調度器的性能，會使用多調度器並引入工作竊取機制處理多調度器中待調度任務隊列的不平衡。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kubernetes v1.21 版本的內置調度器仍然是單線程的，它爲了在全局 5,000 個節點中做出最優的調度決策，需要使用不同的插件遍歷這 5,000 個節點並排序，而這也是影響其擴展性的重要原因之一。全局最優解聽起來是非常美好的設計，但是在調度這種比較複雜的場景中，局部最優解往往也都可以滿足需求，在業務上不需要保證該約束時，就可以通過多調度器來提升性能了。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當對比 Mesos 和靜態分片集羣的資源利用率時，我們會發現 Mesos 在 CPU 和內存的集羣資源利用率上都明顯高於使用靜態分片的集羣，而這個結果也不會造成太多的意外，因爲動態的資源分配策略一般都能夠提高集羣的資源利用率。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/61\/618fc0973b344d09e2f799ee58e88748.png","alt":"mesos-avg-utilization","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"圖 6 - Mesos 和靜態集羣的資源利用率對比"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"瞭解 Mesos 出現時解決的問題以及它的設計可以讓我們更好地理解今天面臨的挑戰，Mesos 在剛剛出現時是非常新穎的技術，與同期的其他產品來講確實提供了很強的靈活性，但是隨着 Yarn、Kubernetes 等技術的出現，它的很多場景也都被新技術取代，而這也是技術發展的必然趨勢。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"推薦閱讀"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/draveness.me\/papers-twine\/","title":null,"type":null},"content":[{"type":"text","text":"Facebook 集羣調度管理系統 · OSDI ‘20"}]}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/draveness.me\/papers-thunderbolt\/","title":null,"type":null},"content":[{"type":"text","text":"數據中心的電力超售 · OSDI ‘20"}]}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"Benjamin Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D. Joseph, Randy Katz, Scott Shenker, and Ion Stoica. 2011. Mesos: a platform for fine-grained resource sharing in the data center. In Proceedings of the 8th USENIX conference on Networked systems design and implementation (NSDI'11). USENIX Association, USA, 295–308. "},{"type":"link","attrs":{"href":"https:\/\/draveness.me\/papers-mesos\/#fnref:1","title":null,"type":null},"content":[{"type":"text","text":"↩︎"}]}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"What’s LXC? "},{"type":"link","attrs":{"href":"https:\/\/linuxcontainers.org\/lxc\/introduction\/","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/linuxcontainers.org\/lxc\/introduction\/"}]},{"type":"text","text":" "},{"type":"link","attrs":{"href":"https:\/\/draveness.me\/papers-mesos\/#fnref:2","title":null,"type":null},"content":[{"type":"text","text":"↩︎"}]}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"調度系統設計精要 "},{"type":"link","attrs":{"href":"https:\/\/draveness.me\/system-design-scheduler\/","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/draveness.me\/system-design-scheduler\/"}]},{"type":"text","text":" "},{"type":"link","attrs":{"href":"https:\/\/draveness.me\/papers-mesos\/#fnref:3","title":null,"type":null},"content":[{"type":"text","text":"↩︎"}]}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本作品採用"},{"type":"link","attrs":{"href":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/","title":null,"type":null},"content":[{"type":"text","text":"知識共享署名 4.0 國際許可協議"}]},{"type":"text","text":"進行許可，轉載時請註明原文鏈接，圖片在使用時請保留全部內容，可適當縮放並在引用處附上圖片所在的文章鏈接。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"文章圖片"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"你可以在 "},{"type":"link","attrs":{"href":"https:\/\/draveness.me\/sketch-and-sketch","title":null,"type":null},"content":[{"type":"text","text":"技術文章配圖指南"}]},{"type":"text","text":" 中找到畫圖的方法和素材。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自："},{"type":"link","attrs":{"href":"https:\/\/draveness.me\/","title":"xxx","type":null},"content":[{"type":"text","text":"面向信仰編程"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/draveness.me\/papers-mesos\/","title":"xxx","type":null},"content":[{"type":"text","text":"集羣管理系統 Mesos 的設計原理 · NSDI '11"}]}]}]}