以物理學思維破解分佈式系統的本質

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"前言"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文的動機在於應用物理學思維模型破解分佈式系統背後不變的本質，並以此解讀分佈式系統裏的各種算法設計、功能設計以及非功能設計。在“分佈式系統” 這個詞語裏，關鍵詞可以分爲“分佈式”及“系統”，而"},{"type":"text","marks":[{"type":"bgcolor","attrs":{"color":"#FC8F99","name":"red"}},{"type":"strong"}],"text":"“系統 = 要素 + 連接 + 目標”"},{"type":"text","text":"，在這個公式裏，要素的變化不會影響系統的本質，而“連接或目標”的變化就會改變系統的本質，其中“分佈式”是系統的連接方式，挖掘分佈式系統背後的本質即是挖掘“分佈式系統”的要素、連接以及目標的本質。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"物理學的價值觀在於追求所有物理現象背後的共同的底層規律，並以此解讀各種物理現象，並且其具有“可解釋、可重複、可預測”的可度量性，這種共同的底層規律，被稱之爲元認知，即第一性原理：“任何變化的背後都有不變的本質”。將這種思維模型應用於挖掘分佈式系統的本質，需要解決兩個問題，即：“ 什麼是分佈式系統的第一性原理？以及如何度量分佈式系統？”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"分佈式系統的價值與目的"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統的出現是爲了解決一個主要矛盾，即：“日益增長的數據計算、傳輸與存儲的需求與當前單點計算機能力無法滿足這個需求之間的矛盾”。分佈式系統可以通過伸展集羣規模解決這個矛盾，因此這就是分佈式系統的價值，而可伸縮性(Scalability，避免與可擴展性extensibility混淆)也是分佈式系統的根本目的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"分佈式系統必知的基礎理論與算法"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統必須理解、必須會的基礎理論算法有：CAP/PACELC、BASE、2PC、3PC、TCC、ACID、PAXOS、RAFT這9個："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CAP: CAP理論認爲以下三者不能同時滿足："}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"PACELC: PACELC理論是CAP理論的擴展，如果有分區partition (P)，系統就必須在availability 和consistency (A and C)之間取得平衡; 否則else (E) 當系統運行在無分區情況下,系統需要在 latency (L) 和 consistency (C)之間取得平衡”；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"BASE: BASE是基本可用（Basically Available）、軟狀態（Soft state）和最終一致性（Eventually consistent）三個短語的縮寫；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2PC：two-phase commit protocol，兩階段提交；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3PC: three-phase commit protocol ，三階段提交，其在兩階段提交的基礎上增加了CanCommit階段，並引入了超時機制；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TCC: Try-Confirm-Cancel，又稱補償事務，其核心思想是：\"針對每個操作都要註冊一個與其對應的確認和補償（撤銷操作）\"；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ACID: 原子性（Atomicity）、一致性（Consistency）、隔離性（Isolation）、持久性（Durablity）；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"PAXOS/RAFT ：PAXOS與RAFT算法都是最有效的解決分佈式一致性問題的算法。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這幾條基礎理論與算法需要自己深入學習理解，其是分佈式系統的必備知識點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"分佈式系統的功能與非功能"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"功能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"功能可按職責劃分爲服務功能與算法功能:"}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統裏的最主要的服務功能有：服務提供，服務註冊，服務配置，服務調用、服務路由、服務治理設計，服務觀測、服務安全這8項；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統裏的最主要的算法功能有：冪等性設計、事務算法設計、端到端的校驗算法設計、路由算法設計、分區分配算法設計、集羣視圖變更算法設計、心跳算法設計、註冊算法設計、複製一致性算法設計以及容量規劃算法設計。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"非功能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"非功能可劃分爲質量與約束："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"質量是分佈式系統在約束條件下的度量方式，其涵蓋：合適的性能（Performant）、可用性(Availability)、可靠性(Reliability)、可伸縮性(Scalability)、韌性(resilience)、可觀測性(Observability)、安全性（security）、易用性（usability）、可運維性（operability）、可測試性(testability)、可維護性(maintainability)、可擴展性(extensibility)、可讀性(readability)等。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"約束是分佈式系統的資源限制：網絡物理容量與計算機節點的物理容量，以及客戶、用戶、團隊的邊界約束。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"bgcolor","attrs":{"color":"#FC8F99","name":"red"}},{"type":"strong"}],"text":"分佈式系統交付的目的是功能的價值，但是產品的功夫卻體現在非功能"},{"type":"text","text":"，分佈式系統的質量是分佈式系統的度量方式，分佈式系統要可度量就需要具有“可解釋、可複製、可預測”的質量保證。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"分佈式系統的第一性原理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"依據李善友老師的定義："},{"type":"text","marks":[{"type":"bgcolor","attrs":{"color":"#FC8F99","name":"red"}}],"text":" "},{"type":"text","marks":[{"type":"bgcolor","attrs":{"color":"#FC8F99","name":"red"}},{"type":"strong"}],"text":"“第一性原理思維 = 邏輯奇點 + 公理化方法 ”"},{"type":"text","text":"，邏輯奇點即基石假設，公理化方法我認爲是\"定公理、推定理、再公式化應用\"。因此欲找出分佈式系統的第一性原理，就需要先挖掘出分佈式系統的公理化定義及其邏輯奇點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先把分佈式系統概念化，Google 出來的對分佈式系統的定義有："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A distributed system is a system whose components are located on different networked computers, which communicate and coordinate their actions by passing messages to one another. The components interact with one another in order to achieve a common goal."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"即："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統是指其組件位於不同的網絡計算機上的系統，這些組件通過相互傳遞消息來進行通信和協調其動作，且彼此相互交互以完成一個共同的任務目標。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"拆解這句話，從中可以看到分佈式系統裏的要素即爲組件，連接即網絡，目標是共同的任務，並且還可以看出4個要點："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統的組件是位於不同的網絡計算機上；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統的組件通過傳遞消息進行通信其動作；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統的組件通過傳遞消息進行協調其動作；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統的組件是通過相互交互以完成一個共同的任務目標；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中最最重要的可以看作是分佈式系統的基石假設的要點是："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1，分佈式系統的組件是位於不同的網絡計算機上；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2，這些組件通過相互傳遞消息來進行通信和協調其動作，且彼此相互交互以完成一個共同的任務目標。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這兩點即爲分佈式系統的邏輯奇點，破除了這兩點那就不是分佈式系統，比如去掉網絡計算機的定義，那就是單機系統，去掉協調以完成共同的任務目標，那就只是一個計算機網絡。這兩點基石假設構成分佈式系統的邏輯奇點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"到此，可以得出分佈式系統的公理化定義："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統是指其組件位於不同的網絡計算機上的系統，這些組件通過相互傳遞消息來進行通信和協調其動作，且彼此相互交互以完成一個共同的任務目標。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以及分佈式系統的邏輯起點："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統是指其組件位於不同的網絡計算機上的系統：即計算機網絡；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這些組件通過相互傳遞消息來進行通信和協調其動作，且彼此相互交互以完成一個共同的任務目標：即協同：“諧調一致，和合共同，協調兩個或者兩個以上的不同資源或者個體，一致地完成某一共同目標”；"}]}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"再進一步抽象，可以推斷出“分佈式系統就是通過計算機網絡進行協同工作的系統”， 至此，可以推出分佈式的公理化定義公式："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"bgcolor","attrs":{"color":"#FC8F99","name":"red"}},{"type":"strong"}],"text":"分佈式系統 = 計算機 + 網絡 + 協同，其以質量爲度量"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個公式就是分佈式的第一性原理公式，是分佈式的本質理論定義，其中“計算機”是分佈式系統的要素，“網絡”是分佈式系統的連接，“協同”是分佈式系統的目標，從這公式裏可以看出分佈式系統的3個原生的難題："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統是基於網絡的系統，那麼網絡自身所具有的所有的優點與缺點它都有，那麼如何提高服務的可靠性？如何保證服務的可用性？如何保證網絡可運維？"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統是基於消息傳遞的系統，消息傳遞是不可靠的，那麼如何保證消息的正確性？如何保證消息傳遞的可靠性？如何傳遞消息到目的地？如何保證消息傳遞的負載均衡？"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統是協同工作的系統，那麼如何協調大量的計算機節點的完成一個共同的目標，如何解決協調的複雜性以及提高協調的可靠性、可用性？那麼如何一起交互完成一個共同的目標任務？如何拆分目標？如何聚合目標，如何度量完成任務的質量與邊界？"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此還需要依據分佈式系統的公理化定義推導出定理化定義。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"分佈式系統的定理化推導"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“公理是不證自明的，而定理是以若干的公理或其他定理爲基礎而推導的”。由公理推定理，從分佈式的公理化公式"},{"type":"text","marks":[{"type":"strong"}],"text":"\"分佈式系統 = 計算機  + 網絡 + 協同\""},{"type":"text","text":"，可知分佈式系統是組件位於“不同的計算機網絡”上一起“協同”工作的系統，這句話得出分佈式系統三要素：“計算機、網絡，協同”。其中計算機是系統要素，改變計算機爲虛機或者容器，不會改變分佈式系統的本質。網絡是系統的連接，改變連接就會改變系統的本質，當連接一臺計算機時，就是單機，當連接兩臺計算機時就是鏡像，當連接 多於兩臺計算機時就是分佈式。協同是系統的目標，當改變協同的目標也就改變了系統的功能，比如共同計算與共同存儲、共同調度，其功能目標是不一樣的。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"計算機"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統是基於不同的計算機上的系統，計算機也是分佈式系統的要素之一，因此分佈式系統也繼承了計算機的原生缺點："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"計算機節點是會出故障的，主板、CPU、網卡、硬盤、內存、電源等都會出故障，比如老化、失效等；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"計算機節點內的操作系統是會突然奔潰不能提供服務的；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"計算機節點是會突然掉電的；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"計算機節點裏的內存下電是不保數據的；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"計算機節點的資源是有限的：CPU是有算力上限的、內存是有大小限制的、網卡有吞吐量限制、硬盤有空間大小限制以及速率限制；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這幾個計算機的原生缺點意味着分佈式系統需要能夠知道計算機節點是失效的，以及在計算機節點失效的同時保證服務質量設計，那麼就應當進行以下幾點保證："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可觀測性（observability）設計：監控、告警、日誌、追蹤；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可靠性（Reliability）設計：冗餘設計、分區分配設計、複製算法設計、冪等性設計、一致性算法設計；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"容量（capacity）規劃設計：計算機節點資源資源有限，就需要分佈式系統進行進行容量規劃；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務治理（Service governance）設計：CPU算力有限、內存有限、網卡吞吐量有限、磁盤IO有限，因此需要進行服務治理之隔板設計以及限流、限併發設計。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"網絡"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統是基於不同的網絡上的系統，網絡是分佈式系統的連接方式之一，連接一臺計算機的系統就是單機系統，連接兩臺計算機的系統就是鏡像系統，連接三臺及以上數目的計算機的系統就是分佈式系統，因此分佈式系統也繼承了網絡的原生缺點，即："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡是不可靠的；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡是會出故障的；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡是有時延的；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡是會抖動的；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡是不安全的；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡是會丟包的；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡是有帶寬限制的；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡消息是會亂序的；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由此，爲了保證分佈式系統的服務質量：性能、可用性、可靠性、安全性等，那麼就需要進行服務質量保證設計，其可以劃分爲："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡是不可靠以及會出現斷網之類的故障的，因此分佈式系統需要進行服務治理容錯設計；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡亂序以及丟包，因此分佈式系統需要冪等性算法設計、端到端的校驗算法設計；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡帶寬有限，因此分佈式系統需要網絡容量設計以及服務治理限流設計；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡不安全：因此分佈式系統需要服務安全設計；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡是有時延的，因此分佈式系統需要進行性能設計：更好的硬件、跟短的IO路徑；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡是會抖動的，因此分佈式系統需要進行服務治理容錯之超時處理設計。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"協同"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"協同是指：“諧調一致，和合共同，協調兩個或者兩個以上的不同資源或者個體，一致地完成某一共同目標“，這些組件通過相互傳遞消息來進行通信和協調其動作，且彼此相互交互以完成一個共同的任務目標"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"協同又可以拆分爲協調動作與共同完成任務，即："}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"協調動作"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統的組件通過傳遞消息進行通信及協調其動作，即：因此依據消息傳遞的特性以及缺點需要進行相應的協調動作設計："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"傳遞消息特性意味着需要進行RPC調用設計；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡裏傳遞的消息是經常不一樣的，因此需要序列化編解碼設計；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息傳遞具有丟消息、丟處理的弊端，爲了解決這個弊端就需要進行 ：冪等性設計、事務處理設計、日誌設計；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息的傳遞是會超時的，因此就需要服務治理容錯之超時處理設計；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息是基於網絡傳輸的，而網絡是可能隨時出故障的，因此需要對消息進行可觀測性設計即消息追蹤設計；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"需要知道消息從哪裏來，往哪裏去就需要一個配置中心管理集羣各個計算機的信息，比如IP，因此需要進行配置中心設計；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"光知道可以發往哪裏還不夠，還要知道發往的節點是活着的可服務的，需要知道計算機節點的服務狀態，還需要保證消息路由的負載均衡，因此這就就需要一個協調的服務註冊中心，進行服務組件的註冊與心跳檢測、消息路由以及按集羣視圖變更算法管理集羣狀態表，那麼就需要註冊中心設計、註冊算法設計、負載均衡算法設計，心跳算法設計、集羣視圖變更算法設計以及集羣狀態表管理設計。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"共同完成任務"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統的組件是通過相互交互以完成一個共同的任務目標，因此需要解決共同完成任務並且保證任務完成質量帶來的難題："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整個分佈式系統需要能接收任務以及返回完成的任務，那麼就需要有提供服務的能力，需要有服務調用接口設計，服務調用客戶端以及可視化界面；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了能讓網絡裏的N臺計算機相互交互完成一個共同的目標，就需要對任務進行拆分以及聚合設計；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"任務拆分後要能知道發給哪個節點，那麼就需要一個配置中心, 從配置中心獲取目標節點信息；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"任務拆分後分發給節點同時要保證處理的性能，那麼就需要進行可伸縮性設計，一臺機器處理不過來就需要N臺機器一起處理從而保證處理的性能質量，因此又衍生出需要路由算法設計或分區分配算法設計，這樣拆分後的任務可以被分發到不同的獨立的節點進行處理，並且，但一個節點不可用時，還可以分發到其他的可用的節點，從而提升了系統性能與可用性；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"任務又可以分爲計算任務與存儲任務，如果是存儲任務，爲了保證數據的可用性以及可靠性，就需要對分區進行冗餘設計，即節點副本設計，如果需要節點副本設計又引入了選主算法設計、數據一致性複製算法設計與冪等性設計；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了解決選主與複製一致性問題，又出現了PAXOS,RAFT,2PC,3PC 等，這樣的基礎一致性協議算法。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"至此，依據分佈式的公理化公式："},{"type":"text","marks":[{"type":"strong"}],"text":"\"分佈式系統 = 計算機 + 網絡 + 協同\""},{"type":"text","text":"，推導出了分佈式的定理化推論，解讀了分佈式系統裏爲什麼需要進行這些功能與非功能設計的問題，接下來還需要講述分佈式系統的度量。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"分佈式系統的度量"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統是依據分佈式公理定義的質量進行度量的，其涵蓋以下幾項內容："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"合適的性能（Performant），性能指標一般包括 TPS, QPS, Latency, IOPS， response time等，這裏用”合適的性能“作爲表達，指的是性能合適即可、夠用即可，高性能當然好，但是高性能也意味着更高的成本，有些場景高性能反而是一種浪費行爲，性能需求需要理解業務場景適可而止；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可用性(Availability)，可用性指的是系統長時間可對外提供服務的能力，通常採用小數點後的9的個數作爲度量指標，按照這種約定“五個九”等於0.99999（或99.999％）的可用性，默認企業級達標的可用性爲6個9。但是當前從時間維度來度量可用性已經沒有太大的意義，因爲設計得好的系統可以在系統出現故障得情況下也能保證對外提供得服務不中斷，因此，當前更合適得可用性度量指標 是請求失敗率；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可靠性(Reliability)，可靠性一般指系統在一定時間內、在一定條件下可以無故障地執行指定功能的能力或可能性， 也是採用小數點後的9的個數作爲度量指標，通常5個9的可靠性就可以滿足企業級達標；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可伸縮性(Scalability)，是指通過向系統添加資源來處理越來越多的工作並且維持高質量服務的能力，其受可用性以及可靠性的制約，集羣規模越大出故障的概率越高從而降低可用性、可靠性，爲了保證可用性以及可靠性達標，需要適配合理的可伸縮性指標；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"韌性(resilience)，通常也叫容錯性（fault-tolerant），也就是健壯和強壯的意思，指的是系統的對故障與異常的處理能力，比如在軟件故障、硬件故障、認爲故障這樣的場景下，系統還能保持正常工作的能力；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可觀測性(Observability)，是一種設計理念，包括告警、監控、日誌與跟蹤，可以實時地更深入地觀測系統內部的工作狀態；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"安全性（security），指的是阻止非授權使用，阻止非法訪問以及使用，保護合法用戶的資產的能力；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"易用性（usability），指的是軟件的使用難易程度，對於產品的易用性來說， 易用性不僅僅 是軟件使用角度的易用，還包括安裝、部署、升級上的易用,升值還包括硬件層面的易用，比如產品的外觀，形狀等；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可運維性（operability），可運維性指的是運維人員對系統進行運維操作的難易程度，主要包含以下幾個方面的難以程度： 系統的部署、升級、修改、監控以及告警等；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可測試性（ testability），指的是單元測試，集成測試，打樁測試等的難易；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可維護性（Maintainability）， 指的是代碼升級，部署，定位bug，添加功能的難易；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可擴展性（ extensibility）， 指的是未來增加新的功能與模塊的難易；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可讀性（ readability），指的是代碼的易理解程度。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"邊界約束：集羣規模、計算機的容量等物理資源的限制，以及客戶、用戶、團隊的約束需求。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"依據這幾項質量度量指標，可以保證分佈式的“可解釋、可複製、可預測”。其中要保證質量的核心思想是"},{"type":"text","marks":[{"type":"strong"}],"text":"“共享資源、消除資源競用性以及平衡負載。”"},{"type":"text","text":"，共享資源需要註冊中心，消除資源競用性就需要服務治理，平衡負載需要好的路由算法。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"分佈式系統的反熵增與數據守恆"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"熵增"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"熵增定律是物理學的基本定律之一，其被定義爲："}]},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":" 熵（Entropy）是用以度量一個系統“內在的混亂程度”，即系統中的無效能量；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":" 熵增定律：在一個孤立系統裏，如果沒有外力做功，其總混亂度（熵）會不斷增大；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"熵增過程是一個自發的由有序向無序發展的過程並且具有必然性，爲了保證有序就必須逆熵增做功。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式系統也是一個孤立的系統，其中的網絡與計算機節點（涵蓋電源、主板、CPU、內存、網卡、硬盤等）等硬件會老化、會出故障，組件之間協同工作也會遇到負載過高、軟件系統出現BUG等問題，這也是一個熵增的過程，並且因爲熵增的必然性，分佈式系統總是自發地或非自發地不斷由有序走向無序，最終不可逆地走向失效不可用。爲了保證分佈式系統是有序可用的就必須逆熵增做功，即對其反熵增，分佈式系統的反熵增過程與方法是："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可運維設計：軟硬件的部署與升級設計、可視化設計、可觀測性（監控、告警、日誌、追蹤）設計；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可服務設計：由團隊解決故障以及提供服務的支持；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務治理設計：熔斷、限流、降級、隔離、容錯，觸使分佈式系統保持在有序狀態；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"智能化設計：參數自我優化、故障自我判斷、工作負載自我預測等；"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"動態平衡設計：動態平衡是一種設計理念，有進有出；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此其中在分佈式系統裏將需要將可運維、可服務、可治理、可智能化、動態平衡的思想融合到架構設計與開發中。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"數據守恆"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"能量守恆定律也是物理學的基本定律之一，其被定義爲："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1，能量既不會憑空產生，也不會憑空消失，它只會從一種形式轉化爲另一種形式，或者從一個物體轉移到其它物體，而能量的總量保持不變；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2，孤立系統的總能量保持不變。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在分佈式系統裏"},{"type":"text","marks":[{"type":"strong"}],"text":"“數據”"},{"type":"text","text":"即是分佈式系統的能量，因此參照“能量守恆”定義，這裏我給分佈式系統一個"},{"type":"text","marks":[{"type":"strong"}],"text":"“數據守恆”"},{"type":"text","text":"定義："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"孤立系統的總數據量保持不變。數據既不會憑空產生，也不會憑空消失，它只會從一種形式轉化爲另一種形式，或者從一個物體轉移到其它物體， 而數據的總量保持不變。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"依據上節的定理推導，我們知道分佈式系統裏網絡是不可靠的、消息傳遞是不可靠的、計算機節點是不可靠的、磁盤是不可靠的、內存是不可靠的、軟件組件是不可靠的等等，這些過程都會丟數據，因此爲了保證分佈式系統裏的"},{"type":"text","marks":[{"type":"strong"}],"text":"“數據守恆”"},{"type":"text","text":"就需要對分佈式系統進行數據可靠性設計：即："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分區設計、冗餘設計、冪等性設計、端到端的校驗設計、日誌設計、事務處理設計，緩存的MESI設計等。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此在分佈式系統裏將也需要將"},{"type":"text","marks":[{"type":"strong"}],"text":"“數據守恆”"},{"type":"text","text":"的思想融合到架構設計與開發中。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"小結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文以物理學思維挖掘分佈式系統的本質，推導出了分佈式系統爲什麼需要這樣的設計的緣由，並且文中闡述了分佈式系統的基礎理論、功能非功能、反熵增與數據守恆。日拱一卒，功不唐捐，分享是最好的學習，與其跟隨不如創新，希望這個知識點對大家有用。另作者能力與認知都有限，”我講的，可能都是錯的“，歡迎大家拍磚留念。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"作者簡介"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"常平，中國科學技術大學碩士研究生，深度學習首席軟件主管工程師，前EMC 大數據資深首席工程師，主要工作背景在深度學習、流式大數據、分佈式中間件以及Linux內核。"}]}]}