分佈式集羣如何實現高效的數據分佈

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"一、前言"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着互聯網的發展，用戶產生的數據越來越多，企業面臨着龐大數據的存儲問題，目前市面上主流的分佈式大數據文件系統，都是對數據切片打散，通過離散方法將數據散列在集羣的所有節點上，本文將帶你瞭解DHT（Distributed Hash Table）：分佈式哈希表是如何實現數據的分佈式離散存儲的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"DHT（Distributed Hash Table）：分佈式哈希表"}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"二、技術背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"互聯網發展早期，數據通常存儲在單個服務器上，初期數據增長較爲緩慢，可以通過提升單機的存儲能力滿足數據的增長需求；隨着互聯網普及程度的推進，用戶數、用戶產生和訪問的數據呈指數增長；單機已無法存儲用戶需要的數據。爲此，迫切需要多個服務器共同協作，存儲量級更大的數據。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"三、傳統 Hash"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"傳統 Hash 通過算法 hash()=X mod S 來對數據進行分散，在元數據比較分散的情況下，數據能夠很好的散列在集羣節點中。由於S代表了集羣的節點數，當進行集羣的擴容縮容時，S的變化會影響到歷史數據的命中問題，因此爲了提高數據命中率，會產生大量測數據遷移，性能較差。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"四、一個簡單的 DHT"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式 Hash 通過構造一個長度爲2的32次方（ipv4地址的數量）的環，將節點散列在 Hash環上，根據不同的 Hash算法計算出來的 Hash值有所不同，本文采用FNV Hash算法來計算 Hash值。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/22/22eadf4ad6b3cab9e2c2adf30442302d.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如圖所示，先將存儲節點通過 Hash計算後添加到 DHT 環上，每個節點距離上一個節點間的這段區間，作爲該節點的數據分區，Hash值落在這個分區的數據將存儲到這個節點上；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然後將數據通過 Hash算法散列到DHT環上，數據落到DHT環上後，按照順時針方向找到離自己最近的節點作爲數據存儲節點，如下所示，數據 ObjectA 落到節點 NodeA上，數據 ObjectB 落到節點 NodeB 上；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/54/54315ea3294290867a694e3f9076f403.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"初始化DHT的源碼如下："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/72/72a18615fbfbd5a16be7d2e32cedd236.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先定義了一個存放集羣節點元數據的Map，用來存儲接入到DHT環中的物理節點數據。然後定義了一個DHT環——vNodes，用來存儲DHT環中的節點位置信息。這樣我們就實現了一個簡單的DHT環，通過addPhysicalNode方法可以模擬集羣節點的加入。加入時會計算節點的Hash值並存放到vNodes中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b5/b539b771da0052a6294dd62d390145d9.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"初始化4個存儲節點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3b/3be96ccae6787905080e153b0520e7fc.webp","alt":null,"title":"","style":[{"key":"width","value":"25%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過countNodeValue方法插入100條數據，在寫數據的過程中，根據數據的 Hash值找到DHT環上最近的一個節點，然後將數據寫入該節點中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/65/65814b545314e796a64a0b10018b796d.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"插入100條數據後，各個節點的數據分佈如下，可以看見4個節點的數據並不均勻，只有一個節點分配到了數據（這與寫入的數據也有一定關係）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ad/ad644893939362fabe57de2cadf853f0.png","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"插入100萬條數據後，各個節點的數據分佈如下，雖然每個節點都分配到了數據，但仍然出現了較大的數據傾斜。這將導致99%的請求都會由Node3進行處理，出現一核有難三核圍觀的情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/85/851a1db4af46307c6119eccfc6d3e66e.webp","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"出現如上問題是什麼原因呢？通過查看DHT環上各節點的hash值不難看出，各節點間距不均勻，插入的數據按順時針查找節點時都找到了Node3，因此數據都寫到了Node3裏面，所以節點區間不均勻會使某些節點能覆蓋更多的數據，導致數據不均衡。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c4/c4bede8de90c2584f5739285f19bb91f.webp","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/03/03f216a1117819d5daad73143cf4f1d0.webp","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"說了一個簡單DHT環的基本原理後，再來思考一個問題：簡單的DHT環數據離散，但仍然存在數據傾斜的情況，還不如用傳統的hash方式分配數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前面提到傳統的hash方式在當時在節點故障後，整個集羣的數據會進行大量的遷移，影響集羣性能，那麼DHT能解決這一問題嗎？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們還是用之前分配好的100萬條數據，模擬節點4故障，如下圖所示，Node4上的數據只遷移到了Node1，對Node2和Node3不產生數據遷移，從而降低了節點故障導致每個節點都需要進行數據遷移帶來的影響。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0b/0bcaec3e05fd632ec2b976e40433db7b.webp","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"五、DHT 的改進"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1、虛擬節點"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大家思考一下，解決數據傾斜的問題可以如何解決？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過增加集羣節點的方式最簡單直接，目的是將更多的節點散列到DHT環上，使得環上所有節點分佈更加均勻，節點間的區間間隔儘可能的均衡，以下是10個節點和20個節點集羣的數據分佈情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/69/692415cd97fc1161882b937126cefa57.webp","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/be/bed9442323b674633b38c39d54622886.webp","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以發現，通過增加節點的方式，仍然無法從根本上解決數據傾斜的問題。並且增加節點會提高集羣的設備成本和維護成本。同時，這種方案還引出了一個嚴重的問題，如果Node20故障了，那麼Node20的數據會全數遷移到下一個節點上，最終導致集羣出現數據傾斜，數據較多的節點還將處理更多的IO請求，容易形成數據熱點，成爲性能瓶頸，引發集羣整體的性能下降。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/68/68b3ed8f0236b87ef605b103630586f6.webp","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"（2） 引入虛擬節點"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了解決數據傾斜的問題，引入了虛擬節點的概念。虛擬節點也就是真實節點的一個邏輯副本，如圖所示，對節點NodeA進行3次虛擬節點hash分佈，形成了虛擬節點NodeA1、NodeA2、NodeA3。當NodeA故障後，指向NodeA的數據會指向NodeB、NodeC。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/fd/fda444558ba631d527d6d613f983d9d9.png","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c1/c1304192cfedd3a47a8ff8d37544420d.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當引入虛擬節點數量爲100時，數據已經分散在各個節點上了，如果虛擬節點足夠多，最終將達到數據均衡的狀態。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/40/40e24fd48273dab00b9f9f581eb05dbc.webp","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"虛擬節點數據1萬時的數據分佈："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2d/2d34be27e60657dea1e9f33158b690a2.webp","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"虛擬節點數量爲100萬時的數據分佈："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/6f/6f054689db840574285e9598fcbffbc1.webp","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當Node3故障後，Node3上的數據被均勻的分散到其他節點上，不會出現數據傾斜的情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/fe/fe62db3714091ec90bf48c2374d87393.webp","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2、負載邊界因子"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣就完美了嗎？我們初始化一個4個節點的DHT環，虛擬節點設置爲100，然後插入100條數據，並打印DHT環的元數據信息如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/62/62be9a58159a4eb74bb3ff5eed5e3b1d.webp","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以發現，雖然設置的虛擬節點，但是仍然無法均衡的將節點散列到DHT環上，導致Node2過載，Node空閒。我們再思考一種極端場景，當我們的數據恰好計算hash值後都在區間A，而這個區間只有NodeA，那麼仍然出現了數據傾斜。如何解決這個問題呢，這裏我們引入一個叫負載邊界因子的概念。DHT環部署了4個節點，總共有100條數據要插入，那麼平均下來每個節點的權重就是100/4+1=26，當數據映射過程中達到了該節點的權重，則映射到下一個節點，下面是代碼實現。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/79/79e7abdce50ae0d672bd024c385aa023.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4d/4d69dcb9cfd51810a91adbd711460848.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"打開負載邊界因子開關的情況："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/71/7183bbb150deea76db931b8f5a3ae2fa.webp","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"boxShadow"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在打開負載邊界因子開關後，數據得到了較好的均衡。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"六、DHT 引發的思考"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上述的只是一個簡單的DHT，數據也做了簡化，數據的存儲和讀取都需要查詢DHT環，如何提升DHT的讀寫性能？如何提升DHT的高可靠？當節點故障後，如何將故障節點的數據遷移到新的節點？如何做好數據備份？如何保證副本數據不集中在一個節點上？也是需要去思考的，本文只是拋磚引玉簡單的介紹了DHT基本的思想，更多生產環境面臨的挑戰，在此不做展開。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到，DHT提供了一種負載均衡的思路。利用hash算法的特性，將數據或業務請求分散到集羣中的各個節點上，提高系統容錯性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"vivo  用戶運營開發團隊"}]}]}]}