千億級數據遷移mongodb成本節省及性能優化實踐

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 線上某IOT核心業務集羣之前採用mysql作爲主存儲數據庫，隨着業務規模的不斷增加，mysql已無法滿足海量數據存儲需求，業務面臨着容量痛點、成本痛點問題、數據不均衡問題等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 400億該業務遷移mongodb後，同樣的數據節省了極大的內存、CPU、磁盤成本，同時完美解決了容量痛點、數據不均衡痛點，並且實現了一定的性能提升。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 此外，遷移時候的mysql數據爲400億，3個月後的現在對應mongodb集羣數據已增長到1000億，如果以1000億數據規模等比例計算成本，實際成本節省比例會更高。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 當前國內很多mongod文檔資料、性能數據等還停留在早期的MMAP_V1存儲引擎，實際上從mongodb-3.x版本開始，mongodb默認存儲引擎已經採用高性能、高壓縮比、更小鎖粒度的wiredtiger存儲引擎，因此其性能、成本等優勢相比之前的MMAP_V1存儲引擎更加明顯。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"關於作者","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 前滴滴出行技術專家，現任 OPPO 文檔數據庫 mongodb 負責人，負責 oppo 數萬億級數據量文檔數據庫 mongodb 內核研發及運維工作，一直專注於分佈式緩存、高性能服務端、數據庫、中間件等相關研發。Github 賬號地址:","attrs":{}},{"type":"link","attrs":{"href":"https://github.com/y123456yz","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"https://github.com/y123456yz","attrs":{}}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"序言","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    本文是《mongodb 源碼實現、調優、最佳實踐系列》專欄的第 21 篇文章，其他文章可以參考如下鏈接：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/304a748ad3dead035a449bd51","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"Qcon-萬億級數據庫MongoDB集羣性能數十倍提升及機房多活容災實踐","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/0c51f3951f3f10671d7d7123e","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"Qcon現代數據架構-《萬億級數據庫MongoDB集羣性能數十倍提升優化實踐》核心17問詳細解答","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/def1add963ccb58680eea472f","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"百萬級高併發mongodb集羣性能數十倍提升優化實踐(上篇)","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/329cd8f770f445d87e26637bc","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"百萬級高併發mongodb集羣性能數十倍提升優化實踐(下篇)","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/372320c6bb93ddc5b7ecd0b6b","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"盤點2020 |我要爲分佈式數據庫mongodb在國內影響力提升及推廣做點事","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/7b2c1dc67de82972faac2812c","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":" 百萬級代碼量mongodb內核源碼閱讀經驗分享","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/180d98535bfa0c3e71aff1662","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"話題討論| mongodb擁有十大核心優勢，爲何國內知名度遠不如mysql高？","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/d4460b807c9835c6d80707410","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"Mongodb網絡模塊源碼實現及性能極致設計體驗","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/1cac5adcd1b4f3fe512a1457a","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"網絡傳輸層模塊實現二","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/971baa2c4351702de4cff69d1","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"網絡傳輸層模塊實現三","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/209eb17f8ea651f63bd187e36","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"網絡傳輸層模塊實現四","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/ed28fa03590e58fd88e987c82","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"command命令處理模塊源碼實現一","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/6de3aef4978a5078da44a8228","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"command命令處理模塊源碼實現二","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/3184cdc42c26c86e2749c3e5c","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"mongodb詳細表級操作及詳細時延統計實現原理(快速定位表級時延抖動)","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/draft/65357","title":"","type":null},"content":[{"type":"text","text":"https://xie.infoq.cn/draft/65357","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/edit/5932858d57db13d43a8b8d62a","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"300條數據變更引發的血案-記某十億級核心mongodb集羣部分請求不可用故障踩坑記","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/06fef667e37bec350769da795","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"記十億級Es數據遷移mongodb成本節省及性能優化實踐","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/draft/66405","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"千億級數據遷移mongodb成本節省及性能優化實踐","attrs":{}}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"1. 業務遷移背景","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 該業務在遷移mongodb前已有約400億數據，申請了64套mysql集羣，由業務通過shardingjdbc做分庫分表，提前拆分爲64個庫，每個庫100張表。主從高可用選舉通過依賴開源orchestrator組建，mysql架構圖如下圖所示：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/68/685960f86e39a8091f18d18dd9406dc9.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"  說明：","attrs":{}},{"type":"text","text":"上圖中紅色代表磁盤告警，很多節點磁盤使用水位即將100%。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 如上圖所示，業務一年多前一次性申請了64套MySQL集羣，單個集羣節點數一主三從，每個節點規格如下：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"cpu：4","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"mem：16G","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"磁盤：500G","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總節點數：64*4=256","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SSD服務器","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 該業務運行一年多時間後，總集羣數據量達到了400億，並以每月200億速度增長，由於數據不均衡等原因，造成部分集羣數據量大，持續性耗光磁盤問題。由於節點衆多，越來越多的集羣節點磁盤突破瓶頸，爲了解決磁盤瓶頸，DBA不停的提升節點磁盤容量。業務和DBA都面臨嚴重痛點，主要如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"①　數據不均衡問題","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"②　節點容量問題","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"③　成本持續性增加","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"④　DBA工作量劇增(部分磁盤提升不了需要遷移數據到新節點)，業務也提心吊膽","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"業務","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"2. 爲何選擇mongodb-附十大核心優勢總結","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 業務遇到瓶頸後，基於mongodb在公司已有的影響力，業務開始調研mongodb，通過和業務接觸瞭解到，業務使用場景都是普通的增、刪、改、查、排序等操作，同時查詢條件都比較固定，用mongodb完全沒任何問題。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 此外，mongodb相比傳統開源數據庫擁有如下核心優索：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"優勢一：模式自由","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongodb爲schema-free結構，數據格式沒有嚴格限制。業務數據結構比較固定，該功能業務不用，但是並不影響業務使用mongodb存儲結構化的數據。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"優勢二：天然高可用支持","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mysql高可用依賴第三方組件來實現高可用，mongodb副本集內部多副本通過raft協議天然支持高可用，相比mysql減少了對第三方組件的依賴。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"優勢三：分佈式-解決分庫分表及海量數據存儲痛點","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongodb是分佈式數據庫，完美解決mysql分庫分表及海量數據存儲痛點，業務無需在使用數據庫前評估需要提前拆多少個庫多少個表，mongodb對業務來說就是一個無限大的表(當前我司最大的表存儲數千億數據，查詢性能無任何影響)。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 此外，業務在早期的時候一般數據都比較少，可以只申請一個分片mongodb集羣。而如果採用mysql，就和本次遷移的IOT業務一樣，需要提前申請最大容量的集羣，早期數據量少的時候嚴重浪費資源。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"優勢四：完善的數據均衡機制、不同分片策略、多種片建類型支持","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關於balance：支持自動balance、手動balance、時間段任意配置balance.","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關於分片策略：支持範圍分片、hash分片，同時支持預分片。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關於片建類型：支持單自動片建、多字段片建","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"優勢五：不同等級的數據一致性及安全性保證","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongodb在設計上根據不同一致性等級需求，支持不同類型的Read Concern 、Write Concern讀寫相關配置，客戶端可以根據實際情況設置。此外，mongodb內核設計擁有完善的rollback機制。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"優勢六：高併發、高性能","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 爲了適應大規模高併發業務讀寫，mongodb在線程模型設計、併發控制、高性能存儲引擎等方面做了很多細緻化優化。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"優勢七：wiredtiger高性能存儲引擎設計","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 網上很多評論還停留在早期MMAPv1存儲引擎，相比MMAPv1，wiredtiger引擎性能更好，壓縮比更高，鎖粒度更小，具體如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"①　WiredTiger提供了低延遲和高吞吐量","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"②　處理比內存大得多的數據，而不會降低性能或資源","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"③　系統故障後可快速恢復到最近一個checkpoint","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"④　支持PB級數據存儲","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"⑤　多線程架構，盡力利用樂觀鎖併發控制算法減少鎖操作","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"⑥　具有hot-caches能力","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"⑦　磁盤IO最大化利用，提升磁盤IO能力","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"⑧　其他","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 更多WT存儲引擎設計細節可以參考：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http://source.wiredtiger.com/3.2.1/architecture.html","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"http://source.wiredtiger.com/3.2.1/architecture.html","attrs":{}}]}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"優勢八：成本節省-WT引擎高壓縮比支持","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"     mongodb對數據的壓縮支持snappy、zlib算法，在以往線上真實的數據空間大小與真實磁盤空間消耗進行對比，可以得出以下結論：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"①　mongodb默認的snappy壓縮算法壓縮比約爲2.2-4.5倍","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"②　zlib壓縮算法壓縮比約爲4.5-7.5倍(本次遷移採用zlib高壓縮算法)","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a5/a5925b23a0f2889f4cf3e556773122aa.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 此外，以線上已有的從mysql、Es遷移到mongodb的真實業務磁盤消耗統計對比，同樣的數據，存儲在mongodb、Mysql、Es的磁盤佔比≈1：3.5：6，不同數據存儲佔比有差距。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"優勢九：天然N機房(不管同城還是異地)多活容災支持","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongodb天然高可用機制及代理標籤自動識別轉發功能的支持，可以通過節點不同機房部署來滿足同城和異地N機房多活容災需求，從而實現成本、性能、一致性的“三豐收”。更多機房多活容災的案例詳見Qcon分享：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://qcon.infoq.cn/2020/shenzhen/track/916","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"OPPO萬億級文檔數據庫MongoDB集羣性能優化實踐","attrs":{}}]}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"優勢十：完善的客戶端均衡訪問策略","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongodb客戶端訪問路由策略由客戶端自己指定，該功能通過Read Preference實現，支持primary 、primaryPreferred 、secondary 、secondaryPreferred 、nearest 五種客戶端均衡訪問策略。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"分佈式事務支持","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongodb-4.2版本開始已經支持分佈式事務功能，當前對外文檔版本已經迭代到version-4.2.12，分佈式事務功能也進一步增強。此外，從mongodb-4.4版本產品規劃路線圖可以看出，mongodb官方將會持續投入開發查詢能力和易用性增強功能，例如union多表聯合查詢、索引隱藏等","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://github.com/y123456yz/reading-and-annotate-mongodb-3.6","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"mongodb源碼分析、更多實踐案例細節","attrs":{}}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"3. mongodb資源評估及部署架構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 業務開始遷移mongodb的時候，通過和業務對接梳理，該集羣規模及業務需求總結如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"①　已有數據量400億左右","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"②　數據磁盤消耗總和30T左右","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"③　讀寫峯值流量4-5W/s左右，流量很小","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"④　同城兩機房多活容災","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"⑤　讀寫分離","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"⑥　每月預計增加200億數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"⑦　滿足幾個月內1500億新增數據需求","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" 說明：","attrs":{}},{"type":"text","text":"數據規模和磁盤消耗按照單副本計算，例如mysql 64個分片，256個副本，數據規模和磁盤消耗計算方式爲：64個主節點數據量之和、64個分片主節點磁盤消耗之和。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.1 mongodb資源評估","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"     分片數及存儲節點套餐規格選定評估過程如下：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"內存評估","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 我司都是容器化部署，以往經驗來看，mongodb對內存消耗不高，歷史百億級以上mongodb集羣單個容器最大內存基本上都是64Gb，因此內存規格確定爲64G。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"分片評估","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"     業務流量峯值3-5W/s，考慮到可能後期有更大峯值流量，因此按照峯值10W/s寫，5w/s讀，也就是峯值15W/s評估，預計需要4個分片。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"磁盤評估","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mysql中已有數據400億，磁盤消耗30T。按照以網線上遷移經驗，mongodb默認配置磁盤消耗約爲mysql的1/3-1/5，400億數據對應mongodb磁盤消耗預計8T。考慮到1500億數據，預計4個分片，按照每個分片400億規模，預計每個分片磁盤消耗8T。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 線上單臺物理機10多T磁盤，幾百G內存，幾十個CPU，爲了最大化利用服務器資源，我們需要預留一部分磁盤給其他容器使用。另外，因爲容器組套餐化限制，最終確定確定單個節點磁盤在7T。預計7T節點，4個分片存儲約1500億數據。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"CPU規格評估","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 由於容器調度套餐化限制，因此CPU只能限定爲16CPU(實際上用不了這麼多CPU)。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"mongos代理及config server規格評估","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 此外，由於分片集羣還有mongos代理和config server複製集，因此還需要評估mongos代理和config server節點規格。由於config server只主要存儲路由相關元數據，因此對磁盤、CUP、MEM消耗都很低；mongos代理只做路由轉發只消耗CPU，因此對內存和磁盤消耗都不高。最終，爲了最大化節省成本，我們決定讓一個代理和一個config server複用同一個容器，容器規格如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"8CPU/8G內存/50G磁盤，一個代理和一個config server節點複用同一個容器。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"分片及存儲節點規格總結：","attrs":{}},{"type":"text","text":"4分片/16CPU、64G內存、7T磁盤。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"mongos及config server規格總結：","attrs":{}},{"type":"text","text":"8CPU/8G內存/50G磁盤","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/80/80442b8981f3e0d07e0ea7ccd3dab206.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.2集羣部署架構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"     由於該業務所在城市只有兩個機房，因此我們採用2+2+1(2mongod+2mongod+1arbiter模式)，在A機房部署2個mongod節點，B機房部署2個mongod節點，C機房部署一個最低規格的選舉節點，如下圖所示：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a7/a7a981963d7c2655aed85802520247e5.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"說明：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"①　每個機房代理部署2個mongos代理，保證業務訪問代理高可用，任一代理掛掉，對應機房業務不受影響。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"②　如果機房A掛掉，則機房B和機房C剩餘2mongod+1arbiter，則會在B機房mongod中從新選舉一個主節點。arbiter選舉節點不消耗資源","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"③　客戶端配置nearest ，實現就近讀，確保請求通過代理轉發的時候，轉發到最近網絡時延節點，也就是同機房對應存儲節點讀取數據。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"④　弊端：如果是異地機房，B機房和C機房寫存在跨機房寫場景。A B爲同城機房，則沒有該弊端，同城機房時延可以忽略。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4. 業務全量+增量遷移方式","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e3/e3249341cd27a565b313a497164a97d0.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 遷移過程由業務自己完成，通過阿里開源的datax工具實現，該遷移工具的更多細節可以參考：","attrs":{}},{"type":"link","attrs":{"href":"https://github.com/alibaba/DataX","title":null,"type":null},"content":[{"type":"text","text":"https://github.com/alibaba/DataX","attrs":{}}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"5. 性能優化過程","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 該集羣優化過程按照如下兩個步驟優化：數據遷移開始前的提前預優化、遷移過程中瓶頸分析及優化、遷移完成後性能優化。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"5.1數據遷移開始前的提前預操作","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 和業務溝通確定，業務每條數據都攜帶有一個設備標識ssoid，同時業務查詢更新等都是根據ssoid維度查詢該設備下面的單條或者一批數據，因此片建選擇ssoid。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"分片方式","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 爲了充分散列數據到4個分片，因此選擇hash分片方式，這樣數據可以最大化散列，同時可以滿足同一個ssoid數據落到同一個分片，保證查詢效率。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"預分片","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongodb如果分片片建爲hashed分片，則可以提前做預分片，這樣就可以保證數據寫進來的時候比較均衡的寫入多個分片。預分片的好處可以規避非預分片情況下的chunk遷移問題，最大化提升寫入性能。","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"sh.shardCollection(\"xxx.xxx\", {ssoid:\"hashed\"}, false, { numInitialChunks: 8192} )","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" 注意事項：","attrs":{}},{"type":"text","text":"切記提前對ssoid創建hashed索引，否則對後續分片擴容有影響。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"就近讀","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    客戶端增加nearest 配置，從離自己最近的節點讀，保證了讀的性能。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"mongos代理配置","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      A機房業務只配置A機房的代理，B機房業務只配置B機房代理，同時帶上nearest配置，最大化的實現本機房就近讀，同時避免客戶端跨機房訪問代理。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"禁用enableMajorityReadConcern","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 禁用該功能後ReadConcern majority將會報錯，ReadConcern majority功能注意是避免髒讀，和業務溝通業務沒該需求，因此可以直接關閉。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongodb默認使能了enableMajorityReadConcern，該功能開啓對性能有一定影響，參考：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://developer.aliyun.com/article/60553","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"MongoDB readConcern 原理解析","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://mongoing.com/archives/29934","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"OPPO百萬級高併發MongoDB集羣性能數十倍提升優化實踐","attrs":{}}]}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"存儲引擎cacheSize規格選擇","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"     單個容器規格：16CPU、64G內存、7T磁盤，考慮到全量遷移過程中對內存壓力，內存碎片等壓力會比較大，爲了避免OOM，設置cacheSize=42G。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"5.2數據全量遷移過程中優化過程","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/57/57ba6c9e492cc75e57b8e83a7ac90e2c.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 全量數據遷移過程中，遷移速度較塊，內存髒數據較多，當髒數據比例達到一定比例後用戶讀寫請求對應線程將會阻塞，用戶線程也會去淘汰內存中的髒數據page，最終寫性能下降明顯。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" wiredtiger存儲引擎cache淘汰策略相關的幾個配置如下:","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f7/f714cdddbb3921cf8b929821576779f9.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    由於業務全量遷移數據是持續性的大流量寫，而不是突發性的大流量寫，因此eviction_target、eviction_trigger、eviction_dirty_target、eviction_dirty_trigger幾個配置用處不大，這幾個參數閥值只是在短時間突發流量情況下調整纔有用。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      但是，在持續性長時間大流量寫的情況下，我們可以通過提高wiredtiger存儲引擎後臺線程數來解決髒數據比例過高引起的用戶請求阻塞問題，淘汰髒數據的任務最終交由evict模塊後臺線程來完成。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      全量大流量持續性寫存儲引擎優化如下：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"db.adminCommand( { setParameter : 1, \"wiredTigerEngineRuntimeConfig\" : \"eviction=(threads_min=4, threads_max=20)\"})","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"5.3全量遷移完成後，業務流量讀寫優化","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e9/e9dcf61564af46078487b4781ef952e2.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 前面章節我們提到，在容器資源評估的時候，我們最終確定選擇單個容器套餐規格爲如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"16CPU、64G內存、7T磁盤。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 全量遷移過程中爲了避免OOM，預留了約1/3內存給mongodb server層、操作系統開銷等，當全量數據遷移完後，業務寫流量相比全量遷移過程小了很多，峯值讀寫OPS約2-4W/s。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 也就是說，前量遷移完成後，cache中髒數據比例幾乎很少，基本上不會達到20%閥值，業務讀流量相比之前多了很多(數據遷移過程中讀流量走原mysql集羣)。爲了提升讀性能，因此做了如下性能調整(提前建好索引)：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"節點cacheSize從之前的42G調整到55G，儘量多的緩存熱點數據到內存，供業務讀，最大化提升讀性能。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"每天凌晨低峯期做一次cache內存加速釋放，避免OOM。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"    ","attrs":{}},{"type":"text","text":"上面的內核優後後，業務測時延監控曲線變化，時延更加平穩，平均時延也有25%左右的性能優後，如下圖所示：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4c/4c9c46791a78c83bd69ba799a0a9b196.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"6. 遷移前後，業務測時延統計對比(Mysql vs mongodb)","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"6.1性能收益對比","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"遷移前業務測時延監控曲線(平均時延7ms,  2月1日數據，此時mysql集羣只有300億數據)：","attrs":{}}]}]}],"attrs":{}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/28/284d094458b73e070bfc4eaa57a09df0.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"遷移mongodb後並且業務流量全部切到mongodb後業務測時延監控曲線(平均6ms,  3月6日數據，此時mongodb集羣已有約500億數據))","attrs":{}}]}]}],"attrs":{}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/6d/6d1434e98b3e5eff60090e1255ff788c.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"總結：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"mysql(300億數據)時延：約7ms","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"mongodb(500億數據)時延：約6ms","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"6.2性能質疑解答","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 該文有部分同學可能質疑性能數據，認爲mongodb實例規格是16CPU/64G內存/7T磁盤，而mysql是4CPU/16G內存/500G磁盤。認爲mongodb規格更高，而mysql資源規格低。但是忽略了單節點數據量和流量這個因素，按照單實例對比，總結如下(由於只記錄了mysql 300億時候、mongodb 500億時候的業務測時延，因此還是以這兩個時間點爲例比較)：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Mysql和mongodb的CPU都不是瓶頸，都很空閒，兩者之間容器規格唯一區別就是內存，單實例規格、數據量、業務測時延等對比總結(單實例mysql數據量約300/64=4.7億，mongodb約125億)：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/65/65f656a3ff9bb99743ecc4635361f5cc.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  如果mysql採用mongodb同樣的規格，由於mysql同樣數據磁盤消耗是mongodb 3.3倍，因此需要22T左右磁盤，並且承擔同樣的數據量和流量，性能會不會好於方案1？這個不是很確定，因爲都是線上環境，不可能爲了驗證這個測試而大費周章。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 如上，方案3和方案1、方案2的性能對比有待驗證。實際上，mongodb當前4個分片已經1000億數據了，客戶端訪問時延基本上沒有變化，還是約6ms，因此實際上如果同等資源規格驗證，客觀數mysql單個節點需要承擔如下數據量和業務流量：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/41/41eee3ef9226c64e3fe61cf635abd424.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"7. 遷移成本收益對比","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"7.1 Mysql集羣規格及存儲數據最大量","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a7/a72ca72f3f95f5d5ec68ec1b95432561.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 原mysql集羣一共64套，每套集羣4副本，每個副本容器規格：4CPU、16G mem、500G磁盤，總共可以存儲400億數據，這時候大部分節點已經開始磁盤90%水位告警，DBA對部分節點做了磁盤容量提升。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總結如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"①　集羣總套數：64","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"②　單套集羣副本數：4","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"③　每個節點規格：4CPU、16G mem、500G磁盤","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"④　該64套集羣最大存儲數據量：400億","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"7.2 mongodb集羣規格及存儲數據最大量","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f8/f8e87653caa381db4bf9bf925809f5dd.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongodb從mysql遷移過來後，數據量已從400億增加到1000億，並以每個月增加200億數據。mongodb集羣規格及存儲數據量總結如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"①　分片數：4","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"②　單分片副本數：4","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"③　每個節點規格：16CPU、64G mem、7T磁盤","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"④　四個分片存儲數據量：當前已存1000億，最大可存1500億數據。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"7.3成本對比計算過程","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" 說明：","attrs":{}},{"type":"text","text":"由於mysql遷移mongodb後，數據不在往mysql中寫入，流量切到mongodb時候mysql中大約存儲有400億數據，因此我們以這個時間點做爲對比時間點。以400億數據爲基準，資源消耗對比如下表(每個分片只計算主節點資源消耗，因爲mysql和mongodb都是4副本)：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/36/3622641e1ebedcc049fb85c11f688215.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 由於mongodb四個分片還有很多磁盤冗餘，該四個分片相比400億數據，還可以寫1200億數據。如果按照1600億數據計算，如果還是按照mysql之前套餐規格，則mysql集羣數需要再增加三倍，也就是總集羣套數需要64*4=256套，資源佔用對比如下：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/6d/6d3f656ca98b4af0ef6b6af703dd2f92.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"7.4收益總結(客觀性對比)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 從上面的內容可以看出，該業務遷移mongodb後，除了解決了業務容量痛點、促進業務快速迭代開發、性能提升外，成本還節省了數倍。成本節省總結如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"①　400億維度計算(mysql和mongodb都存儲相同的400億數據)：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CPU和內存成本比例：4:1","attrs":{}}]},{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"磁盤成本比例：3.3:1","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"②　1500億維度計算(mysql集羣都採用之前規格等比例換算)：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CPU和內存成本比例：16:1","attrs":{}}]},{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"磁盤成本比例：3.3:1","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 從上面的分析可以看出，數據量越大，按照等比例換算原則，mongodb存儲成本會更低，原因如下：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"CPU/內存節省原因：","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 主要是因爲mongodb海量數據存儲及高性能原因，索引建好後，單實例單表即使幾百億數據，讀寫也是ms級返回(注意：切記查詢更新建好索引)。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，由於mongodb分佈式功能，對容量評估更加方便，就無需提前一次性申請很多套mysql，而是根據實際需要可以隨時加分片。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"磁盤節省原因：","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongodb存儲引擎wiredtiger默認高壓縮、高性能。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" 最後，鑑於客觀性成本評價，CPU/內存成本部分可能會有爭議，","attrs":{}},{"type":"text","text":"比如mysql內存和CPU是否申請的時候就申請過大。mongodb對應CPU也同樣存在該問題，例如申請的單個容器是16CPU，實際上真實只消耗了幾個CPU。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" 但是，磁盤節省是實時在在的，是","attrs":{}},{"type":"text","text":"相同數據情況下mysql和mongodb的真實磁盤消耗對比。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當前該集羣總數據量已經達到千億級，並以每個月200億規模增加，單從容器計費層面上換算，1000億數據按照等比例換算，預計可節省極大的成本。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"8. 最後：千億級中等規模mongodb集羣注意事項","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongodb無需分庫分表，單表可以無限大，但是單表隨着數據量的增多會引起以下問題：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"①　切記提前建好索引，否則影響查詢更新性能(數據越多，無索引查詢掃描會越慢)。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"②　切記提前評估好業務需要那些索引，單節點單個表數百億數據，加索引執行時間較長。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"③　服務器異常情況下節點替換時間相比會更長。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"④　切記數據備份不要採用mongodump/mongorestore方式，而是採用熱備或者文件拷貝方式備份。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"⑤　節點替換儘量從備份中拷貝數據加載方式恢復，而不是通過主從全量同步方式，全量同步過程較長。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"9. 未來挑戰(該集羣未來萬億級實時數據規模挑戰)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 隨着時間推移，業務數據增長也會越來越多，單月數據量增長曲線預計會直線增加(當前每月數據量增加200億左右)，預計未來2-3年該集羣總數據量會達到萬億級，分片數也會達到20個分片左右，可能會遇到各自各樣的問題。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是，IOT業務數據存在明顯的冷數問題，一年前的數據用戶基本上不會訪問，因此我們考慮做如下優後來滿足性能、成本的進一步提升：冷數據歸檔到低成本SATA盤","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"冷數據提升壓縮比，最大化減少磁盤消耗","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如何解決冷數據歸檔sata盤過程中的性能問題","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 冷熱歸檔存儲可以參考之前在Qcon、dbaplus、mongodb中文社區分享的另一篇文章：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http://dbaplus.cn/news-162-3666-1.html","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"用最少人力玩轉萬億級數據，我用的就是MongoDB！","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://github.com/y123456yz/reading-and-annotate-mongodb-3.6","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"mongodb源碼分析、更多實踐案例細節","attrs":{}}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"10. 最後說明(業務場景總結)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 本千億級IOT業務使用場景總結如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"①　本分享的業務數據讀、更新、排序等都可以走索引，包括單字段索引、多字段索引、數組索引，所有查詢和更新都能確定走具體的某個最優索引。   ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"②　查詢都是單表查詢，不涉及多表聯合查詢。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 數據庫場景非常重要，脫離業務場景談數據庫優劣無任何意義。例如本文的業務場景，業務能確定需要建那些索引，同時所有的更新、查詢、排序都可以對應具體的最優索引，因此該場景就非常適合mongodb。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]}]}