記某百億級mongodb集羣數據過期性能優化實踐

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 某百億級mongodb業務只保存近期7天的數據，由於數據量大、流量高，數據過期刪除點比較集中，同時不能錯峯方式解決問題，因此如何利用最小物理成本來滿足業務需求就成爲了本集羣性能優化的難點。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過幾輪和業務配合調優，包括存儲引擎調優、數據刪除方式調優、業務錯峯讀寫等，最終完美解決了業務痛點，達到ms級業務讀寫訪問。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"關於作者","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 前滴滴出行專家工程師，現任OPPO文檔數據庫mongodb負責人，負責數萬億級數據量文檔數據庫mongodb內核研發、性能優化及運維工作，一直專注於分佈式緩存、高性能服務端、數據庫、中間件等相關研發。後續持續分享《MongoDB內核源碼設計、性能優化、最佳運維實踐》，Github賬號地址:","attrs":{}},{"type":"link","attrs":{"href":"https://github.com/y123456yz","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"https://github.com/y123456yz","attrs":{}}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"序言","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    本文是《mongodb 源碼實現、調優、最佳實踐系列》專欄的第 22 篇文章，其他文章可以參考如下鏈接：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/304a748ad3dead035a449bd51","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"Qcon-萬億級數據庫MongoDB集羣性能數十倍提升及機房多活容災實踐","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/0c51f3951f3f10671d7d7123e","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"Qcon現代數據架構-《萬億級數據庫MongoDB集羣性能數十倍提升優化實踐》核心17問詳細解答","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/def1add963ccb58680eea472f","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"百萬級高併發mongodb集羣性能數十倍提升優化實踐(上篇)","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/329cd8f770f445d87e26637bc","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"百萬級高併發mongodb集羣性能數十倍提升優化實踐(下篇)","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/372320c6bb93ddc5b7ecd0b6b","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"盤點2020 |我要爲分佈式數據庫mongodb在國內影響力提升及推廣做點事","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/7b2c1dc67de82972faac2812c","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":" 百萬級代碼量mongodb內核源碼閱讀經驗分享","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/180d98535bfa0c3e71aff1662","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"話題討論| mongodb擁有十大核心優勢，爲何國內知名度遠不如mysql高？","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/d4460b807c9835c6d80707410","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"Mongodb網絡模塊源碼實現及性能極致設計體驗","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/1cac5adcd1b4f3fe512a1457a","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"網絡傳輸層模塊實現二","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/971baa2c4351702de4cff69d1","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"網絡傳輸層模塊實現三","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/209eb17f8ea651f63bd187e36","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"網絡傳輸層模塊實現四","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/ed28fa03590e58fd88e987c82","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"command命令處理模塊源碼實現一","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/6de3aef4978a5078da44a8228","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"command命令處理模塊源碼實現二","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/3184cdc42c26c86e2749c3e5c","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"mongodb詳細表級操作及詳細時延統計實現原理(快速定位表級時延抖動)","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/81d2214c49049f5d70e3eb448","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"[圖、文、碼配合分析]-Mongodb write寫(增、刪、改)模塊設計與實現","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/edit/5932858d57db13d43a8b8d62a","title":"","type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"300條數據變更引發的血案-記某十億級核心mongodb集羣部分請求不可用故障踩坑記","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/06fef667e37bec350769da795","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"記十億級Es數據遷移mongodb成本節省及性能優化實踐","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/5aefe08216533ce0aa698339e","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"千億級數據遷移mongodb成本節省及性能優化實踐","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/2bc78d36adef6832ada8ea7c5","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"千億級數據遷移mongodb成本節省及性能優化實踐(附性能對比質疑解答)","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/98daf7330a3107fa0bf1edc9c","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"記某百億級mongodb集羣數據過期性能優化實踐","attrs":{}}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"1. 業務背景","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 線上某業務數據量約100億，白天爲寫流量高峯期，峯值寫約14W/s，如下圖所示：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d2/d268fa19c769edd20f1fd0375fb24b65.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a5/a5520eecf22347126f95e743398c9493.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 業務每天白天生產數據，凌晨批量拉取過期幾天數據做大數據報表分析，整個集羣數據只保留一週七天。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 單條數據約800字節，如下所示：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"1.  {  \n2.        \"_id\" : ObjectId(\"608592008bd3dad61675b491\"),  \n3.        \"deviceSn\" : \"xxxxxxxxxxxxx\",  \n4.        \"itemType\" : 0,  \n5.        \"module\" : \"backup\",  \n6.        \"userId\" : \"609986109\",  \n7.        \"callTimes\" : NumberLong(2),  \n8.        \"capacityAdd\" : NumberLong(0),  \n9.        \"capacityDelete\" : NumberLong(0),  \n10.        \"capacityDownload\" : NumberLong(0),  \n11.        \"capacityModify\" : NumberLong(0),  \n12.        \"createTime\" : NumberLong(\"1619366400003\"),  \n13.        \"expireAt\" : ISODate(\"2021-05-02T22:53:45.497Z\"),  \n14.        \"numAdd\" : NumberLong(2),  \n15.        \"numDelete\" : NumberLong(0),  \n16.        \"numDownload\" : NumberLong(0),  \n17.        \"numModify\" : NumberLong(0),  \n18.        \"osVersion\" : \"other\",  \n19.        \"reversedUserId\" : \"901689909\",  \n20.        \"updateTime\" : NumberLong(\"1619366402106\")  \n21.} ","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"1. mongodb資源評估及部署架構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 通過和業務對接梳理，該集羣規模及業務需求總結如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"①　數據量百億級","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"②　單條數據800字節，100億條預計7.5T數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"③　讀寫分離","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"④　所有數據只保留七天","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.1 mongodb資源評估","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"     分片數及存儲節點套餐規格選定評估過程如下：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"內存評估","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 我司都是容器化部署，以以網經驗來看，mongodb對內存消耗不高，歷史百億級以上mongodb集羣單個容器最大內存基本上都是64Gb，因此內存規格確定爲64G。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"分片評估","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    業務流量峯值10W/s多，預計需要3個分片支持讀寫。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"磁盤評估","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    100億數據7.5T，由於mongodb默認有高壓縮，預計真實磁盤佔用2.5~3T左右。三個分片，一個分片剛好1T。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"CPU規格評估","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 由於容器調度套餐化限制，因此CPU只能限定爲16CPU(實際上用不了這麼多CPU)。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"mongos代理及config server規格評估","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 此外，由於分片集羣還有mongos代理和config server複製集，因此還需要評估mongos代理和config server節點規格。由於config server只主要存儲路由相關元數據，因此對磁盤、CUP、MEM消耗都很低；mongos代理只做路由轉發只消耗CPU，因此對內存和磁盤消耗都不高。最終，爲了最大化節省成本，我們決定讓一個代理和一個config server複用同一個容器，容器規格如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 8CPU/8G內存/50G磁盤，一個代理和一個config server節點複用同一個容器。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"分片及存儲節點規格總結：","attrs":{}},{"type":"text","text":"4分片/16CPU、64G內存、1T磁盤。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"mongos及config server規格總結：","attrs":{}},{"type":"text","text":"8CPU/8G內存/50G磁盤","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ac/ac38c97e14dccb3fd24831a12b6d1665.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.2集羣部署架構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"     該業務數據不是很重要，爲了節省成本，因此我們採用2+1模式部署，也就是：2mongod+1arbiter模式，同城機房部署，部署架構圖如下圖所示：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4f/4f2ca96f5b127448d2ce41e3a859eea3.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 考慮到數據重要性不高，通過2mongod+1arbiter模式即可滿足用戶要求，同時可以最大化節省成本。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4. 性能優化過程","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 該集羣優化過程按照如下兩個步驟優化：業務使用集羣前的性能優化、業務使用過程中的性能優化。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業務提前建好查詢對應的最優索引，同時創建過期索引：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" db.dailyCloudOperateInfo.createIndex( { \"createTime\": 1 }, { expireAfterSeconds: 604800} )","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4.1 業務使用集羣前的性能優化","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 和業務溝通確定，業務每條數據都攜帶有一個設備標識userId，同時業務查詢更新等都是根據userId維度查詢該設備下面的單條或者一批數據，因此片建選擇userId。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"分片方式","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 爲了充分散列數據到3個分片，因此選擇hash分片方式，這樣數據可以最大化散列，同時可以滿足同一個userId數據落到同一個分片，保證查詢效率。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"預分片","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongodb如果分片片建爲hashed分片，則可以提前做預分片，這樣就可以保證數據寫進來的時候比較均衡的寫入多個分片。預分片的好處可以規避非預分片情況下的chunk遷移問題，最大化提升寫入性能。","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"sh.shardCollection(\"xx.xx\", {userId:\"hashed\"}, false, { numInitialChunks: 8192} )","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"就近讀","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"客戶端增加secondaryPreferred配置，優先讀從節點。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"禁用enableMajorityReadConcern","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 禁用該功能後ReadConcern majority將會報錯，ReadConcern majority功能注意是避免髒讀，和業務溝通業務沒該需求，因此可以直接關閉。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongodb默認使能了enableMajorityReadConcern，該功能開啓對性能有一定影響，參考：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://developer.aliyun.com/article/60553","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"MongoDB readConcern 原理解析","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://mongoing.com/archives/29934","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"OPPO百萬級高併發MongoDB集羣性能數十倍提升優化實踐","attrs":{}}]}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"存儲引擎cacheSize規格選擇","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 單個容器規格：16CPU、64G內存、7T磁盤，考慮到全量遷移過程中對內存壓力，內存碎片等壓力會比較大，爲了避免OOM，設置cacheSize=42G。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"5.2 業務使用過程中遇到的問題及性能優化","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5.2.1 第一輪優化：存儲引擎優化","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 業務高峯期主要是數據寫入和更新，內存髒數據較多，當髒數據比例達到一定比例後用戶讀寫請求對應線程將會阻塞，用戶線程也會去淘汰內存中的髒數據page，最終寫性能下降明顯。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" wiredtiger存儲引擎cache淘汰策略相關的幾個配置如下:","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ce/ce594719b28632e5f24a1840dd4c56cb.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    由於業務全量遷移數據是持續性的大流量寫，而不是突發性的大流量寫，因此eviction_target、eviction_trigger、eviction_dirty_target、eviction_dirty_trigger幾個配置用處不大，這幾個參數閥值只是在短時間突發流量情況下調整纔有用。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      但是，在持續性長時間大流量寫的情況下，我們可以通過提高wiredtiger存儲引擎後臺線程數來解決髒數據比例過高引起的用戶請求阻塞問題，淘汰髒數據的任務最終交由evict模塊後臺線程來完成。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"      全量大流量持續性寫存儲引擎優化如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"db.adminCommand( { setParameter : 1, \"wiredTigerEngineRuntimeConfig\" : \"eviction=(threads_min=4, threads_max=20)\"})","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5.2.2第一輪優化後存在的問題","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 經過存儲引擎後臺線程數的調優後，數據寫入和更新瓶頸解決，寫入和更新過程時延都很平滑。但是隨着一週後開始數據過期，業務寫開始大量抖動，如下所示：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3e/3e698d9b48df713ced550570110e5047.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 從上圖可以看出平均時延極端情況下甚至達到了幾百ms，這啥業務完全接受不了的。通過mongostat監控發現如下現象：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"主節點mongostat監控統計","attrs":{}}]}]}],"attrs":{}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/10/100b88f96020075764ae87174f22429d.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 從上面的監控可以看出，三個分片的每個主節點只有4000左右的更新操作，寫流量很低。但是，監控中的髒數據比例持續性的超過20%，超過20%後業務的請求就需要進行髒數據淘汰，最終造成業務請求阻塞抖動。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 通過前面的分析可以看出，業務正常峯值寫入3個分片差不多10W/s。一星期後，七天前的數據需要過期，這時候過期刪除的ops也需要delete刪除10w/S，由於這時候新數據同樣按照10w/s寫入，因此集羣就需要支持20w/s的ops操作才能支持。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 顯然，3個分片支持不了持續性的20w/s左右的ops操作，因此如何不擴容情況下支撐業務需求將是一大難點。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"爲何ttl過期主節點沒用delete統計","attrs":{}}]}]}],"attrs":{}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/68/688112fe52284a71747b380a4883b437.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 上圖爲mongodb單機模塊架構圖，主節點默認啓用一個TTLMonitor線程，藉助查詢引起模塊實時掃描過期索引，然後把滿足條件的數據刪除。整個刪除過程沒有走command命令處理模塊，而命令計數操作只會在command模塊計數，因此主節點的過期刪除不會有delete操作。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 命令處理模塊處理流程及其計數統計詳見：","attrs":{}},{"type":"link","attrs":{"href":"https://github.com/y123456yz/reading-and-annotate-mongodb-3.6","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"mongodb內核源碼模塊化設計與實現專欄","attrs":{}}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5.3 第二輪優化(過期刪除放凌晨低峯期)-不可行","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 從前面的分析可以看出，我們三個分片支撐不了持續性20W/S的更新和刪除操作，因此我們考慮業務改造使用方式，把過期刪除確定到凌晨低峯期。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 但是業務上線後出現其他的問題，由於業務凌晨會持續性的批量拉取分析過去幾天的數據，如果過期和批量讀數據疊加到一起，嚴重影響業務查詢效率。最終，該方案不可行，如果不擴容，則需要其他方案。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5.4 第三輪方案優化(天維度建表，過期刪表)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 爲了不增加成本，同時3個分片又支撐不了20W/s的讀寫刪除等導致，爲了儘量通過3個分片不擴容條件下來滿足用戶需求，因此轉變方式，通過刪表的方式來避免過期，具體實現如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"①　業務改造代碼，以天爲單位建表，每天的數據存到對應的表中。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"②　建表後業務啓用預分片功能，確保數據均衡分佈到三個分片。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"③　業務保存8天數據，第九天凌晨刪除第一天的表","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    該方案業務時延統計對應收益如下：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/19/19c81b4d8bbe427c9e8ba0b9ede89a9a.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/da/da1fc3f574bf108982f694d20bfca053.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 如上圖所示，通過過期方式的優化，最終問題徹底解決，並且讀寫時延控制在0.5ms-2ms。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"6 優化收益總結","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 通過前面的一系列優化，最終沒有擴容，並且解決了業務過期和大量數據寫入更新引起的時延抖動問題，總體收益如下：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"改造優化前時延：經常幾百ms抖動","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"改造優化後時延：0.5-2ms","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]}]}