無需擴容成本，百億集羣數據過期性能優化看這就行

原創

2021-07-15 13:03

節點類型

單個節點容器規格

說明

shardServer存儲節點

16CPU、64G內存、1T磁盤

1個分片存儲約30億數據

3個分片預計存儲100億數據

mongos及config server節點

8CPU\/8G內存\/50G磁盤

一個代理和一個config server節點複用同一個容器"}}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.2 集羣部署架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該業務數據不是很重要，爲了節省成本，因此我們採用2+1模式部署，也就是：2mongod+1arbiter模式，同城機房部署，部署架構圖如下圖所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a9\/a931a25db0c37054ae2a301386bc56c1.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"考慮到數據重要性不高，通過2mongod+1arbiter模式即可滿足用戶要求，同時可以最大化節省成本。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、性能優化過程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該集羣優化過程按照如下兩個步驟優化：業務使用集羣前的性能優化、業務使用過程中的性能優化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業務提前建好查詢對應的最優索引，同時創建過期索引："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\ndb.dailyCloudOperateInfo.createIndex( { \"createTime\": 1 }, { expireAfterSeconds: 604800} )"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.1 業務使用集羣前的性能優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"和業務溝通確定，業務每條數據都攜帶有一個設備標識userId，同時業務查詢更新等都是根據userId維度查詢該設備下面的單條或者一批數據，因此片建選擇userId。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分片方式"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了充分散列數據到3個分片，因此選擇hash分片方式，這樣數據可以最大化散列，同時可以滿足同一個userId數據落到同一個分片，保證查詢效率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"預分片"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"mongodb如果分片片建爲hashed分片，則可以提前做預分片，這樣就可以保證數據寫進來的時候比較均衡的寫入多個分片。預分片的好處可以規避非預分片情況下的chunk遷移問題，最大化提升寫入性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nsh.shardCollection(\"cloud_track.dailyCloudOperateInfo\", {userId:\"hashed\"}, false, { numInitialChunks: 8192} )\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"就近讀"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"客戶端增加secondaryPreferred配置，優先讀從節點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"禁用enable Majority Read Concern"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"禁用該功能後Read Concern majority將會報錯，Read Concern majority功能注意是避免髒讀，和業務溝通業務沒該需求，因此可以直接關閉。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"mongodb默認使能了enable Majority Read Concern，該功能開啓對性能有一定影響，參考："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MongoDB readConcern 原理解析："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/developer.aliyun.com\/article\/60553"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"OPPO百萬級高併發MongoDB集羣性能數十倍提升優化實踐："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/mongoing.com\/archives\/29934"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"存儲引擎cacheSize規格選擇"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"單個容器規格：16CPU、64G內存、7T磁盤，考慮到全量遷移過程中對內存壓力，內存碎片等壓力會比較大，爲了避免OOM，設置cacheSize=42G。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.2 業務使用過程中遇到的問題及性能優化"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"1）第一輪優化：存儲引擎優化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業務高峯期主要是數據寫入和更新，內存髒數據較多，當髒數據比例達到一定比例後用戶讀寫請求對應線程將會阻塞，用戶線程也會去淘汰內存中的髒數據page，最終寫性能下降明顯。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"wiredtiger存儲引擎cache淘汰策略相關的幾個配置如下:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ab\/ab7893eb65833ce784dbe1db0d919227.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於業務全量遷移數據是持續性的大流量寫，而不是突發性的大流量寫，因此eviction_target、eviction_trigger、eviction_dirty_target、eviction_dirty_trigger幾個配置用處不大，這幾個參數閥值只是在短時間突發流量情況下調整纔有用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是，在持續性長時間大流量寫的情況下，我們可以通過提高wiredtiger存儲引擎後臺線程數來解決髒數據比例過高引起的用戶請求阻塞問題，淘汰髒數據的任務最終交由evict模塊後臺線程來完成。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"全量大流量持續性寫存儲引擎優化如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\ndb.adminCommand( { setParameter : 1, \"wiredTigerEngineRuntimeConfig\" : \"eviction=(threads_min=4, threads_max=20)\"})"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"2）第一輪優化後存在的問題"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經過存儲引擎後臺線程數的調優後，數據寫入和更新瓶頸解決，寫入和更新過程時延都很平滑。但是隨着一週後開始數據過期，業務寫開始大量抖動，如下所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/52\/52d804815fb9be94435aa0eb71b5e5ed.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從上圖可以看出平均時延極端情況下甚至達到了幾百ms，這是業務完全接受不了的。通過mongostat監控發現如下現象："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主節點mongostat監控統計"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/af\/af883e2eb2480823f259160b27821ff2.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從上面的監控可以看出，三個分片的每個主節點只有4000左右的寫更新操作(注意：實際上還有4萬左右的delete操作，由於主節點過期delete不會統計，因此只能通過從節點查看，詳見後面分析，實際單個分片主節點寫和刪除操作4.4W\/s)，寫流量很低。但是，監控中的髒數據比例持續性的超過20%，超過20%後業務的請求就需要進行髒數據淘汰，最終造成業務請求阻塞抖動。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過前面的分析可以看出，業務正常峯值寫入3個分片差不多10W\/s。一星期後，七天前的數據需要過期，這時候過期刪除的ops也需要delete刪除10w\/S，由於這時候新數據同樣按照10w\/s寫入，因此集羣就需要支持20w\/s的ops操作才能支持。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"顯然，3個分片支持不了持續性的20w\/s左右的ops操作，因此如何不擴容情況下支撐業務需求將是一大難點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲何ttl過期主節點沒用delete統計"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/4b\/4bce188119306f8892b61be29d0a4e65.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖爲mongodb單機模塊架構圖，主節點默認啓用一個TTLMonitor線程，藉助查詢引起模塊實時掃描過期索引，然後把滿足條件的數據刪除。整個刪除過程沒有走command命令處理模塊，而命令計數操作只會在command模塊計數，因此主節點的過期刪除不會有delete操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"命令處理模塊處理流程及其計數統計詳見："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"mongodb內核源碼模塊化設計與實現專欄"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"更多mongodb模塊化源碼設計實現詳見："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"https:\/\/github.com\/y123456yz\/reading-and-annotate-mongodb-3.6"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.3 第二輪優化(過期刪除放凌晨低峯期)-不可行"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從前面的分析可以看出，我們三個分片支撐不了持續性20W\/S的更新和刪除操作，因此我們考慮業務改造使用方式，把過期刪除確定到凌晨低峯期。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是業務上線後出現其他的問題，由於業務凌晨會持續性的批量拉取分析過去幾天的數據，如果過期和批量讀數據疊加到一起，嚴重影響業務查詢效率。最終，該方案不可行，如果不擴容，則需要其他方案。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.4 第三輪方案優化(天維度建表，過期刪表)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了不增加成本，同時3個分片又支撐不了20W\/s的讀寫刪除等導致，爲了儘量通過3個分片不擴容條件下來滿足用戶需求，因此轉變方式，通過刪表的方式來避免過期，具體實現如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業務改造代碼，以天爲單位建表，每天的數據存到對應的表中。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"建表後業務啓用預分片功能，確保數據均衡分佈到三個分片。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業務保存8天數據，第九天凌晨刪除第一天的表"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 該方案業務時延統計對應收益如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/46\/46f61dbfa09b645087d64a7809a3e86d.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/c8\/c83d31b7066fbce52eb6d71f2bccfe90.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上圖所示，通過過期方式的優化，最終問題徹底解決，並且讀寫時延控制在0.5ms-2ms。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、優化收益總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過前面的一系列優化，最終沒有擴容，並且解決了業務過期和大量數據寫入更新引起的時延抖動問題，總體收益如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"改造優化前時延：經常幾百ms抖動"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"改造優化後時延：0.5-2ms"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者介紹"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"楊亞洲，"},{"type":"text","text":"前滴滴出行專家工程師，現任OPPO文檔數據庫mongodb負責人，負責數萬億級數據量文檔數據庫mongodb內核研發、性能優化及運維工作，一直專注於分佈式緩存、高性能服務端、數據庫、中間件等相關研發。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：dbaplus社羣（ID：dbaplus）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/hlbmoeGbXi550Xhsa18WOA","title":"xxx","type":null},"content":[{"type":"text","text":"無需擴容成本，百億集羣數據過期性能優化看這就行！"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}