300條數據變更引發的血案-記某十億級核心mongodb集羣部分請求不可用故障踩坑記

原創

2021-05-10 17:48

庫名

表名

功能說明

userbucket

whitexxx/expxxx

用戶路由信息表，約300條數據。用戶訪問feeds_xxxxxxx庫前必須先獲取該表得用戶路由數據

feeds_xxxxxxx

feeds_xxx_pool

近億數據，未啓用分片

hardware_xxx_cost

未啓用分片，少量數據

news_xxx_profile

數億數據，未啓用分片

resource_xxx_info

數千萬數據，啓用分片

resource_xxx_info

未啓用分片，數億數據

resource_xxx_info

未啓用分片，數億數據

"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/13/13c9961e254ce1a186fb972aafde53b4.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 上面的描述可以總結爲下圖：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ec/ec62520c3c6d51daa2af695f06338b73.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 從上圖可以看出，分片2和分片3幾乎沒起到任何作用；由於分片3有兩個節點爲低IO的sata盤，可能影響userbucket庫的讀寫，因此考慮直接removeShard從集羣中剔除分片3和分片2。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"2. 操作過程","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 由於分片3爲低IO服務器，有潛在抖動集羣抖動分享；同時分片2和分片3幾乎都是浪費的分片，因此打散直接通過如下removeshad命令刪除分片3和分片2信息，騰出無用服務器資源，如下圖所示：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/bb/bb4380ca059fca344585906683f18a6e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"步驟1：登陸任一一個代理，假設是代理mongos1。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"步驟2：由於分片3(也就是shard_8D5370B4分片)爲userbucket庫的主分片，因此報錯了，提示\"you need to drop or movePrimary these databases\"，意思是我們需要提前把該庫的主分片信息遷移到其他分片。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"步驟3：通過movePrimary命令把userbucket庫的主分片從分片3遷移到分片1。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 登陸監控列表中的其他兩個代理mongos2、mongos3，通過如下命令強制刷新路由信息： db.adminCommand({\"flushRouterConfig\":1})","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"注意事項：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" ","attrs":{}},{"type":"text","text":"由於movePrimary過程，其他代理不會感知到該庫的主分片變化，因此需要強制刷新路由信息或者重啓其他節點的mongos，參考如下：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f6/f6b244ed54796208e4a40b41399cd614.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"3. 用戶反饋大部分請求業務請求不可用","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 對含有300條數據的userbucket庫變更後，當我還在若無其事的處理其他集羣性能調優的時候，用戶突然很急的電話我反饋該核心集羣整個訪問不可用（注意：是整個10億數據的集羣不可用）。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 收到電話後很突然，和業務人員詳細對接後可以基本上確定是因爲這300條數據變更引起。業務獲取這300條數據的時候，部分請求獲取成功，部分請求獲取失敗，說明肯定和movePrimary有關係。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 於是，除了對監控列表中的所有代理做flushRouterConfig強制路由刷新外，還重啓了所有的代理，但是業務反饋，還是有部分請求獲取不到數據。比較棘手，我自己通過所有的mongos代理查看userbucket庫下面的300條數據，完全可以獲取到數據。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 於是懷疑是不是還有未刷新路由的mongos代理，於是登陸任一mongos代理獲取config.mongos表，查看結果如下：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f4/f4e39477f4bc732bfa1126f233ef9178.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 上面的config.mongos表記錄了該集羣所有的代理信息，同時記錄了這些代理和集羣最後一次ping通信的詳細時間信息。很明顯，該表中記錄的代理原不止集羣監控列表中的代理個數，比監控列表中的個數要多。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 最終，把config.mongos表中羅列的當前在線的所有代理強制通過flushRouterConfig刷新路由後，業務恢復。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4. 問題總結","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 通過前面的分析可以得出，由於早期集羣監控中漏掉了部分代理，造成這部分代理對應的userbucket路由信息是movePrimary前的路由信息，也就是指向了錯誤的分片，因此出現了路由不到數據的情況，如下圖所示：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2e/2eff527973e8342eae12dad44c410573.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"爲何用戶userbucket庫對應表中數據有的成功有的失敗？","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 因爲部分代理在moveprimary後，沒有強制刷新該表路由信息，造成部分代理路由獲取數據的時候路由錯誤。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"爲何該300條數據部分路由信息錯誤會造成整個10億集羣部分訪問不可用？","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"和業務實現邏輯有關係，因爲業務在獲取這10億條數據前首先需要獲取業務的路由信息，剛好業務路由信息存在了userbucket庫對應表中，業務在獲取數據前必須要獲取到業務的路由信息數據，如果userbucket數據獲取不到，用戶就無法確定指向feeds_xxxxxxx數據。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"爲何會遺漏部分代理重啓或者強制路由刷新？","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"歷史原因，造成部分代理業務代碼有配置，但是服務端集羣監控元數據遺漏了，也就是服務端集羣監控漏掉了部分代理，這部分代理沒有監控起來。也有可能是mongos代理擴容，但是集羣監控列表中沒有加入元數據。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"movePrimary操作最安全的操作方法？","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 官方建議movePrimary操作成功後需要強制路由刷新或者重啓mongos，但是movePrimary操作成功和mongos重啓這個過程中有個中間狀態，如果中間狀態業務讀或者些該遷移的庫下面的表，還是可能路由錯誤。因此，最佳安全的moveprimary可以通過如下兩個方法操作：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" 法一：","attrs":{}},{"type":"text","text":"shutdown所有代理，只留下一個代理，等該代理moveprimary成功後在重啓其他mongos代理。切記別遺漏代理，出現本文踩坑類似情況，提前檢查config.mongos表。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" 法二：","attrs":{}},{"type":"text","text":"對主分片在該需要removeShard的分片的庫中的所有表啓用分片功能，啓用分片功能後會有chunk信息，mongodb會自動遷移該分片的chunk到其他分片，整個過程可以保證路由信息一致。","attrs":{}}]}]}