300条数据变更引发的血案-记某十亿级核心mongodb集群部分请求不可用故障踩坑记

原創

2021-05-10 17:48

库名

表名

功能说明

userbucket

whitexxx/expxxx

用户路由信息表，约300条数据。用户访问feeds_xxxxxxx库前必须先获取该表得用户路由数据

feeds_xxxxxxx

feeds_xxx_pool

近亿数据，未启用分片

hardware_xxx_cost

未启用分片，少量数据

news_xxx_profile

数亿数据，未启用分片

resource_xxx_info

数千万数据，启用分片

resource_xxx_info

未启用分片，数亿数据

resource_xxx_info

未启用分片，数亿数据

"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/13/13c9961e254ce1a186fb972aafde53b4.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 上面的描述可以总结为下图：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ec/ec62520c3c6d51daa2af695f06338b73.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 从上图可以看出，分片2和分片3几乎没起到任何作用；由于分片3有两个节点为低IO的sata盘，可能影响userbucket库的读写，因此考虑直接removeShard从集群中剔除分片3和分片2。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"2. 操作过程","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 由于分片3为低IO服务器，有潜在抖动集群抖动分享；同时分片2和分片3几乎都是浪费的分片，因此打散直接通过如下removeshad命令删除分片3和分片2信息，腾出无用服务器资源，如下图所示：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/bb/bb4380ca059fca344585906683f18a6e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"步骤1：登陆任一一个代理，假设是代理mongos1。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"步骤2：由于分片3(也就是shard_8D5370B4分片)为userbucket库的主分片，因此报错了，提示\"you need to drop or movePrimary these databases\"，意思是我们需要提前把该库的主分片信息迁移到其他分片。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"步骤3：通过movePrimary命令把userbucket库的主分片从分片3迁移到分片1。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 登陆监控列表中的其他两个代理mongos2、mongos3，通过如下命令强制刷新路由信息： db.adminCommand({\"flushRouterConfig\":1})","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"注意事项：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" ","attrs":{}},{"type":"text","text":"由于movePrimary过程，其他代理不会感知到该库的主分片变化，因此需要强制刷新路由信息或者重启其他节点的mongos，参考如下：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f6/f6b244ed54796208e4a40b41399cd614.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"3. 用户反馈大部分请求业务请求不可用","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 对含有300条数据的userbucket库变更后，当我还在若无其事的处理其他集群性能调优的时候，用户突然很急的电话我反馈该核心集群整个访问不可用（注意：是整个10亿数据的集群不可用）。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 收到电话后很突然，和业务人员详细对接后可以基本上确定是因为这300条数据变更引起。业务获取这300条数据的时候，部分请求获取成功，部分请求获取失败，说明肯定和movePrimary有关系。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 于是，除了对监控列表中的所有代理做flushRouterConfig强制路由刷新外，还重启了所有的代理，但是业务反馈，还是有部分请求获取不到数据。比较棘手，我自己通过所有的mongos代理查看userbucket库下面的300条数据，完全可以获取到数据。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 于是怀疑是不是还有未刷新路由的mongos代理，于是登陆任一mongos代理获取config.mongos表，查看结果如下：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f4/f4e39477f4bc732bfa1126f233ef9178.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 上面的config.mongos表记录了该集群所有的代理信息，同时记录了这些代理和集群最后一次ping通信的详细时间信息。很明显，该表中记录的代理原不止集群监控列表中的代理个数，比监控列表中的个数要多。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 最终，把config.mongos表中罗列的当前在线的所有代理强制通过flushRouterConfig刷新路由后，业务恢复。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4. 问题总结","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 通过前面的分析可以得出，由于早期集群监控中漏掉了部分代理，造成这部分代理对应的userbucket路由信息是movePrimary前的路由信息，也就是指向了错误的分片，因此出现了路由不到数据的情况，如下图所示：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2e/2eff527973e8342eae12dad44c410573.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"为何用户userbucket库对应表中数据有的成功有的失败？","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 因为部分代理在moveprimary后，没有强制刷新该表路由信息，造成部分代理路由获取数据的时候路由错误。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"为何该300条数据部分路由信息错误会造成整个10亿集群部分访问不可用？","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"和业务实现逻辑有关系，因为业务在获取这10亿条数据前首先需要获取业务的路由信息，刚好业务路由信息存在了userbucket库对应表中，业务在获取数据前必须要获取到业务的路由信息数据，如果userbucket数据获取不到，用户就无法确定指向feeds_xxxxxxx数据。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"为何会遗漏部分代理重启或者强制路由刷新？","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"历史原因，造成部分代理业务代码有配置，但是服务端集群监控元数据遗漏了，也就是服务端集群监控漏掉了部分代理，这部分代理没有监控起来。也有可能是mongos代理扩容，但是集群监控列表中没有加入元数据。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"movePrimary操作最安全的操作方法？","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 官方建议movePrimary操作成功后需要强制路由刷新或者重启mongos，但是movePrimary操作成功和mongos重启这个过程中有个中间状态，如果中间状态业务读或者些该迁移的库下面的表，还是可能路由错误。因此，最佳安全的moveprimary可以通过如下两个方法操作：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" 法一：","attrs":{}},{"type":"text","text":"shutdown所有代理，只留下一个代理，等该代理moveprimary成功后在重启其他mongos代理。切记别遗漏代理，出现本文踩坑类似情况，提前检查config.mongos表。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" 法二：","attrs":{}},{"type":"text","text":"对主分片在该需要removeShard的分片的库中的所有表启用分片功能，启用分片功能后会有chunk信息，mongodb会自动迁移该分片的chunk到其他分片，整个过程可以保证路由信息一致。","attrs":{}}]}]}