mongodb 源码实现系列 - command 命令处理模块源码实现二

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"关于作者","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 前滴滴出行技术专家，现任OPPO文档数据库mongodb负责人，负责oppo千万级峰值TPS/十万亿级数据量文档数据库mongodb内核研发及运维工作，一直专注于分布式缓存、高性能服务端、数据库、中间件等相关研发。后续持续分享《MongoDB内核源码设计、性能优化、最佳运维实践》，Github账号地址:","attrs":{}},{"type":"link","attrs":{"href":"https://github.com/y123456yz","title":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"https://github.com/y123456yz","attrs":{}}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"背景","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"     <>中分享了mongodb内核底层网络IO处理相关实现，包括套接字初始化、一个完整mongodb报文的读取、获取到DB数据发送给客户端等。Mongodb支持多种增、删、改、查、聚合处理、cluster处理等操作，每个操作在内核实现中对应一个command，每个command有不同的功能，mongodb内核如何进行command源码处理将是本文分析的重点","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，mongodb提供了mongostat工具来监控当前集群的各种操作统计。Mongostat监控统计如下图所示：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/05/05182f8de9521d9d4ca86389914aa5db.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 其中，insert、delete、update、query这四项统计比较好理解，分别对应增、删、改、查。但是，comand、getmore不是很好理解，command代表什么统计？getMore代表什么统计？，这两项相对比较难理解。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 此外，通过本文字分析，我们将搞明白这六项统计的具体含义，同时弄清这六项统计由那些操作进行计数。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"1. 《Command命令处理模块一》回顾","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 《Mongodb command命令处理模块源码实现一》中我们分析了一个客户端请求到来后，mognodb服务端大体处理流程如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"①　从message中解析初报文头部，从而确定一个完整的mongodb报文","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"②　从body中解析初OpCode操作码信息，3.6版本默认OpCode操作码为OP_MSG","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"③　根据解析初的OP_MSG操作码，构造对应OpMsg类，真实命令请求以bson数据格式保存在该类成员body中。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"④　从body中解析出command命令字符串信息(如“insert”、“update”等)。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"⑤　从全局_commands map表中查找是否支持该命令，如果支持则执行该命令处理，如果不支持则直接报错提示。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"⑥　最终找到对应command命令后，执行command的功能run接口。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Mongodb内核支持的command命令信息保存在一个全局map表_commands中，从命令请求bson中解析出command命令字符串后，就是从该全局map表查找，如果找到该命令则说明mongodb支持该命令，找不到则说明不支持，整个过程归纳为下图所示：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ef/efc040eb2e9f6a9947b61bc25f6cde82.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 从OpMsg类中解析出命令名字符串后(例如：”insert”、”delete”等)，从全局map表_commands查找，找到则执行对应命令。如果找不到，说明不支持该命令操作，进行异常提示处理。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Mongodb不同实例支持那些command命令完全取决于全局map表_commands，下面继续分析该全局map来源。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"2. Command命令处理模块源码目录结构","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongodb集群中通常包含3种节点实例角色：mongos、mongod(ShardServer)、mongod(ConfigServer)。这3种实例校色功能如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"①　Mongos：代理，从shardServer获取路由信息，转发客户端请求到shard。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"②　mongod(ShardServer)：数据存储节点，所有客户端数据记录到shard中。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"③　mongod(ConfigServer)：记录数据路由信息以及一些元数据。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Mongos代理进程名唯一，也就是”mongos”，代理mongos支持的命令信息比较好确认。但是ShardServer和ConfigServer的进程名都是”mongod”，如何区分各自支持那些命令呢？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" configServer实际上是一种特殊的shardServer，它拥有shard数据分片的功能外，还拥有特殊的元数据管理功能，例如记录chunk元数据信息、mongos信息、分片操作日志信息等。因此，configServer除了支持shardServer的命令外，还会支持更多的特有命令。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongos代理支持的命令信息全部在src/mongo/s/commands目录中实现，源码文件如下:","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e1/e1148c5bef1781bfe1e1d58e5ebe0949.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongod(shardServer)支持的命令信息全部在src/mongo/db/commands目录中实现，源码文件如下：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a0/a049a8cb6a558f373dde269af50f50e2.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongod(configServer)几乎支持所有shardServer支持的命令(说明：也有个别别特例，如”mapreduce.shardedfinish”)，还支持特有的一些命令，这些特意命令在src/mongo/db/s/config目录中实现，源码文件如下：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/74/746f20bfd9c6dc98e8524ebef3322d6e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 从上面的不同实例支持命令的源码目录文件可以看出，mongodb内核源码设计之优秀，从目录结构即可一眼确定不同实例角色支持的各自不同命令信息，代码可读性非常好。目录结构可以总结为下表：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8d/8d7a48875c12b7bd87edf86ebbee0a60.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" configServer和shardServer各自支持的命令范围类似于下图包含与被包含的关系，小椭圆代表shardServer，大圆代表configServer：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/14/1471a6cecd7ae1636d2d007b4aa93a05.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"3. command模块类继承关系","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 第2章节代码目录结构可以看出，绝大部分命令功能由对应源码文件实现，例如find_cmd.cpp源码文件进行find”命令处理。此外，也有部分源码文件，一个文件对应多个命令实现，例如write_commands.cpp源码文件，同时负责”insert”、”update”、”delete”增删改处理。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 由于命令众多，了解了代码目录结构后，在进行核心代码分析前，我们先了解一下command类的各种继承关系。不同命令有不同功能，也就需要不同的实现，但是所有命令也会有一些共同的接口特性，例如该命令是否需要认证、是否支持从节点操作、是否支持WriteConcern操作等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 不同command命令有相同的共性，也会有各自不同的独有特性。所以，mongodb在源码实现中充分考虑了这些问题，抽象出一些共有的特性接口由基类实现，command用于的一些独有的特性，则在继承类中实现。command命令处理模块相关核心源码类主要继承关系图如下：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9d/9dad98780a07c6f1c761b85b7b52f77f.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 如上图，command命令处理模块相关实现类按照父子继承关系可以包含四层，每层功能说明如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"①　CommandInterface类：虚拟接口类，只定义虚拟接口，不做具体实现。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"②　Command类：完成一些基本功能检查，例如是否支持从节点操作、是否需要认证、是否支持WriteConcern、获取命令名、是否只能在admin库操作等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"③　BasicCommand类：认证相关接口实现、定义虚拟run接口。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"④　具体命令类：每个命令都有一个相应的类定义，都是在该层实现，真正的命令run接口实现在该层完成。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4. command命令注册核心代码实现","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 前面分析提到，当解析到对应命令字符串(如：”insert”、”update”等)后，从全局map表中_commands查找，找到说明支持该命令，找不到则不支持。全局_commands表中保存了实例支持的command命令信息，不同命令需要提前注册到该map表中，注册方式有两种：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"①　每个命令定义一个对应全局类变量","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"②　new()一个该命令类信息","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 类注册过程源码实现由command类初始化构造接口完成，注册过程核心代码如下所示：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"cpp"},"content":[{"type":"text","text":"1.//命令注册，所有注册的命令最终全部保存到_commands全局map表中  \n2.//name和oldName实际上是同一个command，只是可能因为历史原因，命令名改名了  \n3.Command::Command(StringData name, StringData oldName)   \n4.    //命令名字符串  \n5.    : _name(name.toString()),   \n6.     //对应命令执行统计，total代表总的，failed代表执行失败的次数  \n7.     _commandsExecutedMetric(\"commands.\" + _name + \".total\", &_commandsExecuted),  \n8.     _commandsFailedMetric(\"commands.\" + _name + \".failed\", &_commandsFailed) {  \n9.    //如果_commands map表还没有生成，则new一个  \n10.    if (_commands == 0)  \n11.        _commands = new CommandMap();  \n12.    ......  \n13.    //把name命令对应的command添加到map表中  \n14.    Command*& c = (*_commands)[name];  \n15.    if (c)  \n16.        log() <;  \n10.    ......  \n11.    //获取命令名  \n12.    const std::string& getName() const final {  \n13.        return _name;  \n14.    }  \n15.    ......  \n16.    //应答保留填充字段长度  \n17.    std::size_t reserveBytesForReply() const override {  \n18.        return 0u;  \n19.    }  \n20.    //该命令是否只能在admin库执行，默认不可以  \n21.    bool adminOnly() const override {  \n22.        return false;  \n23.    }  \n24.    //该命令是否需要权限认证检查？默认不需要  \n25.    bool localHostOnlyIfNoAuth() override {  \n26.        return false;  \n27.    }  \n28.    //该命令执行后是否进行command操作计数  \n29.    bool shouldAffectCommandCounter() const override {  \n30.        return true;  \n31.    }  \n32.    //该命令是否需要认证  \n33.    bool requiresAuth() const override {  \n34.        return true;  \n35.    }  \n36.    //help帮助信息  \n37.    void help(std::stringstream& help) const override;  \n38.    //执行计划信息  \n39.    Status explain(...) const override;  \n40.    //日志信息相关  \n41.    void redactForLogging(mutablebson::Document* cmdObj) override;  \n42.    BSONObj getRedactedCopyForLogging(const BSONObj& cmdObj) override;  \n43.    //该命令是否为maintenance模式，默认false  \n44.    bool maintenanceMode() const override {  \n45.        return false;  \n46.    }  \n47.    //maintenance是否支持，默认支持  \n48.    bool maintenanceOk() const override {  \n49.        return true;   \n50.    }  \n51.    //本地是否支持非本地ReadConcern,默认不支持  \n52.    bool supportsNonLocalReadConcern(...) const override {  \n53.        return false;  \n54.    }  \n55.    //是否允许AfterClusterTime，默认允许  \n56.    bool allowsAfterClusterTime(const BSONObj& cmdObj) const override {  \n57.        return true;  \n58.    }  \n59.    //3.6版本默认opCode=OP_MSG,所以对应逻辑操作op为LogicalOp::opCommand  \n60.    LogicalOp getLogicalOp() const override {  \n61.        return LogicalOp::opCommand;  \n62.    }  \n63.    //例如find就是kRead，update  delete insert就是kWrite，非读写操作就是kCommand  \n64.    ReadWriteType getReadWriteType() const override {  \n65.        return ReadWriteType::kCommand;  \n66.    }  \n67.    //该命令执行成功统计  \n68.    void incrementCommandsExecuted() final {  \n69.        _commandsExecuted.increment();  \n70.    }  \n71.  \n72.    //该命令执行失败统计  \n73.    void incrementCommandsFailed() final {  \n74.        _commandsFailed.increment();  \n75.    }  \n76.  \n77.    //真正得命令运行  \n78.    bool publicRun(OperationContext* opCtx, const OpMsgRequest& request, BSONObjBuilder& result);  \n79.  \n80.    //获取支持的所有命令信息  ListCommandsCmd获取所有支持的命令 db.listCommands()  \n81.    static const CommandMap& allCommands() {  \n82.        return *_commands;  \n83.    }  \n84.  \n85.    //没用  \n86.    static const CommandMap& allCommandsByBestName() {  \n87.        return *_commandsByBestName;  \n88.    }  \n89.  \n90.    //收到不支持命令的统计，例如mongo shell敲一个mongodb无法识别得命令，这里就会统计出来  \n91.    static Counter64 unknownCommands;  \n92.    //根据命令字符串名查找对应命令  \n93.    static Command* findCommand(StringData name);  \n94.    //执行结果  \n95.    static void appendCommandStatus(...);  \n96.    //是否启用了command test功能  \n97.    static bool testCommandsEnabled;  \n98.    //help帮助信息  \n99.    static bool isHelpRequest(const BSONElement& helpElem);  \n100.    static const char kHelpFieldName[];  \n101.    //认证检查，检查是否有执行该命令得权限  \n102.    static Status checkAuthorization(Command* c,  \n103.                                     OperationContext* opCtx,  \n104.                                     const OpMsgRequest& request);  \n105.    ......  \n106.private:  \n107.    //添加地方见Command::Command(    \n108.    //所有的command都在_commands中保存  \n109.    static CommandMap* _commands;  \n110.    //暂时没用  \n111.    static CommandMap* _commandsByBestName;  \n112.    //执行对应命令run接口  \n113.    virtual bool enhancedRun(OperationContext* opCtx,  \n114.                             const OpMsgRequest& request,  \n115.                             BSONObjBuilder& result) = 0;  \n116.    //db.serverStatus().metrics.commands命令查看，本命令的执行统计,包括执行成功和执行失败的  \n117.    Counter64 _commandsExecuted;   \n118.    Counter64 _commandsFailed;  \n119.   //命令名，如\"find\" \"insert\" \"update\" \"createIndexes\" \"deleteIndexes\"  \n120.    const std::string _name;  \n121.  \n122.    //每个命令执行是否成功通过MetricTree管理起来，也就是db.serverStatus().metrics.commands统计信息  \n123.    //通过MetricTree特殊二叉树管理起来  \n124.    ServerStatusMetricField _commandsExecutedMetric;  \n125.    ServerStatusMetricField _commandsFailedMetric;  \n126.};  ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" command作为默认接口类，主要完成一些命令基本接口初始化操作及默认配置设置，该类最基本的接口主要如下：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"reserveBytesForReply","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ReserveBytesForReply()接口主要完成该命令应答填充字段长度，默认值为0。对应命令可以在具体命令类中修改。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"adminOnly","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 该命令是否只能在admin库操作，默认为false。也可以在对应命令继承类中修改，例如\"moveChunk\"命令则在MoveChunkCommand继承类中设置为true，也就是该命令只能在admin库操作。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"localHostOnlyIfNoAuth","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 该命令是否支持在实例所在本机不认证操作，默认值false。对应命令可以在具体继承类中修改。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"l shouldAffectCommandCounter","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 该命令是否需要command统计，也就是mongostat中的command统计计数是否需要使能。默认值true，也就是该命令会进行command计数统计。对应命令可以在具体继承类中修改。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"requiresAuth","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 该命令是否需要认证，默认为true。对应命令可以在具体继承类中修改。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"allowsAfterClusterTime","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 该命令是否支持AfterClusterTime，默认为true。对应命令可以在具体继承类中修改。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"getLogicalOp","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 该命令是否为逻辑opCommand命令。3.6版本默认opCode=OP_MSG,所以对应逻辑操作op为LogicalOp::opCommand。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"getReadWriteType","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 如果为读命令则type对应kRead，写命令type对应kWrite，其他读写以外的命令对应kCommand。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"incrementCommandsExecuted","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 该命令执行成功统计，通过db.serverStatus().metrics.commands获取该命令统计。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"_commandsFailed","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 该命令执行失败统计，通过db.serverStatus().metrics.commands获取该命令统计。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上列举除了command基类的几个核心功能默认值信息，如果继承类中没有修改这些接口值，则该命令对应功能就是这些默认值。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" 说明：","attrs":{}},{"type":"text","text":"各种不同命令如果不适用command基类的默认接口，则可以在继承类中修改对应接口值即可更改对应功能。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 命令除了上面提到的基本功能是否支持外，command类还有其他几个核心接口功能。例如，该命令是否认证成功、是否有操作权限、允许对应run命令等。command类函数接口功能总结如下表所示：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b8/b84e51f5d800e358f33c13c194b22707.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"7. 命令run","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 结合《命令处理模块源码实现一》和本章节对command处理流程可以得出，runCommandImpl接口通过如下调用流程最终执行特定命令的run接口，这里以insert写入和读取流程为例，mongod实例写入调用过程如下图所示:","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/95/955d985bec1064f7cefa226c905c08fe.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 最终，mongod和mongos实例调用相关命令得run接口完成具体的command命令处理操作。mongos、mongod(shardServer)、mongod(configServer)相关常用的操作命令(以最基本的读写命令为例)入口及功能说明总结如下表所示：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/41/41aae387bf0c0228ad76f785ac56e241.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"8. command模块统计信息","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongodb command命令处理模块相关统计包含三类：单个命令统计、汇总型统计、读写时延统计。其中，单个命令统计针对所有接受到的命令名字符串进行统计，汇总型统计则是把同一类型的命令总结为一个整体统计(例如commands统计)。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"8.1单个命令统计","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongodb会收集所有操作命令执行结果，如果本次命令执行成功，则该命令成功统计自增加1，同理如果该命令执行过程失败，则失败统计自增加1，这些统一归类为”单个命令统计信息”。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 单个命令统计由command类的_commandsExecuted和_commandsFailed实现命令执行成功统计和失败统计，相关核心代码实现如下：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"cpp"},"content":[{"type":"text","text":"1.//该命令执行成功统计  \n2.void incrementCommandsExecuted() final {  \n3.    _commandsExecuted.increment();  \n4.}  \n5.  \n6.//该命令执行失败统计  \n7.void incrementCommandsFailed() final {  \n8.    _commandsFailed.increment();  \n9.}  \n\n1.//命令入口  \n2.void execCommandDatabase(...)  \n3.{  \n4.    ......  \n5.    //该命令执行次数统计  db.serverStatus().metrics.commands可以获取统计信息  \n6.    command->incrementCommandsExecuted();  \n7.    ......  \n8.    //真正的命令执行在这里面  \n9.    retval = runCommandImpl(opCtx, command, request, replyBuilder, startOperationTime);  \n10.  \n11.    //该命令失败次数统计  \n12.    if (!retval) {  \n13.        command->incrementCommandsFailed();  \n14.    }  \n15.    ......  \n16.}  ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongodb默认会统计每个客户端发往服务端的命令，即使是无法识别的命令也会统计，命令统计可以通过db.serverStatus().metrics.commands获取，如下图所示：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a3/a392161d98367db2101cb0abe9c0e425.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"8.2汇总型commands命令统计","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 从前面的单个命令统计可以看出，单个命令会记录所有发送给mongodb的命令信息。mongodb支持的命令百余个，由于命令众多，因此mongodb为了更加直观明了的获取统计信息，除了提供单个命令统计外，还对外提供汇总型命令统计。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 汇总型命令统计可以通过db.serverStatus().opcounters命令获取，mongostat中的增删改查等信息也来自于该统计，如下图：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/05/05182f8de9521d9d4ca86389914aa5db.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 从上图可以看出，整个mongostat监控统计可以归类为小表：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b1/b113916ad98106ecab53af67351ee7f0.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" insert、delete、update、find分别对应增删改查四个命令操作，getMore对应批量游标操作命令。这五个命令，对应命令执行的时候统计信息自增，核心代码实现如下：","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"8.2.1 insert操作统计","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" insert操作统计在代理mongos和分片存储节点mongod都会统计，两种角色的insert统计核心代码如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1. 代理mongos insert统计核心代码实现","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"cpp"},"content":[{"type":"text","text":"1. bool insertBatchAndHandleErrors(...) {  \n2.    ......  \n3.    //一次性一条一条插入，上面的固定集合是一次性插入  \n4.    for (auto it = batch.begin(); it != batch.end(); ++it) {  \n5.        //insert操作计数  \n6.        globalOpCounters.gotInsert();   \n7.    }  \n8.    ......  \n9.} ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. 分片存储节点mongod insert统计核心代码实现","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"cpp"},"content":[{"type":"text","text":"1.//mongod代理insert统计核心流程  \n2.bool ClusterWriteCmd::enhancedRun(...) {  \n3.    ......  \n4.    if (_writeType == BatchedCommandRequest::BatchType_Insert) {  \n5.        //insert计数  \n6.     for (size_t i = 0; i shouldAffectCommandCounter()) {  \n5.        OpCounters* opCounters = &globalOpCounters;  \n6.        opCounters->gotCommand();  \n7.    }  \n8.    ......  \n9.}  ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. 存储节点mongod commands统计核心代码实现：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"cpp"},"content":[{"type":"text","text":"1.void execCommandDatabase(...) {  \n2.    ......  \n3.    //是否进行command统计  \n4.    if (command->shouldAffectCommandCounter()) {  \n5.        OpCounters* opCounters = &globalOpCounters;  \n6. //commands计数自增\n7.        opCounters->gotCommand();  \n8.    }  \n9.    ......  \n10.}  ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 从上面的代码可以看出，只有对应命令类中shouldAffectCommandCounter()为true的命令才会进行commands计数。前面章节中我们提到，所有命令都有一个对应类实现相应功能，所有命令实现类都继承一个功能class command {}类，该类对shouldAffectCommandCounter()接口进行初始化。代码实现如下：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"cpp"},"content":[{"type":"text","text":"1.class Command : {  \n2.    ......  \n3.    //该命令是否进行command操作计数，默认需要。如果不需要进行command统计，可在命令继承类中置为false  \n4.    bool shouldAffectCommandCounter() const override {  \n5.        return true;  \n6.    }  \n7.    ......  \n8.}  ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 该接口默认为true，如果对应命令不需要进行commands计数统计，则需要在对应命令实现类中把该接口置为false。通过分析代码，可以看出，只有以下命令子类把shouldAffectCommandCounter()接口设置为false，搜索结果如下:","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/bd/bd462ab5621a43385fda68149bcbedfe.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 分析代码可以得出如下结论：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1)  mongos代理中的clase Cluster","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"find","attrs":{}},{"type":"text","text":"cmd { }类和class Cluster","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"getmore","attrs":{}},{"type":"text","text":"cmd {}类的shouldAffectCommandCounter()接口置为false，这两个类分别对应代理的“find”和“getMore”命令操作，也就说明mongos代理de这两个命令操作不会统计到commands中。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2)  mongod分片存储节点的clase Find","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"cmd{}、class Getmore","attrs":{}},{"type":"text","text":"cmd {}、class Write_commands{}三个类中把shouldAffectCommandCounter()接口置为false。这三个类分别对应mongod存储实例的如下几个命令：“find”、“getMore”、“insert”、“update”、“delete”五个命令。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"   mongos和mongod实例commands统计信息总结如下：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/52/526e7e8e2b5fecadda59edfd08e34d89.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"8.3慢日志、时延统计","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 每次客户端请求执行实践如果超过了log level配置的最大慢日志时间，则会把该操作详细信息记录下来，同时把本操作执行时间添加到对应的读或者写计数及时延统计中。命令处理模块中，时延相关统计包括以下两种统计：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"①　慢日志统计","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"②　读写计数及时延统计","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"8.3.1慢日志统计","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 当启用了慢日志记录功能后，mongod会把执行时间超过指定阀值的慢日志记录下来。慢日志默认记录到服务日志文件(","attrs":{}},{"type":"link","attrs":{"href":"#systemLog.path","title":"systemLog.path"},"content":[{"type":"text","text":"systemLog.path","attrs":{}}]},{"type":"text","text":"配置项设置)，同时会记录日志到”system.profile”集合中。慢日志核心代码实现如下:","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"cpp"},"content":[{"type":"text","text":"1.DbResponse ServiceEntryPointMongod::handleRequest(...) {  \n2.    ......  \n3.    //记录开始时间  \n4.    //获取当前操作对应curop  \n5.    CurOp& currentOp = *CurOp::get(opCtx);  \n6.  \n7.    ......  \n8.    //执行请求对应命令  \n9.    runCommands(opCtx, m);  \n10.    ......  \n11.  \n12.    //记录结束时间  \n13.    currentOp.ensureStarted();  \n14.    currentOp.done();   \n15.    //获取开始和结束时间差，也就是命令执行时间  \n16.    debug.executionTimeMicros = durationCount  \n17.                 (currentOp.elapsedTimeExcludingPauses());  \n18.    //记录超过阀值的慢日志到日志文件  \n19.    if (shouldLogOpDebug || (shouldSample && debug.executionTimeMicros > logThresholdMs * 1000LL)) {  \n20.        ......  \n21.    //记录慢日志到日志文件  \n22.        log() <  \n17.                 (currentOp.elapsedTimeExcludingPauses());  \n18.    ......  \n19.    //记录慢日志到system.profile集合   \n20.    //mongod读写的时间延迟统计  db.serverStatus().opLatencies获取   \n21.    Top::get(opCtx->getServiceContext())  \n22.        .incrementGlobalLatencyStats(   //读写统计\n23.            opCtx,  \n24. //时延 \n25.            durationCount(currentOp.elapsedTimeExcludingPauses()),  \n26.            currentOp.getReadWriteType());  //读写类型\n27.    ......  \n28.}  ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" mongod实例读、写、command操作计数及其各自时延统计可以通过db.serverStatus()接口获取，用户可用采样来计算对应的tps和平均时延信息。获取操作统计和时延统计的命令如下:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" db.serverStatus().opLatencies","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"9. 问题回顾","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    《Mongodb command命令处理模块源码实现一》一文中提到的commands统计信息到这里就可以得到答案了，如下表所示：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"   mongos和mongod实例commands统计信息总结如下：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/52/526e7e8e2b5fecadda59edfd08e34d89.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f8/f8a71989227db75e6e47e6d1b25a2899.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"mongos代理mongostat统计可以汇总为下图所示：","attrs":{}}]}],"attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/45/459c1cb5c532bada32710c819822374c.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"numberedlist","attrs":{"start":2,"normalizeStart":2},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"mongod代理mongostat统计可以汇总为下图所示：","attrs":{}}]}],"attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/22/22e6e893c363ef5f7cd256eea54b0e31.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}