mongodb内核源码实现、性能调优、最佳运维实践系列-百万级代码量mongodb内核源码阅读经验分享

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 开源mongodb代码规模数百万行，本篇文章内容主要引导大家快速进行百万级别规模源码阅读，做到不同大工程源码”举一反三”快速阅读的目的。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"前言"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Mongodb内核源码由第三方库third"},{"type":"text","marks":[{"type":"italic"}],"text":"party和mongodb服务层源码组成，其中mongodb服务层代码在不同模块实现中依赖不同的third"},{"type":"text","text":"party库，第三方库是mongodb服务层代码实现的基础(例如:网络底层IO实现依赖asio-master库,底层存储依赖wiredtiger存储引擎库)，其中第三方库也会依赖部分其他库(例如：wiredtiger库依赖snappy算法库，asio-master依赖boost库)。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 虽然Mongodb内核源码数百万行，工程量巨大，但是mongodb服务层代码实现层次非常清晰，代码目录结构、类命名、函数命名、文件名命名都非常一目了然，充分体现了10gen团队的专业精神。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"说明：mongodb内核除第三方库third_party外的代码，这里统称为mongodb服务层代码。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 本文以mongodb服务层transport实现为例来说明如何快速阅读整个mongodb代码，我们在走读代码前，建议遵循如下准则："}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"关于作者"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 前滴滴出行技术专家，现任OPPO文档数据库mongodb负责人，负责oppo千万级峰值TPS/十万亿级数据量文档数据库mongodb研发和运维工作，一直专注于分布式缓存、高性能服务端、数据库、中间件等相关研发。后续持续分享《MongoDB内核源码设计、性能优化、最佳运维实践》，Github账号地址:"},{"type":"link","attrs":{"href":"https://github.com/y123456yz","title":null},"content":[{"type":"text","marks":[{"type":"underline"}],"text":"https://github.com/y123456yz"}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"1. 熟悉mongodb基本功能和使用方法"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 首先，我们需要熟悉mongodb的基本功能，明白mongodb是做什么用的，用在什么地方，这样才能体现mongodb的真正价值。此外，我们需要提前搭建一个mongodb集群玩一玩，这样也可以进一步促使我们了解mongodb内部的一些常用基本功能。千万不要急于求成，如果连mongodb是做什么的都不知道，或者连mongodb的运维操作方法都没玩过，直接读取代码会非常不适合，没有目的的走读代码不利于分析整个代码，同时阅读代码过程会非常痛苦。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"2. 下载代码编译源码"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 熟悉了mongodb的基本功能，并搭建集群简单体验后，我们就可以从github下载源码，自己编译源码生成二进制文件，编译文档存放于docs/building.md代码目录中，源码编译步骤如下:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1. 下载对应releases中对应版本的源码"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. 进入对于目录，参考docs/building.md文件内容进行相关依赖工具安装"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3. 执行buildscripts/scons.py编译出对应二进制文件，也可以直接scons mongod mongos这样编译。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4. 编译成功后的生产可执行文件存放于./build/opt/mongo/目录"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 在正在编译代码并运行的过程中，发现以下两个问题："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1. 编译出的二进制文件占用空间很大，如下图所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/dd/dd7050dea27bac20b42b511914dc3999.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 从上图可以看出，通过strip处理工具处理后，二进制文件大小已经和官方二进制包大小一样了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. 在一些低版本操作系统运行的时候出错，找不到对应stdlib库，如下图所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4e/4ee1b2aff38b3248d493d96326180433.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/fb/fb91c39c68fa9bd4a9d45ca9e81f5661.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 如上图所示，当编译出的二进制文件拷贝到线上运行后，发现无法运行，提示libstdc库找不到。原因是我们编译代码时候依赖的stdc库版本比其他操作系统上面的stdc库版本更高，造成了不兼容。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 解决办法："},{"type":"text","text":"编译的时候编译脚本中带上-static-libstdc++，把stdc库通过静态库的方式进行编译，而不是通过动态库方式。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"3. 了解代码日志模块使用方法，试着加打印调试"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 由于前期我们对代码整体实现不熟悉，不知道各个接口的调用流程，这时候就可以通过加日志打印进行调试。Mongodb的日志模块设计的比较完善，从日志中可以很明确的看出由那个功能模块打印日志，同时日志模块有多种打印级别。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.日志打印级别设置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 启动参数中verbose设置日志打印级别，日志打印级别设置方法如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Mongod -f ./mongo.conf -vvvv   "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 这里的v越多，表明日志打印级别设置的越低，也就会打印更多的日志。一个v表示只会输出LOG(1)日志，-vv表示LOG(1) LOG(2)都会写日志。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. 如何在.cpp文件中使用日志模块记录日志"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    如果需要在一个新的.cpp文件中使用日志模块打印日志，需要进行如下步骤操作："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"i) 添加宏定义"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" #define MONGO_LOG_DEFAULT_COMPONENT ::mongo::logger::LogComponent::kExecutor"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ii) 使用LOG(N)或者log()来记录想要输出的日志内容，其中LOG(N)的N代表日志打印级别，log()对应的日志全记录到文件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 例如:"},{"type":"text","text":" LogComponent::kExecutor代表executor模块相关的日志，参考log_component.cpp日志模块文件实现，对应到日志文件内容如下："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/02/02581e8a108f782872df10a69ddd5af2.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4. 学会用gdb调试mongodb代码"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Gdb是linux系统环境下优秀的代码调试工具，支持设置断点、单步调试、打印变量信息、获取函数调用栈信息等功能。gdb工具可以绑定某个线程进行线程级调试，由于mongodb是多线程环境，因此在用gdb调试前，我们需要确定调试的线程号，mongod进程包含的线程号及其对应线程名查看方法如下:"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/dd/dd22cd0999535760f2008d20584ffd19.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 注意："},{"type":"text","text":"在调试mongod工作线程处理流程的时候，不要选择adaptive动态线程池模式，因为线程可能因为流量低引起工作线程不饱和而被销毁，从而造成调试过程因为线程销毁而中断，synchronous线程模式是一个链接一个线程，只要我们不关闭这个链接，线程就会一直存在，不会影响我们理解mongodb服务层代码实现逻辑。synchronous线程模式调试的时候可以通过mongo shell链接mongod服务端端口来模拟一个链接，因此调试过程相对比较可控。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 在对工作线程调试的时候，发现gdb无法查找到mongod进程的符号表，无法进行各种gdb功能调试，如下图所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c1/c10949dcebd8662036c7e375787fbe98.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 上述gdb无法attach到指定线程调试的原因是无法加载二进制文件符号表，这是因为编译的时候没有加上-g选项引起，mongodb通过SConstruct脚本来进行scons编译，要启用gdb功能需要在scons编译代码的时候指定gdbserver选项:scons --gdbserver=GDBSERVER -j 2。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 编译出新的二进制文件后，就可以gdb调试了，如下图所示，可以很方便的定位到某个函数之前的调用栈信息，并进行单步、打印变量信息等调试："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/35/35f8d1c46fb9dfffc916f31706eb09dd.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"5. 熟悉代码目录结构、模块细化拆分"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 在进行代码阅读前还有很重要的一步就是熟悉代码目录及文件命名实现，mongodb服务层代码目录结构及文件命名都有很严格的规范。下面以truansport网络传输模块为例，transport模块的具体目录文件结构："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ec/ecdd0371895439971d92e650929c4a15.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 从上面的文件分布内容，可以清晰的看出，整个目录中的源码实现文件大体可以分为如下几个部分："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1. message"},{"type":"text","marks":[{"type":"italic"}],"text":"compressor"},{"type":"text","text":"*网络传输数据压缩子模块"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. service"},{"type":"text","marks":[{"type":"italic"}],"text":"entry"},{"type":"text","text":"point*服务入口点子模块"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3. service_executor*服务运行子模块，即线程模型子模块"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4. service"},{"type":"text","marks":[{"type":"italic"}],"text":"state"},{"type":"text","text":"machine*服务状态机处理子模块"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5. Session*回话信息子模块"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"6. Ticket*数据分发子模块"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"7. transport_layer*套接字处理及传输层模式管理子模块"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b0/b063b538ec3dea4af0a5633cb0941431.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 通过上面的拆分，整个大的transport模块实现就被拆分成了7个小模块，这7个小的子模块各自负责对应功能实现，同时各个模块相互衔接，整体实现网络传输处理过程的整体实现，下面的章节将就这些子模块进行简单功能说明。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"6. 从main入口开始大体走读代码"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 前面5个步骤过后，我们已经熟悉了mongodb编译调试以及transport模块的各个子模块的相关代码文件实现及大体子模块作用。至此，我们可以开始走读代码了，mongos和mongod的代码入口分别在mongoSMain()和mongoDbMain()，从这两个入口就可以一步一步了解mongodb服务层代码的整体实现。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 注意："},{"type":"text","text":"走读代码前期不要深入各种细节实现，大体了解代码实现即可，先大体弄明白代码中各个模块功能由那些子模块实现，千万不要深究细节。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"7. 总结"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本章节主要给出了数百万级mongodb内核代码阅读的一些建议，整个过程可以总结为如下几点："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1. 提前了解mongodb的作用及工作原理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. 自己搭建集群提前学习下mongodb集群的常用运维操作，可以进一步帮助理解mongodb的功能特性，提升后期代码阅读的效率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3. 自己下载源码编译二进制可执行文件，同时学会使用日志模块，通过加日志打印的方式逐步开始调试。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4. 学习使用gdb代码调试工具调试线程的运行流程，这样可以更进一步的促使快速学习代码处理流程，特别是一些复杂逻辑，可以大大提升走读代码的效率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5. 正式走读代码前，提前了解各个模块的代码目录结构，把一个大模块拆分成各个小模块，先大体浏览各个模块的代码实现。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"6. 前期走读代码千万不要深入细节，捋清楚各个模块的大体功能作用后再开始一步一步的深入细节，了解深层次的内部实现。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"7. 从main()入口逐步开始走读代码，结合log日志打印和gdb调试。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"跳过整体流程中不熟悉的模块代码，只走读本次想弄明白的模块代码实现。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}