mongodb內核源碼實現、性能調優、最佳運維實踐系列-百萬級代碼量mongodb內核源碼閱讀經驗分享

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 開源mongodb代碼規模數百萬行，本篇文章內容主要引導大家快速進行百萬級別規模源碼閱讀，做到不同大工程源碼”舉一反三”快速閱讀的目的。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"前言"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Mongodb內核源碼由第三方庫third"},{"type":"text","marks":[{"type":"italic"}],"text":"party和mongodb服務層源碼組成，其中mongodb服務層代碼在不同模塊實現中依賴不同的third"},{"type":"text","text":"party庫，第三方庫是mongodb服務層代碼實現的基礎(例如:網絡底層IO實現依賴asio-master庫,底層存儲依賴wiredtiger存儲引擎庫)，其中第三方庫也會依賴部分其他庫(例如：wiredtiger庫依賴snappy算法庫，asio-master依賴boost庫)。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 雖然Mongodb內核源碼數百萬行，工程量巨大，但是mongodb服務層代碼實現層次非常清晰，代碼目錄結構、類命名、函數命名、文件名命名都非常一目瞭然，充分體現了10gen團隊的專業精神。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"說明：mongodb內核除第三方庫third_party外的代碼，這裏統稱爲mongodb服務層代碼。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 本文以mongodb服務層transport實現爲例來說明如何快速閱讀整個mongodb代碼，我們在走讀代碼前，建議遵循如下準則："}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"關於作者"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 前滴滴出行技術專家，現任OPPO文檔數據庫mongodb負責人，負責oppo千萬級峯值TPS/十萬億級數據量文檔數據庫mongodb研發和運維工作，一直專注於分佈式緩存、高性能服務端、數據庫、中間件等相關研發。後續持續分享《MongoDB內核源碼設計、性能優化、最佳運維實踐》，Github賬號地址:"},{"type":"link","attrs":{"href":"https://github.com/y123456yz","title":null},"content":[{"type":"text","marks":[{"type":"underline"}],"text":"https://github.com/y123456yz"}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"1. 熟悉mongodb基本功能和使用方法"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 首先，我們需要熟悉mongodb的基本功能，明白mongodb是做什麼用的，用在什麼地方，這樣才能體現mongodb的真正價值。此外，我們需要提前搭建一個mongodb集羣玩一玩，這樣也可以進一步促使我們瞭解mongodb內部的一些常用基本功能。千萬不要急於求成，如果連mongodb是做什麼的都不知道，或者連mongodb的運維操作方法都沒玩過，直接讀取代碼會非常不適合，沒有目的的走讀代碼不利於分析整個代碼，同時閱讀代碼過程會非常痛苦。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"2. 下載代碼編譯源碼"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 熟悉了mongodb的基本功能，並搭建集羣簡單體驗後，我們就可以從github下載源碼，自己編譯源碼生成二進制文件，編譯文檔存放於docs/building.md代碼目錄中，源碼編譯步驟如下:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1. 下載對應releases中對應版本的源碼"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. 進入對於目錄，參考docs/building.md文件內容進行相關依賴工具安裝"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3. 執行buildscripts/scons.py編譯出對應二進制文件，也可以直接scons mongod mongos這樣編譯。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4. 編譯成功後的生產可執行文件存放於./build/opt/mongo/目錄"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 在正在編譯代碼並運行的過程中，發現以下兩個問題："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1. 編譯出的二進制文件佔用空間很大，如下圖所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/dd/dd7050dea27bac20b42b511914dc3999.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 從上圖可以看出，通過strip處理工具處理後，二進制文件大小已經和官方二進制包大小一樣了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. 在一些低版本操作系統運行的時候出錯，找不到對應stdlib庫，如下圖所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4e/4ee1b2aff38b3248d493d96326180433.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/fb/fb91c39c68fa9bd4a9d45ca9e81f5661.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 如上圖所示，當編譯出的二進制文件拷貝到線上運行後，發現無法運行，提示libstdc庫找不到。原因是我們編譯代碼時候依賴的stdc庫版本比其他操作系統上面的stdc庫版本更高，造成了不兼容。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 解決辦法："},{"type":"text","text":"編譯的時候編譯腳本中帶上-static-libstdc++，把stdc庫通過靜態庫的方式進行編譯，而不是通過動態庫方式。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"3. 瞭解代碼日誌模塊使用方法，試着加打印調試"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 由於前期我們對代碼整體實現不熟悉，不知道各個接口的調用流程，這時候就可以通過加日誌打印進行調試。Mongodb的日誌模塊設計的比較完善，從日誌中可以很明確的看出由那個功能模塊打印日誌，同時日誌模塊有多種打印級別。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.日誌打印級別設置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 啓動參數中verbose設置日誌打印級別，日誌打印級別設置方法如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Mongod -f ./mongo.conf -vvvv   "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 這裏的v越多，表明日誌打印級別設置的越低，也就會打印更多的日誌。一個v表示只會輸出LOG(1)日誌，-vv表示LOG(1) LOG(2)都會寫日誌。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. 如何在.cpp文件中使用日誌模塊記錄日誌"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    如果需要在一個新的.cpp文件中使用日誌模塊打印日誌，需要進行如下步驟操作："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"i) 添加宏定義"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" #define MONGO_LOG_DEFAULT_COMPONENT ::mongo::logger::LogComponent::kExecutor"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ii) 使用LOG(N)或者log()來記錄想要輸出的日誌內容，其中LOG(N)的N代表日誌打印級別，log()對應的日誌全記錄到文件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 例如:"},{"type":"text","text":" LogComponent::kExecutor代表executor模塊相關的日誌，參考log_component.cpp日誌模塊文件實現，對應到日誌文件內容如下："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/02/02581e8a108f782872df10a69ddd5af2.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4. 學會用gdb調試mongodb代碼"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Gdb是linux系統環境下優秀的代碼調試工具，支持設置斷點、單步調試、打印變量信息、獲取函數調用棧信息等功能。gdb工具可以綁定某個線程進行線程級調試，由於mongodb是多線程環境，因此在用gdb調試前，我們需要確定調試的線程號，mongod進程包含的線程號及其對應線程名查看方法如下:"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/dd/dd22cd0999535760f2008d20584ffd19.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 注意："},{"type":"text","text":"在調試mongod工作線程處理流程的時候，不要選擇adaptive動態線程池模式，因爲線程可能因爲流量低引起工作線程不飽和而被銷燬，從而造成調試過程因爲線程銷燬而中斷，synchronous線程模式是一個鏈接一個線程，只要我們不關閉這個鏈接，線程就會一直存在，不會影響我們理解mongodb服務層代碼實現邏輯。synchronous線程模式調試的時候可以通過mongo shell鏈接mongod服務端端口來模擬一個鏈接，因此調試過程相對比較可控。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 在對工作線程調試的時候，發現gdb無法查找到mongod進程的符號表，無法進行各種gdb功能調試，如下圖所示："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c1/c10949dcebd8662036c7e375787fbe98.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 上述gdb無法attach到指定線程調試的原因是無法加載二進制文件符號表，這是因爲編譯的時候沒有加上-g選項引起，mongodb通過SConstruct腳本來進行scons編譯，要啓用gdb功能需要在scons編譯代碼的時候指定gdbserver選項:scons --gdbserver=GDBSERVER -j 2。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 編譯出新的二進制文件後，就可以gdb調試了，如下圖所示，可以很方便的定位到某個函數之前的調用棧信息，並進行單步、打印變量信息等調試："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/35/35f8d1c46fb9dfffc916f31706eb09dd.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"5. 熟悉代碼目錄結構、模塊細化拆分"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 在進行代碼閱讀前還有很重要的一步就是熟悉代碼目錄及文件命名實現，mongodb服務層代碼目錄結構及文件命名都有很嚴格的規範。下面以truansport網絡傳輸模塊爲例，transport模塊的具體目錄文件結構："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ec/ecdd0371895439971d92e650929c4a15.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 從上面的文件分佈內容，可以清晰的看出，整個目錄中的源碼實現文件大體可以分爲如下幾個部分："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1. message"},{"type":"text","marks":[{"type":"italic"}],"text":"compressor"},{"type":"text","text":"*網絡傳輸數據壓縮子模塊"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. service"},{"type":"text","marks":[{"type":"italic"}],"text":"entry"},{"type":"text","text":"point*服務入口點子模塊"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3. service_executor*服務運行子模塊，即線程模型子模塊"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4. service"},{"type":"text","marks":[{"type":"italic"}],"text":"state"},{"type":"text","text":"machine*服務狀態機處理子模塊"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5. Session*回話信息子模塊"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"6. Ticket*數據分發子模塊"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"7. transport_layer*套接字處理及傳輸層模式管理子模塊"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b0/b063b538ec3dea4af0a5633cb0941431.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 通過上面的拆分，整個大的transport模塊實現就被拆分成了7個小模塊，這7個小的子模塊各自負責對應功能實現，同時各個模塊相互銜接，整體實現網絡傳輸處理過程的整體實現，下面的章節將就這些子模塊進行簡單功能說明。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"6. 從main入口開始大體走讀代碼"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 前面5個步驟過後，我們已經熟悉了mongodb編譯調試以及transport模塊的各個子模塊的相關代碼文件實現及大體子模塊作用。至此，我們可以開始走讀代碼了，mongos和mongod的代碼入口分別在mongoSMain()和mongoDbMain()，從這兩個入口就可以一步一步瞭解mongodb服務層代碼的整體實現。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" 注意："},{"type":"text","text":"走讀代碼前期不要深入各種細節實現，大體瞭解代碼實現即可，先大體弄明白代碼中各個模塊功能由那些子模塊實現，千萬不要深究細節。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"7. 總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本章節主要給出了數百萬級mongodb內核代碼閱讀的一些建議，整個過程可以總結爲如下幾點："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1. 提前瞭解mongodb的作用及工作原理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. 自己搭建集羣提前學習下mongodb集羣的常用運維操作，可以進一步幫助理解mongodb的功能特性，提升後期代碼閱讀的效率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3. 自己下載源碼編譯二進制可執行文件，同時學會使用日誌模塊，通過加日誌打印的方式逐步開始調試。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4. 學習使用gdb代碼調試工具調試線程的運行流程，這樣可以更進一步的促使快速學習代碼處理流程，特別是一些複雜邏輯，可以大大提升走讀代碼的效率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5. 正式走讀代碼前，提前瞭解各個模塊的代碼目錄結構，把一個大模塊拆分成各個小模塊，先大體瀏覽各個模塊的代碼實現。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"6. 前期走讀代碼千萬不要深入細節，捋清楚各個模塊的大體功能作用後再開始一步一步的深入細節，瞭解深層次的內部實現。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"7. 從main()入口逐步開始走讀代碼，結合log日誌打印和gdb調試。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"跳過整體流程中不熟悉的模塊代碼，只走讀本次想弄明白的模塊代碼實現。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}