7張圖揭曉RocketMQ存儲設計的奧妙

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RocketMQ作爲一款基於磁盤存儲的中間件，具有無限積壓能力，並提供高吞吐、低延遲的服務能力，其最核心的部分必然是它優雅的存儲設計。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1、存儲概述","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RocketMQ存儲的文件主要包括Commitlog文件、ConsumeQueue文件、Index文件。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RocketMQ將所有主題的消息存儲在同一個文件中，確保消息發送時按順序寫文件，盡最大能力確保消息發送的高可用性與高吞吐量。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但消息中間件一般都是基於主題的訂閱與發佈模式，消息消費時必須按照主題進行帥選消息，顯然從Commitlog文件中按照topic去篩選消息會變得及其低效，爲了提高根據主題檢索消息的效率，RocketMQ引入了ConsumeQueue文件，俗成消費隊列文件。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關係型數據庫可以按照字段屬性進行記錄檢索，作爲一款主要面向業務開發的消息中間件，RocketMQ也提供了基於消息屬性的檢索能力，底層的核心設計理念是爲Commitlog文件建立","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"哈希索引，並存儲在Index文件中","attrs":{}},{"type":"text","text":"。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在RocketMQ中順序寫入到Commitlog文件後，ConsumeQueue與Index文件都是異步構建的，其數據流向圖如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c9/c90fb2f9b17c09625f2258a843a04e4c.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2、存儲文件組織方式","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RocketMQ在消息寫入過程中追求極致的磁盤順序寫。所有主題的消息全部寫入一個文件，即Commitlog文件。所有消息按抵達順序依次追加到文件中，消息一旦寫入，不支持修改。Commitlog文件的具體佈局如下圖所示：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f6/f6038834a88ca0b8cf9c4a304a4be206.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於文件編程與基於內存編程有一個很大的不同是在基於內存的編程模式中我們有現成的數據結構，例如 List、HashMap，對數據的讀寫非常方便，那麼一條一條消息存入文件Commitlog後，該如何查找呢？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"正如關係型數據會爲每一條數據引入一個","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"ID字段","attrs":{}},{"type":"text","text":"，在基於文件編程的模型中，也會爲一條消息引入一個身份標誌：","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"消息物理偏移量","attrs":{}},{"type":"text","text":"，即消息存儲在文件的起始位置。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"正是有了物理偏移量的概念，Commitlog的文件名命名也是極具技巧性，使用了存儲在該文件的第一條消息在整個Commitlog文件組中的偏移量來命名，例如第一個 Commitlog文件爲 0000000000000000000，第二個文件爲00000000001073741824，然後依次類推。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣做的好處是給出任意一個消息的物理偏移量，例如消息偏移量爲 73741824，可以通過二分法進行查找，快速定位這個文件在第一個文件中，然後用消息的物理偏移量減去該文件的名稱所得到的差值，就是在該文件中的絕對地址。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Commitlog文件的設計理念是追求極致的消息寫，但我們知道消息消費模型是基於主題的訂閱機制，即一個消費組是消費特定主題的消息。如果根據主題從commitlog文件中檢索消息，我們會發現這絕不是一個好主意，只能從文件的第一條消息逐條檢索，其性能可想而知，故爲了解決基於topic的消息檢索問題，RocketMQ引入了consumequeue文件，consumequeue的結構如下圖所示。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c8/c8206e7b53f63679a4d6dd5ffa1a3b1b.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ConsumeQueue文件是消息消費隊列文件，是Commitlog文件","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"基於Topic的索引文件","attrs":{}},{"type":"text","text":"，主要用於消費者根據Topic消費消息，其組織方式爲/topic/queue，同一個隊列中存在多個文件。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Consumequeue的設計極具技巧，每個條目長度固定（8字節commitlog物理偏移量、4字節消息長度、8字節tag hashcode）。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏不是存儲tag的原始字符串，而選擇存儲hashcode，","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"目的就是確保每個條目的長度固定，可以使用訪問類似數組下標的方式快速定位條目，極大地提高了ConsumeQueue文件的讀取性能。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"試想一下，消息消費者根據topic、消息消費進度（consumeuqe邏輯偏移量），即第幾個Consumeque條目，這樣的消費進度去訪問消息的方法爲使用邏輯偏移量logicOffset * 20即可找到該條目的起始偏移量（consumequeue文件中的偏移量），然後讀取該偏移量後20個字節即得到一個條目，無須遍歷consumequeue文件。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RocketMQ與Kafka相比具有一個強大的優勢，就是支持按消息屬性檢索消息，引入consumequeue文件解決了基於topic查找的問題，但如果想基於消息的某一個屬性查找消息，consumequeue文件就無能爲力了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RocketMQ引入了Index索引文件，實現","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"基於文件的哈希索引","attrs":{}},{"type":"text","text":"。IndexFile的文件存儲結構如下圖所示：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/73/7393a78e29c67899f8f28b994b4364d8.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"IndexFile文件基於物理磁盤文件實現Hash索引。其文件由40字節的文件頭、500萬個哈希槽，每個哈希槽4個字節，最後由2000萬個Index條目，每個條目由20個字節構成，分別爲4字節索引key的hashcode、8字節消息物理偏移量、4字節時間戳、4字節的前一個Index條目（哈希衝突的鏈表結構）。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"即建立了索引Key的hashcode與物理偏移量的映射關係","attrs":{}},{"type":"text","text":"，根據key先快速定義到commitlog文件，","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"關於Hash索引具體到工作機制，可以參考筆直《RocketMQ技術內幕》第二版4.5.3節的詳細介紹","attrs":{}},{"type":"text","text":"。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3、順序寫","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於磁盤的讀寫，提高其寫入性能的另外一個設計原理是","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"磁盤順序寫","attrs":{}},{"type":"text","text":"。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"磁盤順序寫廣泛用在基於文件的存儲模型中，大家不妨思考一下 MySQL Redo 日誌的引入目的，我們知道在 MySQL InnoDB 的存儲引擎中，會有一個內存 Pool，用來緩存磁盤的文件塊，當更新語句將數據修改後，會首先在內存中進行修改，然後將變更寫入到 redo 文件(刷寫到磁盤)，然後定時將InnoDB內存池中的數據刷寫到磁盤。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f9/f9655f5fafe22a72db498cadcbbcdc84.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲什麼不一有數據變更，就直接更新到指定的數據文件中呢？以MySQL InnoDB中一個庫存在上千張，每一個張的數據會使用單獨的文件存儲，如果每一個表的數據發生變更，就刷寫到磁盤，就會存在大量的隨機寫入，性能無法得到提升，故引入一個redo文件，順序寫redo文件，從表面上多了一步刷盤操作，但由於是順序寫，相比隨機寫，帶來的性能提升是非常顯著的。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4、內存映射機制","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"雖然基於磁盤的順序寫可以極大提高IO的寫效率，但如果基於文件的存儲採用常規的JAVA文件操作API，例如 FileOutputStream等，其性能提升會很有限，RocketMQ引入了內存映射，將磁盤文件映射到內存中，以操作內存的方式操作磁盤，性能又提升了一個檔次。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在JAVA中可通過FileChannel的map方法創建內存映射文件。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Linux服務器中由該方法創建的文件使用的就是操作系統的pagecache，即頁緩存。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Linux操作系統中的內存使用策略時會盡可能地利用機器的物理內存，並常駐內存中，就是所謂的頁緩存。在操作系統的內存不夠的情況下，採用緩存置換算法，例如LRU將不常用的頁緩存回收，即操作系統會自動管理這部分內存。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果RocketMQ Broker進程異常退出，存儲在頁緩存中的數據並不會丟失，操作系統會定時將頁緩存中的數據持久化到磁盤，做到數據安全可靠。不過如果是機器斷電等異常情況，存儲在頁緩存中的數據就有可能丟失。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"5、靈活多變的刷盤策略","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有了","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"順序寫和內存映射","attrs":{}},{"type":"text","text":"的加持，RocketMQ的寫入性能得到了極大的保證，但凡事都有利弊，引入了內存映射和頁緩存機制，消息會先寫入到頁緩存，此時消息並沒有真正持久化到磁盤。那麼broker收到客戶端的消息發送後，是存儲到頁緩存中就直接返回成功，還是要持久化到磁盤中才返回成功呢？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這是一個“艱難”的抉擇，是在性能與消息可靠性方面進行權衡。爲此，RocketMQ提供了多種策略：同步刷盤、異步刷盤。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"5.1 同步刷盤","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同步刷盤在RocketMQ的實現中成爲","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"組提交","attrs":{}},{"type":"text","text":"，並不是每一條消息都必須刷盤。其設計理念如圖所示：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f6/f6e6de5efcc52f37f18d2189509b23d1.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"採用同步刷盤，每一個線程將數據追到到內存後，並向刷盤線程提交刷盤請求，然後會阻塞；刷盤線程從任務隊列中獲取一個任務，","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"然後觸發一次刷盤，但並不只刷與請求相關的消息，而是會直接將內存中待刷盤的所有消息一次批量刷盤","attrs":{}},{"type":"text","text":"，然後就可以","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"喚醒一組請求線程，實現組刷盤","attrs":{}},{"type":"text","text":"。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"5.2 異步刷盤","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同步刷盤的優點是能保證消息不丟失，即向客戶端返回成功就代表這條消息已被持久化到磁盤，即消息非常可靠，但這是以犧牲寫入響應延遲性能爲代價的，由於RocketMQ的消息是先寫入 pagecache，故消息丟失的可能性較小，如果能容忍一定機率的消息丟失，可以考慮使用異步刷盤。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"異步刷盤指的是broker將消息存儲到pagecache後就立即返回成功，然後開啓一個異步線程定時執行FileChannel的forece方法，將內存中的數據定時刷寫到磁盤，默認間隔爲500ms。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"6、內存級讀寫分離","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RocketMQ爲了降低pagecache的使用壓力引入了transientStorePoolEnable機制，即內存級別的讀寫分離機制。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"默認情況下RocketMQ將消息寫入pagecache，消息消費時從pagecache中讀取，這樣在高併發時pagecache的壓力會比較大，容易出現瞬時","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"broker busy","attrs":{}},{"type":"text","text":"，故RocketMQ還引入了transientStorePoolEnable，將消息先寫入堆外內存並立即返回，然後異步將堆外內存中的數據提交到pagecache，再異步刷盤到磁盤中。其工作機制如下圖所示：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4d/4d09a010514f7e95305893a02c341f8d.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息在消費讀取時不會嘗試從堆外內存中讀，而是從pagecache中讀取，這樣就形成了","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"內存級別的讀寫分離","attrs":{}},{"type":"text","text":"，即消息寫入時主要面對堆外內存，而讀消息時主要面對pagecache。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該方案的優點是消息是直接寫入堆外內存，然後異步寫入pagecache。相比每條消息追加直接寫入pagechae，其最大的優勢是將消息寫入pagecache操作批量化。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該方案的缺點是如果由於某些意外操作導致Broker進程異常退出，那麼存儲在堆外內存的數據會丟失，但如果是放入pagecache，broker異常退出並不會丟失消息。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"文章首發於","attrs":{}},{"type":"link","attrs":{"href":"https://www.codingw.net/posts/3364a7c5.html","title":"","type":null},"content":[{"type":"text","text":"https://www.codingw.net/posts/3364a7c5.html","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作者簡介：丁威，《RocketMQ 技術內幕》一書作者、RocketMQ 開源社區優秀佈道師，公衆號「中間件興趣圈」維護者，主打成體系剖析 Java 主流中間件，已發佈 Kafka、RocketMQ、Dubbo、Sentinel、Canal、ElasticJob 等中間件 15 個專欄。","attrs":{}}]}]}