騰訊宣佈開源 RoP：Apache Pulsar 支持原生 RocketMQ 協議

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RocketMQ 用戶可以無縫遷移到 Apache Pulsar 了。自此，Apache Pulsar 補齊了兼容主流消息隊列協議的能力。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們很高興地宣佈騰訊雲中間件開源 RoP！RoP 將 RocketMQ 協議處理插件引入 Pulsar broker，這樣 Pulsar 就能支持原生 RocketMQ 協議了。本次 RoP 開源，感謝合作伙伴 StreamNative 在架構設計與實現方案中予以的支持，在後續騰訊雲中間件也將聯合 StreamNative 共同維護本項目。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"什麼是 RoP？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與 KoP、MoP 和 AoP 相似，RoP 是一種可插拔的協議處理插件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將 RoP 協議處理插件添加到現有 Pulsar 集羣后，用戶無需修改代碼，便能將現有的 RocketMQ 應用程序和服務遷移到 Pulsar，同時還能使用 Pulsar 的強大功能，例如："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"計算與存儲分離"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"多租戶"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"跨地域複製"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分層分片"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"輕量化計算框架 – Pulsar Functions"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"…"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"爲什麼開發 RoP？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Apache Pulsar 是下一代雲原生分佈式消息流平臺，集消息、存儲、輕量化函數式計算爲一體。自 2016 年開源以來，Pulsar 已被廣泛採用，並於 2018 年被指定爲 Apache 頂級項目。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RocketMQ 是一款強大的開源分佈式消息系統，基於高可用分佈式集羣技術，提供低延時的、高可靠的消息發佈與訂閱服務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Pulsar 和 RocketMQ 擁有廣泛的用戶羣體和強勁的開發支持，全球許多頭部公司都在使用這兩種消息服務。同時，我們也收到了用戶的需求，希望能在 Pulsar 與 RocketMQ 之間傳輸數據，並充分利用這兩種消息系統的優勢。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Apache Pulsar 通過對 Consumer 層的抽象，提供了隊列和流兩種消費模型的統一抽象。在 Client 與 Broker 的交互中，Pulsar 基於 Protobuf 的二進制協議，提供更高的性能和更低的延遲。除此之外，通過 Protobuf 協議，Pulsar 可以更容易地支持並實現多語言的客戶端，比如：Java、CPP、Python 和 Go 語言等客戶端。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是，對於使用其他消息傳輸協議編寫的應用程序（例如，RocketMQ），由於使用的消息處理協議和 Pulsar 不同，如果 Pulsar 想要兼容 RocketMQ 協議，爲了將 RocketMQ 的協議適配到 Pulsar 的消息協議層中，用戶需要重寫整個協議層，這給用戶的遷移和切換帶來了很大的成本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了解決這個問題，最直觀的處理方式是使用類似 Pulsar Connector 的形式，將用戶在 RocketMQ 中的現存數據通過 RocketMQ Wrapper 的方式導入到 Pulsar 集羣，但是這需要業務端更改自己的業務代碼邏輯，同時需要確保兩邊的數據能夠保證一致，這給使用 RocketMQ 的用戶帶來了很大的技術挑戰。所以，能否給用戶提供一個開箱即用的遷移策略和方案並且用戶無需做任何代碼修改呢？這便是 RoP 誕生的最初目的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"怎樣開發 RoP？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Apache Pulsar 在 PIP-41("},{"type":"link","attrs":{"href":"https:\/\/github.com\/apache\/pulsar\/wiki\/PIP-41%3A-Pluggable-Protocol-Handler","title":"","type":null},"content":[{"type":"text","text":"https:\/\/github.com\/apache\/pulsar\/wiki\/PIP-41%3A-Pluggable-Protocol-Handler"}]},{"type":"text","text":") 中介紹了一種全新的接入方式。通過在 Broker 端暴露 Protocol Handler 插件，將 Netty 的 channel 和 Pulsar 的 Broker Service("},{"type":"link","attrs":{"href":"https:\/\/github.com\/apache\/pulsar\/blob\/907fcb5ba8\/pulsar-broker\/src\/main\/java\/org\/apache\/pulsar\/broker\/service\/BrokerService.java","title":"","type":null},"content":[{"type":"text","text":"https:\/\/github.com\/apache\/pulsar\/blob\/907fcb5ba8\/pulsar-broker\/src\/main\/java\/org\/apache\/pulsar\/broker\/service\/BrokerService.java"}]},{"type":"text","text":") 對象暴露給用戶。這允許用戶直接操作和調用 Pulsar 中比較低階的 API（例如：PersistentTopic 和 ManagerLedger）。基於這個協議，用戶無需更改代碼，只需將服務請求轉發到 RoP 中，RoP 利用 Protocol Handler 的插件將用戶的請求轉發到 Pulsar 中即可。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"RoP 架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/a7\/a7df4b886704355dc9279091f9ed72d7.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過對比 Pulsar 和 RocketMQ 之間的協議可以發現，二者在消息處理的思路上有不少相似之處，比如這兩種協議都包含如下操作："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Topic Lookup: 所有 Clients 與任意 Broker 建立連接之前，會先去查找當前 Topic 的 Owner Broker。獲取到對應的 metadata 之後，Clients 會與 Owner Broker 之間建立 TCP 連接進行數據的交互。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Produce: Clients 與 Topic 所在的所有 Owner Broker 之間進行通信並將消息 append 到對應的分佈式日誌中。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Consume: Clients 與 Topic 所在的所有 Owner Broker 之間進行通信並從分佈式日誌中讀取指定的消息。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Offset: Producer 生產到 topic 中的消息會分配一個唯一的 offset，Pulsar 中使用 MessageID 來標識 offset。消費者可以通過 offset 去日誌中獲取指定位置的消息。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Apache Pulsar 的存儲層使用了 Apache BookKeeper，Pulsar 相當於 BookKeeper 的 Client，通過調用 ManagerLedger 對象能夠很容易的達到爲分佈式日誌操作的目的。基於此，RoP 可以很好的將 RocketMQ 中對 commitLog 和 queueLog 的操作映射到 BookKeeper 中來。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"RoP 概念"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Offset 和 MessageID"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 RocketMQ 中，使用 offset 來標識消息的位置，當消息被生產到指定的 Topic 之後，會爲每一個消息分配一個唯一的 offset；在 Pulsar 中，使用 MessageID 來唯一標識每條消息，每一個 MessageID 由三部分組成，"},{"type":"codeinline","content":[{"type":"text","text":"ledgerID"}]},{"type":"text","text":"、"},{"type":"codeinline","content":[{"type":"text","text":"entryID"}]},{"type":"text","text":" 和 "},{"type":"codeinline","content":[{"type":"text","text":"partitionID"}]},{"type":"text","text":"。我們通過合理的劃分將 messageID 和 offset 進行映射，來唯一標識 Topic 中的每一條消息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Message"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於一條消息，RocketMQ 和 Pulsar 都包含消息的 headers 和 payload 等字段，通過對消息協議的解析，我們可以輕鬆的將 RocketMQ message 轉換爲 Pulsar 的 message 格式。爲了更好的兼容 Tag 消息的功能，在消息協議的處理方面增加了 8 字節的特殊字段，用來區分該消息是否屬於 Tag 消息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Topic Lookup"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 Pulsar 中，client 與 broker 建立連接之前，會根據當前傳入的 Topic 執行 Lookup 操作，在 Broker 集羣中尋找當前 Topic 所在的 Owner Broker，然後將該 Owner Broker 的地址返回並與 client 建立 TCP 連接，再進行數據交互。在 RocketMQ 中，client 與 broker 建立連接之前，會先處理 "},{"type":"codeinline","content":[{"type":"text","text":"GET_ROUTEINTO_BY_TOPIC"}]},{"type":"text","text":" 命令，獲取 topic 所在的路由信息後，建立對應的 TCP 連接，再進行數據交互。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如何使用 RoP?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，RoP 發佈了 0.1.0 版本，你可以用過以下任一方式參與該項目："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"想上手試試？"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可在以下網址下載 RoP 和查閱用戶指南："},{"type":"link","attrs":{"href":"https:\/\/github.com\/streamnative\/rop\/blob\/master\/README.md%E3%80%82%E6%97%A0%E8%AE%BA%E6%98%AF%E5%BF%AB%E9%80%9F%E5%90%AF%E5%8A%A8","title":"","type":null},"content":[{"type":"text","text":"https:\/\/github.com\/streamnative\/rop\/blob\/master\/README.md。無論是快速啓動"}]},{"type":"text","text":" standalone RoP 或在現有 Pulsar 集羣中部署 RoP，都可輕鬆實現。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，爲了方便快速使用並驗證 RoP，我們提供了 RocketMQ 的常見使用場景和用例，你可以直接使用這些代碼示例驗證服務："},{"type":"link","attrs":{"href":"https:\/\/github.com\/streamnative\/rop\/tree\/master\/examples\/src\/main\/java\/org\/streamnative\/rocketmq\/example","title":"","type":null},"content":[{"type":"text","text":"https:\/\/github.com\/streamnative\/rop\/tree\/master\/examples\/src\/main\/java\/org\/streamnative\/rocketmq\/example"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"想解決問題？"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如有任何問題，可以在 RoP GitHub repo 中 創建 issue 或加入 RoP 微信羣進行討論。無論哪種方式，RoP 資深專家都隨時在線："},{"type":"link","attrs":{"href":"https:\/\/github.com\/streamnative\/rop\/issues\/new","title":"","type":null},"content":[{"type":"text","text":"https:\/\/github.com\/streamnative\/rop\/issues\/new"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"想參與貢獻？"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RoP 源碼開放並託管在 GitHub 上："},{"type":"link","attrs":{"href":"https:\/\/github.com\/streamnative\/rop%EF%BC%89%E3%80%82%E3%80%82%E5%A6%82%E9%9C%80%E6%94%B9%E8%BF%9B%E5%8A%9F%E8%83%BD%E6%88%96%E4%BF%AE%E5%A4%8D","title":"","type":null},"content":[{"type":"text","text":"https:\/\/github.com\/streamnative\/rop）。如需改進功能或修復"}]},{"type":"text","text":" bug，歡迎提交 PR。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}