告別傳統金融消息架構：Apache Pulsar 在平安證券的實踐

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在金融場景中，伴隨着業務的擴展，應用系統也相應地增加更多的場景，這些新場景對消息系統提出更多樣的需求，導致原有架構面臨一系列挑戰。在嘗試使用 Apache Pulsar 後，平安證券決定在生產環境中進行實踐。本文介紹了平安證券選擇 Apache Pulsar 的原因，使用 Apache Pulsar 的場景，Apache Pulsar 實踐應用中遇到的問題，以及使用 Apache Pulsar 的未來規劃。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景介紹"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"傳統金融公司或券商一般會使用統一接入服務或組件來處理對外業務。接收到用戶請求後，根據相應的業務規則將請求轉到對應業務系統 \/ 模塊。有些請求會轉發給消息隊列，請求寫入後，下游業務系統從消息隊列中獲取請求，並在處理後通過消息隊列原路返回給客戶，整個請求過程封閉運行，功能有限。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"消息隊列下傳統架構帶來的挑戰"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"平安證券採用的就是上述的傳統架構，目前只支持消息隊列。雖然我們有一定的開發能力，但也難以獲取到該消息隊列的細節信息。同時，由於是定製開發的系統，支持的語言比較有限。現有的消息隊列對業務發展和業務創新等有以下不足："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"黑盒系統，難以觀測"},{"type":"text","text":"：消息隊列是一個黑盒系統，我們難以觀測到架構的細節；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"直接交換（Direct Exchange），無法路由"},{"type":"text","text":"：由於架構目前只支持消息隊列，無法支持需要路由的場景；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"弱校驗接入，安全風險高"},{"type":"text","text":"：現有系統的密碼認證、校驗等檢驗較弱，安全風險較高；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"定製系統，有限語言支持"},{"type":"text","text":"：定製系統接入語言的支持有限，導致我們選擇範圍少，難以在原有系統基礎上進行改革。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着業務擴展和架構改進，公司現有的消息隊列系統 \/ 組件面臨着一系列挑戰，而系統存在的許多問題如安全性等在金融場景中刻不容緩。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"金融場景的業務需求"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們的業務需求主要分爲三類：身份識別 & 安全控制、路由分發、審計。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"身份識別 & 安全控制"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"身份識別主要用於確定接入消息隊列的客戶端和接入者的身份信息，指定相應的安全規則，拒絕不合法接入者，進而實現預期的安全要求。從最基礎的層面看，需要識別控制接入的系統、IP，根據業務場景及特定需求，進行權限限制。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"路由分發"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"路由分發指消息根據相應的規則由寫入隊列路由至對應的隊列。現有的消息隊列支持的場景有限，若想要支持更多場景，需要投入大量的時間和精力來進行開發（涉及上下游系統的改造），同時會引入其他問題。較好的解決方案是消息隊列系統原生支持更多模式及特性，比如 TOPIC 模式、流式消息處理。如果消息隊列系統可以支持路由，那麼系統的接入複雜度就會大大降低，可以通過更優的方式對接入層進行操作，每個系統只需要對接一組 topic，路由負責分發；也可以更有針對性地優化性能（路由、轉發、協議轉化都是消耗性能的操作）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原有系統架構通訊機制是點對點，封閉運行，請求消息無法共享，只能間接採用適配器或日誌採集方式實現分發，此類做法難以有效滿足實時性要求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"審計"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息的發佈者 \/ 接收者都屬於整個系統的參與者，並且是重中之重。系統安全性的主要影響因素就是系統的所有參與者；因此，從安全角度出發，對消息的審計要求相對較高。另一項比較急迫的需求是對消息流向進行控制。如果可以進行身份識別和安全控制，則可以在審計時完善和優化安全信息，進而保證在業務入口處拒絕無效、非法請求，保證內部系統健壯。此外，記錄接入的消息發佈者 \/ 接收者的信息還可以用於異常情況監控、稽覈審計。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"新增業務的系統需求"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新增業務對消息系統提出了更高要求，主要包括可用性、消息發送延遲、擴縮容、消息回溯等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"需求一：高可用、低延遲"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於互聯網行業而言，高可用低延遲是系統的基本要求。從單點到災備，到同城跨機房，再到異城跨多中心，或者是先跨城、災備，再跨城多中心（兩地三中心）的模式都已經越來越常態化，很多公司的業務系統正在或將會往這方向發展。這樣的系統對高可用、低延遲的要求比較高。因此需要考慮當系統複雜度增加（如災備、跨城等場景）時，如何將延遲降到最低。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"需求二：快速擴容與恢復"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於金融業而言，業務的主要特性之一是請求可能會在某個時間段或某個週期激增，過了這個時間窗口後，流量逐漸恢復正常。這一特性要求系統可以快速橫向擴縮容，出於成本考慮，按照最高流量部署整個系統架構顯然不合理。最好的解決方案是系統可以根據單層流量合理安排系統架構或系統部署方式，在流量突然增加時，系統可以快速擴容，支撐業務。最理想的情況是系統的所有組件都有快速擴縮容、恢復能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"需求三：消息有序、消息防重"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在一些特殊業務場景中，需要保證消息有序或防重。我們經常對一些接口進行冪等操作，如果可以保證上游消息不重複，就可以減小下游的壓力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"需求四：可回溯、序列化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果業務系統出現問題，但在測試環境中難以復現這一問題，就需要引入消息回溯。消息回溯指重放一遍出現問題的時間窗口中的所有請求，驗證是否能復現問題，並排查問題，這樣可以大大減輕排查問題的工作量。此外，我們還可以藉助這一功能進行灰度驗證和並行驗證。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"選擇 Apache Pulsar"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於上述業務需求和系統需求，發現 Apache Pulsar 的諸多特性完美契合了我們的需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"集羣模式，支持跨集羣同步。建設系統雙活，跨集羣的地域複製在客戶端無感的情況下實現消息同步。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"計算存儲分離。根據使用情況橫向擴展存儲 \/ 計算，客戶端對此操作無感知。基於二級存儲的功能，擴展消息的使用場景，爲數據分析、消息審計提供可能。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"客戶端接入認證模塊插件化，支持自定義開發。因業務需求，在客戶端接入時，要求鑑權、認證，有效保證消息的來源可靠、可控。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"完善的 Rest API，可查看隊列情況。之前使用的消息系統有很好的性能，但在可觀測性方面有所欠缺，給系統排障造成困難，同時消息系統的管理方式較爲原始，難以適配大規模系統管理的要求。而 Apache Pulsar 完善的 Rest API 不僅可以獲取系統運行指標，且有助於集羣的高效管理。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"基於 Functions 可實現消息的路由開發、過濾和統計等。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":6,"align":null,"origin":null},"content":[{"type":"text","text":"可設置消息的持久化模式和過期時間，允許消息重放。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":7,"align":null,"origin":null},"content":[{"type":"text","text":"多語言支持，快速便捷接入。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Apache Pulsar 在平安證券的業務場景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"平安證券使用 Apache Pulsar 構建統一消息平臺，期望整合客戶、交易、行情、資金四大數據流，應用於行情分發、實時風控等。本文主要介紹如何將 Apache Pulsar 應用於三個業務場景：請求路由、數據廣播和消息通知，新架構的優勢和不足，以及其對開發、運維團隊的影響。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"場景一：請求路由——簡化系統"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們的消息路由流程如下圖。從 A 組件發出的請求寫入到 Topic A，然後由路由模塊將 topic 中的信息進行路由，分發到多個對應的 topic，訂閱了這些 topic 的下游組件就可以處理相關的消息。組件 A 只需要向固定的隊列寫入消息，不需要關注 Topic B、C、D 的信息，下游系統只需要瞭解接收消息的隊列，不需要關注 Topic A，從而簡化整個網絡的結構。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/04\/04c105972e07c0d1383c5a6d11d9a7ea.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這種消息路由模式簡化了系統的整體架構，目前我們的路由系統仍待優化："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"雖然路由分發的工作量得以減輕，但排查問題的步驟有所增加。比如在組件 A 發送消息後，組件 B 沒有收到消息時，需要先檢查組件 A 是否寫入消息到 Topic A、路由模塊是否成功路由這一消息，再看組件 B 是否正確訂閱了這條消息。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"從目前的測試效果看，由於消息鏈路變長，時延增加。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"由於每個隊列的消息都會持久化，導致存儲和隊列中都出現數據冗餘。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"路由模塊是新增模塊，運維的學習成本較高。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"場景二：數據廣播——降低時延"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據廣播是我們使用 Apache Pulsar 的另一個業務場景。數據廣播採用發送 \/ 訂閱模式，主要用於同步消息。很久之前，我們不需要同步行情到業務系統，或是通過其他方式（如同步數據庫）實現。但隨着業務的增長，同步時效和用戶體驗的競爭度越來越激烈。如何可以讓用戶更快地看到信息？以同步行情的場景爲例，先同步數據庫再查閱的方式，時延相對較長；而在廣播模式中，業務系統只需訂閱所有需要的 Topic，查閱時即可直接讀取數據，有效降低時延。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/22\/223bda4c9ad8de6049ea82ce2abd6e11.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"場景三：消息通知——安全管控"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們使用到 Apache Pulsar 的第三個場景是消息通知。雖然消息通知涉及到的業務相對較少，但這一業務場景十分重要。整體業務流程圖如下。由於信號源不唯一，因此在消息發佈到計算引擎後，計算引擎需要根據信號源的信息進行邏輯、安全等方面的計算。計算完成後調起 Task，再由激活的 Task 向相關業務系統發送業務請求，執行後將結果返回給發起信號源的服務，該服務根據返回的結果觸發下一個信號源。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這一場景涉及到的業務對安全和管控的要求非常嚴格，不僅需要限制信號源發送的消息或信號，截斷 \/ 過濾某些信號，還需要對返回的結果進行處理：哪些可以返回，哪些需要過濾掉或轉換成其他內容。如果不使用消息隊列方式，消息源會直接發送消息給計算引擎，在計算引擎執行安全或管控策略後，將消息發送到 Task; 在 Task 執行完成後，其結果需要再進行一輪安全管控處理。這一部分的重複操作對性能影響較大，同時策略更新、信號狀態查看的時效性沒有那麼實時。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"引入 Apache Pulsar 後，我們將管控審計模塊剝離出來，專門針對信號隊列和結果隊列進行過濾、審計、統計等操作，並實時輸出結果到管理端。運維或審計人員在看到這些信息後，可以控制、更新相應策略。這一模式不僅可以精簡數據流，還可以增加數據補充渠道，也更清晰地定義了各服務模塊的邊界。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/f5\/f5cae14de6052873949c44bd45351896.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"問題發現與解決方案"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前我們主要在上述三個場景中探究了 Apache Pulsar 的使用，並逐步上線生產。在使用過程中，我們發現了幾個問題，並在此分享我們的解決方案以供參考。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"實現 REQ-REP 模式"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們遇到的第一個問題是如何實現請求 - 響應（REQ-REP）模式，我們的解決方案是通過總線模式進行兼容。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前常見的調用方式是客戶端發起調用請求，服務端處理完成後返回響應即可。但引入總線以後（同步轉異步），在多節點的部署場景中，節點 1 發出請求，服務端收到請求後返回處理結果，所有節點都需要監聽這條處理結果，節點 2 收到歸屬節點 1 的響應消息時應該如何處理？節點 2 需要先訂閱並獲取回包的消息，判斷是不是自身節點發起請求的響應，如果不是，則丟棄此消息。假如按照這種模式進行實現，則在發送消息時，每個節點都需要緩存自身發送的消息 ID；服務端處理完以後，按照協議回包數據需要帶上請求的消息 ID，每個節點都訂閱獲取所有回包，並校驗緩存中是否有該消息 ID，若不存在，則丟棄消息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/b3\/b357d0ff4cf3f06f7b592c61b0052cfd.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該實現方式下存在一個非常嚴峻的問題亟待解決：節點發起一個查詢大量數據的請求時，假定 Apache Pulsar 設置一個消息 的大小爲 8M，TPS 爲 1000，那是不是每個節點都要收到這麼多請求的回包流量呢？假如有 5 個節點，每個節點本應該只接收 200 個請求的回包流量就夠了，但現在的模式需要每個節點承受 1000 個請求的回包流量，而其目的僅僅是爲了過濾操作。如果節點負載性能達到上限，需要擴容節點，將導致網絡帶寬成倍增加。由於 Apache Pulsar 可以支持大量 Topic，雖然通過給每個節點配置一個回包隊列等方式可以解決這一問題，但我們想嘗試通過 broker 的 FILTER 功能，來解決該問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2. 實現讀寫分離"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息廣播場景會涉及到讀寫分離。如果增加大量訂閱節點，最好避免將所有節點的鏈接集中在 Topic 的 owner broker 上。針對這個問題，可行的解決方案是合理分配使用的 Topic 和 Partition。我們目前使用的 Apache Pulsar 2.7.2 還不支持讀寫分離，計劃把 Apache Pulsar 升級到 2.8，就可以輕鬆實現讀寫分離，滿足消息廣播場景的需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/ed\/edeb8a44c307847cc740ac806e18e551.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3. 解決多網卡問題"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於公司網絡安全考慮，內部存在多種網絡分區及網段，不同的網絡分區 \/ 網段使用不同的 IP，服務器存在多個網卡，供跨分區系統間通信。目前如果使用 IP 註冊 broker，只能註冊某個網段的 IP；如果使用域名註冊 broker，則不同網絡區域的 DNS 解析又需要進行不同的配置。如果 broker 可以支持多網卡通信，這些問題就不存在了。目前我們的解決方案是用 proxy 代理客戶端的請求，外部系統也只連接到 proxy，我們也會爲 proxy 增加一些高可用的配置。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/8c\/8ca92a27b68fd3a3b30e5a9cf69657ca.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"未來規劃"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前 ，我們在單機房、單集羣線上小規模運行 Apache Pulsar，在上線初期未考慮建設雙活。作爲業務系統的基礎設施，Apache Pulsar 自身可用性極爲重要。因此，我們計劃基於同城雙中心單集羣建設進行雙活規劃，如圖如示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/a3\/a3e5adce1372c01d11c3b80ce243492f.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在測試和使用 Apache Pulsar 的過程中，我們遇到了一些問題，感謝 Apache Pulsar 社區的積極響應。我們期待更多地參與到 Apache Pulsar 的研發中，也期待爲 Apache Pulsar 和 Apache Pulsar 社區做出貢獻。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者簡介："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"王東松，平安證券經紀業務事業部研發工程師。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"陳翔，平安證券經紀事業部架構師。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}