RocketMQ 在金融企業技術中臺的落地實踐

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"線上各位的朋友們大家好,很榮幸能和大家交流RocketMQ使用的相關經驗。我今天和大家分享的主題是《RocketMQ 在金融企業技術中臺的落地實踐》。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先做下簡單的自我介紹,我叫陳培新,來着國信證券。目前參與國信證券技術中臺建設相關工作,專注的技術有微服務、Serverless。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本次分享的內容包含如下兩個部分。首先介紹國信證券技術中臺的背景、發展歷史、架構等情況,然後再介紹RocketMQ在技術中臺的各種落地場景情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們先看下技術中臺的簡介。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先是技術中臺的背景,在當前數字化轉型的浪潮中,券商行業面臨有三個挑戰:1是長久以來,券商信息系統建設模式以廠商爲主,導致大量煙囪式的異構系統,數據集成困難,系統運維壓力大;2是行業競爭和業務複雜化,對信息技術提出更高的要求,存在業務需求響應不及時,跨部門協作成本高的問題。3是缺乏自主掌控,創新後勁不足。那麼爲了應對上述挑戰,國信證券提出建設全棧式敏捷研發技術中臺作爲數字化轉型重點,通過架構支撐、研發賦能,實現統一開發框架、快速交付、自主掌控的目標。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來我們再看下國信技術中臺架構進化路線圖。我們可以看到從早期的總線結構、到微服務、到容器化、到DevOps、以及正在進行的Serverless。國信技術架構工作,每1-2年都會上一個臺階,應用範圍也越來越廣。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/e1\/e1665567c2aab1b6dad2e71480a7c35f.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面是整個技術中臺的技術全景圖。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/38\/38a330bb53b5762b1b3e56094cf0f837.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"包括互聯互通、共享服務、基礎架構、持續交付等四個子域。其中互聯互通解決安全高性能接入的問題,支持消息推送,其中互聯網關爲基於Netty自研的API網關。共享服務提供基礎服務的複用能力,抽象提煉了自選股中心、用戶中心、行情中心、擴展交易中心等公共基礎服務,可實現業務應用的快速建設。基礎架構包括自研的Zebra微服務、消息隊列、分佈式技術、服務網格等,其中Zebra微服務已經Gitee社區開源,歡迎大家使用和多提意見。持續交付子域主要包含Simba研發效能平臺,微服務自動發佈平臺,實現開發運維端到端的自動化,可顯著提升開發、測試、運維研發流程的一體化程度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後看下技術中臺在國信證券的應用效果。技術中臺全面支撐國信證券各類應用,如手機金太陽、郵件系統、企業微信等。金太陽手機證券APP支撐了包括交易、理財、業務辦理、資訊、開戶、擴展行情等業務,承擔了超過50%的互聯網流量,其中接入的國信金太陽手機證券APP總註冊用戶超過1600萬,月活數據324萬,動態有效用戶超200萬,日併發峯值超過27000+TPS,日推送個性化信息量達到1840萬筆,接入以來未出現任何平臺級的生產事件,爲國信證券信息系統的穩定、高效、安全迭代提供有力的支撐。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/52\/520cb97f3443bbb3895cdca8667bdb2a.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上就是國信技術中臺的相關介紹,那麼接下來我們來看看RocketMQ在國信中臺是如何落地應用的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先看下集羣搭建方面。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a5\/a500fa9c56169eeb61f4eee82fae06f7.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們知道,如果是通過手工的搭建RocketMQ,或多或少存在誤配置的問題,同時對於RocketMQ相關性能調優的經驗也難以複用沉澱。爲此在基礎架構方面,我們在開發測試網段、生產網段都提供了RocketMQ的集羣自動構建流程,將構建流程自動化。開發或運維可根據需要填入所需的集羣信息,待流程審批通過後,平臺會自動做集羣構建。這樣一來可大大減少集羣搭建的時間,減少出錯概率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來看RocketMQ在推送平臺的應用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b3\/b3fa31586cea096c3ed5fdcb4ae4648e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"推送平臺支撐國信網金、營業部、總部分支機構等業務部門的信息推送,和當前主流手機廠商都做了對接,如蘋果、華爲、小米和OV等,目前日均推送信息數量爲200w。推送平臺提供了多維度的統計查詢功能,可方便讓推送者查詢其推送信息的各個狀態情況。這樣就要求推送平臺對消息推送的狀態做記錄,對每個人的每條消息都要記錄推送狀態日誌,每條消息大概有10個狀態,那麼總計每天日誌記錄量爲2000w左右。同時消息推送特點是集中在交易日的開市 9點到10點左右。如果直接採用JDBC入庫的話,那麼會對數據庫造成較大的壓力,同時也可能影響到消息推送的速度。爲了避免上述問題,推送平臺使用RocketMQ發送推送狀態日誌,推送統計模塊拉取RocketMQ推送日誌,然後對日誌做批量入庫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在這個場景中,我們使用RocketMQ來做削峯。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二個是在API網關中的應用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/c3\/c30a6de9d9089874e4a985fa77e6291d.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"國信API網關是基於Netty自主研發的網關。它支持了多協議(HTTP、TCP、WebSocket),提供路由、協議轉換、安全認證等特性,是國信Zebra微服務的總入口。我們知道,API網關有一個重要的特性是路由,簡單的說就是進來一個請求,網關要根據URL等信息,將請求分發到對應微服務的某個接口。和常見的Zuul或者SpringCloud Gateway不同,Zebra後端微服務是基於gRPC協議,這裏的API網關是將URL映射爲微服務的gRPC接口。在API網關運行過程中,後端微服務不斷地做的迭代升級,提供的接口在不停變化。爲了不對客戶造成影響,要求API網關要支持做到不停服務更新路由規則。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"API網關的路由規則放在自研的配置中心,當對應的API網關規則有變化時,會將變更的路由規則通過RocketMQ發送出來。API網關在收取到路由更新RocketMQ消息後,實時更新內存中的路由規則,從而達到路由規則的熱更新。下面的API網關的路由規則配置樣例,在方法映射中,左邊是URL前綴,右邊是微服務接口名。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/0e\/0ef2189ae1acdeeca1f3104bb2d95a3e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於微服務的配置熱更新也是類似,微服務配置的熱更新一般用於開關、閾值、熔斷限流更新等場景。配置中心和微服務框架使用另外一個RocketMQ Topic。使用廣播的模式,Tag爲對應的微服務名稱,這樣不同的微服務可根據Tag對消息做過濾,只處理本微服務的配置變更消息。在配置持久化後,配置中心將更新的配置通過RocketMQ消息發送,對應微服務接收到對應消息後,更新對應內存中的配置。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上述兩個場景中,我們使用RocketMQ來做消息通知。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來再看看RocketMQ在分佈式任務監控中的應用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ba\/ba001bb7e4a5f583a8e7413f675297ad.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"國信採用ElasticJob作爲分佈式任務框架,同時聯合運維開發了一個對任務運行業務狀態做的監控組件。可以通過這個監控組件對執行出問題的的任務做告警,還可以查詢任務歷史執行記錄。舉個例子,比如任務在指定的時間觸發了,但是所依賴的資源(比如數據,或者微服務掛掉了),從業務角度來看,任務執行結果爲失敗,此時應該做告警。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們使用了2張表作爲任務的監控(一張任務狀態表、一張任務歷史表)。一開始我們提供的分佈式任務SDK通過直接連接數據庫的方式寫入任務的監控信息,技術上雖然可行。但是不優雅,和監控數據庫的耦合性太強,每個任務都必須配置和數據庫的連接信息。如果數據庫密碼修改或者字段有變化,得通知到各個微服務的運維做修改,工作量龐大且容易遺漏。所以在後面我們通過RocketMQ將微服務監控和對應的監控數據庫做了解耦,SDK使用RocketMQ發送任務結果,同時新增一個“分佈式任務狀態處理”組件來拉取監控消息然後統一入庫。這樣監控數據庫有變更時,只需變更這個新增的組件即可,變更工作量和風險大大降低。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到我們使用RocketMQ來做系統組件的之間的解耦。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來我們再來看看RocketMQ最後一個落地場景-異步任務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7e\/7e741f6f74e569860c3bbc71252cc5f4.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在國信內部開發測試環境和生產環境隔離,安全規定不允許開放端口,也就是說兩個環境的應用之間無法通過接口相互調用。我們可以看到DevOps平臺部署在開發測試網段(比如Jira、Gitlab等),ITIL、容器、微服務配置中心等在生產環境部署。但是去年我們平臺支撐的項目在做持續交付三級的事情,標準裏面又間接地要求技術中臺將上訴兩個環境打通。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"舉幾個例子:比如“部署與發佈管理”中“過程可視化”對上線流程的要求,在開發側中Jira可以自動創建ITIL(要求開發環境的Jira可以通過生產的ITIL創建工單),ITIL裏面各類流轉信息要在開發側可見(我們通過回寫到Jira上線單),對於部署平臺比如容器平臺,數據庫發佈平臺的日誌在開發側也要可見。第二個是配置的管理,比如配置晉級,微服務的配置要從開發、測試最後一路晉級到生產環境。也要打通到生產。最後和安全組達成一個辦法,將兩個網段直接的系統調用系統通過公司的安全文件通過來存放。然後兩邊各部署一套通道服務來定期獲取文件,解析並調用對應的系統接口。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於對接系統和接口較多,我們部署對每個系統都部署對應的任務處理器,啓用任務獲取任務定期獲取文件,做簡單解析後,然後通過RocketMQ發送任務消息,對應任務處理器接收到RocketMQ再做相應的處理。使用RocketMQ可保持任務處理器之間的獨立性和可擴展性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後再簡單看下中臺微服務框架對RocketMQ的支持,微服務提供了集成RocketMQ的文檔說明,使用方法等,同時微服務的腳手架也支持可視化引入RocketMQ依賴。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/dd\/ddb09b55efefde586ed2c2809cff54a1.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/2a\/2ad24a518766808af03b6d412fd60b5f.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上就是今天分析的國信技術中臺落地實踐RocketMQ的場景,希望能夠對大家在日常開發的方案設計有所幫助,謝謝大家。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]}]}
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章