騰訊大數據實時分析引擎Hermes揭祕

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hermes 是騰訊數據平臺部自研的實時分析平臺，在公司內服務於上百個業務，集羣規模 5000 個節點，每日數據接入量 4 萬億，查詢量千萬級別。作爲一個公共的平臺，面對的業務場景非常複雜，包括在線高併發分析、即席交互分析、海量日誌分析、實時接入數據和近實時增量更新。這樣一個萬億級的實時計算開發引擎到底是怎麼實現的？研發過程中遇到那些難點？作爲開發者，我該怎麼借鑑和避免；作爲用戶，又有哪些新的思考？"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在2021年4月22-24日舉辦的 "},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/beijing\/?utm_source=web&utm_campaign=9&utm_medium=infoq&utm_content=dengqibin","title":"xxx","type":null},"content":[{"type":"text","text":"QCon全球軟件開發大會（北京站）"}]},{"type":"text","text":"“"},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/beijing\/track\/1004?utm_source=web&utm_campaign=9&utm_medium=infoq&utm_content=dengqibin","title":"xxx","type":null},"content":[{"type":"text","text":"大數據實時計算與分析"}]},{"type":"text","text":"“專題上，騰訊高級後臺開發工程師"},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/beijing\/presentation\/3265?utm_source=web&utm_campaign=9&utm_medium=infoq&utm_content=dengqibin","title":"xxx","type":null},"content":[{"type":"text","text":"鄧啓斌"}]},{"type":"text","text":"老師將分享"},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/beijing\/presentation\/3265?utm_source=web&utm_campaign=9&utm_medium=infoq&utm_content=dengqibin","title":"xxx","type":null},"content":[{"type":"text","text":"《騰訊萬億實時分析引擎 Hermes 的實現與應用"}]},{"type":"text","text":"》，在會前 InfoQ 帶着疑問對鄧老師進行了採訪。幫助大家瞭解高性能實時分析系統的底層技術實現原理，以及瞭解海量數據和大規模集羣的實時分析平臺的運營經驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Hermes 研發的背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：請介紹下自己？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"鄧啓斌："},{"type":"text","text":" 我是鄧啓斌 (kewndeng)，騰訊大數據專家工程師。"},{"type":"link","attrs":{"href":"https:\/\/s.geekbang.org\/search\/c=0\/k=Hermes\/t=?utm_source=web&utm_campaign=9&utm_medium=infoq&utm_content=dengqibin","title":"xxx","type":null},"content":[{"type":"text","text":"Hermes"}]},{"type":"text","text":" 實時分析平臺的研發負責人，主導並負責了海量日誌存儲平臺、智慧零售搜索引擎、Hermes 實時分析平臺。目前 Hermes 團隊聚焦於騰訊廣告的實時 OLAP 分析，致力於提升查詢分析性能和數據時效性，降低機器資源成本，並提升業務數據分析效能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：Hermes 研發的背景是？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"鄧啓斌："},{"type":"text","text":"Hermes 項目始於 2012 年，當時主要面向公司內的用戶畫像業務，提供多維分析、用戶洞察、人羣交叉等能力，讓萬級維度、千億級數據下的秒級統計分析變爲現實。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 Hermes 1.0，數據通過離線的 MR 任務生成再同步到 Hermes，通常數據時效性是 T-1。但由於業務對數據時效性有越來越高的要求，譬如服務器日誌、交易流水、安全風控需要以近實時方式入庫和快速可查，於是在 2015 年研發 Hermes 2.0，業務把數據接入到 TubeMQ（騰訊大數據開源的消息中間件），再由 Hermes 進行消費，實現從數據接入到可查只需秒級的延遲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而在面向海量數據的 OLAP 實時分析，業界沒有通用高效的解決方案，爲解決公司廣告系統複雜分析場景，我們在 2019 年開始把重心放在分析型列存和計算引擎的研發，並推出了針對實時分析型業務的 Hermes 3.0。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而在我加入Hermes團隊的時候，Hermes 2.0 已在中小規模的業務中使用，單個業務每天百億到千億的實時接入量。我的首個任務是優化寫入、查詢性能和可用性，從而支撐起了萬億級數據量的業務，實現了業務的規模化接入。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：Hermes 能解決什麼問題？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"鄧啓斌："},{"type":"text","text":" 通過前面 Hermes 背景的介紹，大家應該都瞭解到 Hermes 是應用於什麼樣的業務。下面我舉三個案例讓大家加深理解。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"支撐騰訊超大規模用戶畫像，80 億用戶賬號、上萬維度、百億標籤，支持任意維度組合（包括下鑽、號碼包提取）分析等能力，有效支持了業務洞察分析、人羣分析、廣告推薦包分析以及廣告推送包提取等，"},{"type":"text","marks":[{"type":"strong"}],"text":"99% 的分析都是在 3 秒內完成"},{"type":"text","text":"。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"支撐微信支付的海量日誌存儲和檢索，自從接入以來，日誌量持續增長，"},{"type":"text","marks":[{"type":"strong"}],"text":"目前單個集羣每天接入量超萬億、存儲規模PB級，同時查詢 SLA 達到4 個 9"},{"type":"text","text":"。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"支撐騰訊廣告中的在線報表、運營分析、流量日誌分析，其中在廣告實驗分析，對策略組和對照組的實驗在任意維度下關聯計算數千個指標，支持千億實驗數據關聯、秒級響應，這種基礎能力使業務由過去 T-1 時效、人工開發的報表，升級爲交互式的即席查詢，大大提升了業務分析和決策的效率。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"通過前面的採訪，我們對 Hermes 有了一個比較直觀的瞭解，正是因爲知道 Hermes 是騰訊數據平臺部自研的實時分析平臺，面對的業務場景非常複雜。這樣一個萬億級的實時計算開發引擎到底是怎麼實現的？研發過程中遇到那些難點？"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"研發過程中的坑"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：在研發 Hermes 過程中遇到了哪些讓人頭疼的問題，當時是怎麼解決的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"鄧啓斌："},{"type":"text","text":" 在實時分析的技術領域中，沒有哪個項目在各個場景都是通喫的。業界的 "},{"type":"link","attrs":{"href":"https:\/\/s.geekbang.org\/search\/c=0\/k=ElasticSearch\/t=?utm_source=web&utm_campaign=9&utm_medium=infoq&utm_content=dengqibin","title":"xxx","type":null},"content":[{"type":"text","text":"ElasticSearch"}]},{"type":"text","text":"、"},{"type":"link","attrs":{"href":"https:\/\/s.geekbang.org\/search\/c=0\/k=Kylin\/t=?utm_source=web&utm_campaign=9&utm_medium=infoq&utm_content=dengqibin","title":"xxx","type":null},"content":[{"type":"text","text":"Kylin"}]},{"type":"text","text":"、"},{"type":"link","attrs":{"href":"https:\/\/s.geekbang.org\/search\/c=0\/k=Druid\/t=?utm_source=web&utm_campaign=9&utm_medium=infoq&utm_content=dengqibin","title":"xxx","type":null},"content":[{"type":"text","text":"Druid"}]},{"type":"text","text":"、"},{"type":"link","attrs":{"href":"https:\/\/s.geekbang.org\/search\/c=0\/k=ClickHouse\/t=?utm_source=web&utm_campaign=9&utm_medium=infoq&utm_content=dengqibin","title":"xxx","type":null},"content":[{"type":"text","text":"ClickHouse"}]},{"type":"text","text":"，分別在各自的細分領域做得很強大。在業務選型時 Hermes 經常被拿來跟它們進行對比，並挑戰在性能上有哪些優勢，功能是否更強大，爲什麼要用 Hermes？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，找準賽道是最讓人頭痛的問題，只有清晰認識到要做一個什麼樣的項目，才能決定使用什麼樣的技術，並持續把技術做好和做深。經過數年研發的演進，Hermes 具備了差異化的能力，在公司不少業務在遇到實時分析的落地難題時或走了彎路後找上門來。譬如騰訊視頻用戶畫像，由於字段個數多而且標籤個數不是固定的，因此需採用 schema-less 的引擎，業務原本使用 ES+ClickHouse，但遇到數據導入不穩定、查詢維度或標籤組合過多導致服務不可用，後來業務遷移到 Hermes，由於 Hermes 在該領域的成熟度，很輕易地解決了他們這些痛點，並且還原生地支持了他們用戶包導入關聯分析的產品需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，微信日誌每天萬億的實時接入量，除了滿足讀寫性能的要求，可用性要達到 4 個 9，機器資源成本是社區 ES 的一半。還有廣告實驗系統，業務原本使用一個自研的引擎，但是擴縮容不靈活、增加維度或維度基數膨脹後性能會急劇下降，已不能滿足後續的業務增長的預期和實驗系統的設計，因而促使業務遷移到 Hermes，當時經過測試性能提升 10 倍，後面業務還上線了一些廣告維度的指標，如 adjust_cost、超收欠收比、空耗率等，這些在過去都只能通過離線方式計算。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"至於實時分析系統相關的技術，大家容易想到的包括 SQL 編譯優化、CodeGeneration、向量化計算、列式存儲、分佈式緩存，幾乎各個引擎都用到了，但在實踐過程中細節纔是魔鬼。譬如我們在做數據緩存的時候，由於 Hermes 是用 Java 實現的，所以應該用堆外內存來做緩存，避免 GC 所帶來的壓力。但我們實現下來後，"},{"type":"text","marks":[{"type":"strong"}],"text":"發現數據從堆外拷貝到堆內存產生消耗不可忽視"},{"type":"text","text":"，並且跟數據量成正比的。而在海量數據的分析場景中，這些消耗會非常突出，是不能接受的。於是我們採用了全鏈路堆內內存複用的方案，其中在向量化計算模型中，數據掃描和計算都以固定大小的page爲單位，每個page複用內存進行傳遞和計算，對內存使用和 GC 壓力的優化是十分顯著的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"又譬如列存裏的稀疏索引，通常業界採用統計信息，如 max、min、null num 等，但在查詢分析時這些統計信息很難被應用上，因此需要研發適合在大數據場景的更細粒度的索引，但這些細粒度索引提升查詢性能，同時增加內外存的空間，導致機器資源成本比其他競品要高，受到業務和運管的挑戰。但通過對數據特性的挖掘和和技術實驗，"},{"type":"text","marks":[{"type":"strong"}],"text":"最終使用 LZ4+BitShuffle 壓縮技術使存儲空間降低 50%"},{"type":"text","text":"，"},{"type":"text","marks":[{"type":"strong"}],"text":"且對查詢性能無損；另外通過對冷熱數據索引降級，以釋放 25% 的存儲空間。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於鄧老師來說，他們在研發過程中的坑，首先是要找準賽道，然後纔是往技術深處走；往技術深處走的時候，發現數據從堆外拷貝到堆內存產生消耗不可忽視，然後開始採用了堆內內存複用的方案，結合向量化的計算模型去解決。我們從別人的實踐中，總是能發覺一些可以借鑑的點；對於鄧老師團隊來說，還是有很多架構實現的經驗給到大家。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"經驗覆盤"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：Hermes 技術架構的實現過程中，你認爲有哪些關鍵因素？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"鄧啓斌："},{"type":"text","text":" 對非在線的數據分析業務來說，易用性和穩定性是非常重要的，甚至高於性能的體驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"易用性分兩方面:"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一方面是 Hermes 的使用接口，提供了 JDBC 接口、RESTful 接口，業務使用起來門檻低，另外提供了安全管理、Session 管理、限流管控，業務免於考慮這些實現，從而降低業務系統的複雜度；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另一方面是自動化運營，關鍵點是業務接入的便捷性和可評估性，以及運營診斷能力。業務接入的便捷性是業務通過可視化方式自助接入來自於 MQ、HDFS、hive 上的數據，快速可查；可評估性是在業務接入時在運營系統上執行標準化的測試流程，後者根據數據特性、查詢 Pattern、QPS 要求，自動調整表的索引類型、分片數、查詢引擎的參數，以及輸出性能測試報告和線上的機器資源評估；至於運營診斷能力就是業務能直觀分析慢查詢、異常查詢原因，結合查詢診斷從而調整使用方式。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在穩定性上，Hermes 能基於節點統計信息進行平滑限流，以及全鏈路內存複用的管控，能避免資源被打滿導致的整個服務不可用；另外，實時分析的一大痛點是大查詢把資源跑滿，小查詢得不到資源而餓死，Hermes 在資源隊列管控上，實現了高精度的 CPU 和內存使用統計，以及數據塊粒度的執行單元的調控。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"而索引和列存是實時分析性能的基石"},{"type":"text","text":"，根據我們多年運營經驗的總結，"},{"type":"text","marks":[{"type":"strong"}],"text":"大多數的業務之間的數據和分析場景是有共性的"},{"type":"text","text":"，從而對它們的查詢模式和資源消耗進行抽象設計，實現了多種類型的索引和列存，包括低基數列存、壓縮型列存、併發型列存，以及跳錶索引、倒排索引、主鍵索引和LBS索引等，做到對大部分業務場景的覆蓋，滿足業務的功能和性能要求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"InfoQ：在 Hermes 研發過程中，有哪些經驗你想和大家分享。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"鄧啓斌："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"首先是要建設一個完善的運營系統，集羣管控、監控告警是基礎運維能力，而業務接入自助化、業務數據和查詢走勢監控、異常查詢診斷分析可解放研發和運營同學的生產力，從而使得大家可聚焦在底層技術研發和業務優化。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"其次是如何優化資源成本，實時數據分析的場景由於數據量和計算量都非常大，並要求低延遲的響應，機器資源非常可觀，動輒大幾十 \/ 幾百臺機器、SSD 硬盤，往往比業務線上的系統的機器資源還要高。因此資源成本是一個非常重要的優化方向，大家在開發和設計實時分析系統時，需要關注冷熱數據分級、存算分離、資源彈性伸縮的架構，以支撐未來幾年業務的增長趨勢和系統技術演進。以下是我們線上業務成本優化的幾點實踐："}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過對數據冷熱分級，爲微信支持、財付通海量的歷史數據節省了將近 70% 的存儲成本。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過多租戶資源隔離，使得在面臨騰訊雲不同的日誌查詢場景下，做到了不同的業務互相不干擾，且可以動態規劃資源。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過存算分離方案，使得數據在下沉到廉價存儲時，順利的完成了上 P 規模的數據在線遷移，而對線上業務無感知。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過對 Hermes 實時分析平臺的研發背景的瞭解，以及研發過程中的坑的具體分析，作爲開發者，你該怎麼借鑑經驗和避免研發過程中的坑呢？作爲用戶，你又有哪些新的思考？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"歡迎你在 4月22-24日 來到 "},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/beijing\/?utm_source=web&utm_campaign=9&utm_medium=infoq&utm_content=dengqibin","title":"xxx","type":null},"content":[{"type":"text","text":"QCon全球軟件開發大會（北京站）"}]},{"type":"text","text":"“"},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/beijing\/track\/1004?utm_source=web&utm_campaign=9&utm_medium=infoq&utm_content=dengqibin","title":"xxx","type":null},"content":[{"type":"text","text":"大數據實時計算與分析"}]},{"type":"text","text":"“專題，"},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/beijing\/presentation\/3265?utm_source=web&utm_campaign=9&utm_medium=infoq&utm_content=dengqibin","title":"xxx","type":null},"content":[{"type":"text","text":"鄧啓斌"}]},{"type":"text","text":"老師的分享"},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/beijing\/presentation\/3265?utm_source=web&utm_campaign=9&utm_medium=infoq&utm_content=dengqibin","title":"xxx","type":null},"content":[{"type":"text","text":"《騰訊萬億實時分析引擎 Hermes 的實現與應用"}]},{"type":"text","text":"》，相信你會有更多收穫，本專題下還有，螞蟻集團資深技術專家潘毅老師分享的"},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/beijing\/track\/1004?utm_source=web&utm_campaign=9&utm_medium=infoq&utm_content=dengqibin","title":"xxx","type":null},"content":[{"type":"text","text":"《自研分佈式數據庫 OceanBase 對 HTAP 的探索》"}]},{"type":"text","text":"、阿里巴巴資深技術專家金曉軍老師分享的"},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/beijing\/track\/1004?utm_source=web&utm_campaign=9&utm_medium=infoq&utm_content=dengqibin","title":"xxx","type":null},"content":[{"type":"text","text":"《HSAP 服務\/分析一體化系統設計與實現》"}]},{"type":"text","text":"以及百度資深研發工程師陳明雨老師分享的"},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/beijing\/track\/1004?utm_source=web&utm_campaign=9&utm_medium=infoq&utm_content=dengqibin","title":"xxx","type":null},"content":[{"type":"text","text":"《PB級 MPP 數據庫 Apache Doris 系統架構解析及實時數據探索》"}]},{"type":"text","text":"。期待在 QCon 現場面基。"}]}]}