華爲自研PB級分佈式時序數據庫揭祕第一期：初識GaussDB(for Influx)

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"前言","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着雲計算規模越來越大，以及物聯網應用逐漸普及，在物聯網（AIoT）以及運維監控（AIOps）領域，存在海量的時序數據需要存儲管理。以華爲雲監控服務（Cloud Eye Service，CES）爲例，單個Region需要監控7000多萬監控指標，每秒需要處理90萬個上報的監控指標項，假設每個指標50個字節，一年的數據將達到PB級。另以地震監測系統爲例，數萬監測站點24小時不間斷採集數據，平均每天要處理的指標數據達到TB級，一年的數據同樣達到PB級，並且數據需要永久存儲。傳統的關係型數據庫很難支撐這麼大的數據量和寫入壓力，Hadoop等大數據解決方案以及現有的開源時序數據庫也面臨非常大的挑戰。對時序數據實時交互、存儲和分析的需求，將推動時序數據庫在架構、性能和數據壓縮等方面不斷進行創新和優化。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"GaussDB(for Influx)時序數據庫依靠華爲在數據存儲領域多年的實踐經驗，整合華爲雲的計算、存儲、服務保障和安全等方面的能力，大膽在架構、性能和數據壓縮等方面進行了技術創新，達到了較好的效果，對內支撐了華爲雲基礎設施服務，對外以服務的形式開放，幫助上雲企業解決相關業務問題。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"雲原生存儲與計算分離架構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/76/760c41be02fae8fb96c20b753934a9eb.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"GaussDB(for Influx)接口完全兼容InfluxDB，寫入接口兼容OpenTSDB、Prometheus和Graphite。從架構上看，一個時序數據庫集羣可以分爲三大組件。它們分別是：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Shard節點","attrs":{}},{"type":"text","text":"：節點採用無狀態設計，主要負責數據的寫入和查詢。在節點內，除了分片和時間線管理之外，還支持數據預聚合、數據降採樣和TAG分組查詢等專爲時序場景而優化的功能。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Config集羣","attrs":{}},{"type":"text","text":"：存儲和管理集羣元數據，採用三節點的複製集模式，保證元數據的高可靠性。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"分佈式存儲系統","attrs":{}},{"type":"text","text":"：集中存儲持久化的數據和日誌，數據採用三副本方式存放，對上層應用透明。存儲系統爲華爲自研，經過多年產品實踐檢驗，系統的高可用和高可靠性都得到了驗證。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/19/191d5dffd677d303166ce513b470c749.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相比InfluxDB等開源時序數據庫，採用存儲與計算分離的雲原生數據庫設計具備以下優勢：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"容忍N-1節點故障，更高可用","attrs":{}},{"type":"text","text":"。存儲與計算分離，可以複用成熟的分佈式存儲系統，提供系統的極致可靠性。時序數據通常會持續高性能寫入，同時還有大量的查詢業務，任何系統故障導致業務中斷甚至數據丟失都會造成嚴重的業務影響，而利用經過驗證的成熟的分佈式存儲系統，能夠顯著的提升系統可靠性，降低數據丟失風險。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"分鐘級計算節點擴容，秒級存儲擴容","attrs":{}},{"type":"text","text":"。解除在傳統Shared Nothing架構下，數據和節點物理綁定的約束，數據只是邏輯上歸宿於某個節點，使的計算節點無狀態化。這樣在擴容計算節點時，可以避免在計算節點間遷移大量數據，只需要邏輯上將部分數據從一個節點移交給另一個節點即可，可以將集羣擴容的耗時從以天爲單位縮短爲分鐘級別。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"消除多副本冗餘，降低存儲成本","attrs":{}},{"type":"text","text":"。通過將多副本複製從計算節點卸載到分佈式存儲節點，可以避免用戶以Cloud Hosting形態在雲上自建數據庫時，分佈式數據庫和分佈式存儲分別做3副本複製導致總共9副本的冗餘問題，能夠顯著降低存儲成本。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f2/f25563bfbd9f0722f9962debac536acd.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"GaussDB(for Influx)採用雲原生存儲與計算分離架構，具有支持億級時間線、極致寫入性能、低存儲成本、高性能多維聚合查詢和極致彈性擴縮容等5大特性。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"支持億級時間線","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e1/e1732aa4a5e9c19a645d8514ee6d6d47.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在時序數據庫系統中，存在大量併發查詢和寫入操作，合理控制內存的使用量顯得十分重要。開源時序數據庫VictoriaMetrics和InfluxDB在寫入數據的時間線增加到千萬級別時，進程會因內存耗盡而OOM退出。爲了避免寫入海量時間線數據導致內存資源被耗盡，GaussDB（for Influx）做了如下優化：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在內存分配上，大量使用內存池複用技術，減少臨時對象內存申請，降低內存碎片。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在內存回收上，實現算法根據內存負載，動態調整GC頻率，加快內存空間回收。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在單查詢上，實行Quota控制，避免單查詢耗盡內存。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在緩存使用上，針對不同節點規格提供不同的最優配置。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經過改進，在海量時間線下，系統寫入性能保持穩定，大幅超出InfluxDB開源實現。對於涉及海量時間線的聚合查詢，如高散列聚合查詢，查詢性能提升更爲顯著。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"極致寫入性能：支持每天萬億條數據寫入","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/29/29bfbcf79f77e7f609901b1dd1213acf.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相比單機模式，集羣模式可以將寫入負載分散到集羣中各個計算節點上，從而支持更大規模的數據寫入。GaussDB（for Influx）支持每天萬億條數據寫入，在工程實現上進行了以下優化：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先，時序數據按照時間線做Hash Partition，利用所有節點並行寫入，充分發揮集羣優勢。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其次，Shard節點採用針對寫場景優化的LSM-Tree佈局，寫WAL後確保日誌持久化，再寫入內存Buffer即可返回。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後，數據庫多副本複製卸載到分佈式存儲，降低計算節點到存儲節點的網絡流量。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在大規模寫入場景下，GaussDB（for influx）的寫入性能線性擴展度大於80%。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"低存儲成本：只需1/20的存儲成本","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在時序數據庫面對的AIOps運維監控和AIoT物聯網兩個典型應用場景中，每天會產生數GB甚至數TB的時序數據。如果無法對這些時序數據進行很好的管理和壓縮，那將會給企業帶來非常高的成本壓力。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"GaussDB（for Influx）對數據採用列式存儲，相同類型的數據被集中存儲，更有利於數據壓縮。採用自研的時序數據自適應壓縮算法，在壓縮前對數據進行抽樣分析，根據數據量、數據分佈以及數據類型選擇最合適的數據壓縮算法。在壓縮算法上，相比原生的InfluxDB，重點針對Float、String、Timestamp這三種數據類型進行了優化和改進。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Float數據類型","attrs":{}},{"type":"text","text":"： 對Gorilla壓縮算法進行了優化，將可以無損轉換的數值轉爲整數，再根據數據特點，選擇最合適的數據壓縮算法。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"String數據類型","attrs":{}},{"type":"text","text":"：採用了壓縮效率更好的ZSTD壓縮算法，並根據待壓縮數據的Length使用不同Level的編碼方法。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Timestamp數據類型","attrs":{}},{"type":"text","text":"：採用差量壓縮方法，最後還針對數據文件內的Timestamp進行相似性壓縮，進一步降低時序數據存儲成本。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖是分別採用實際業務場景的事件日誌數據（數據集1)和雲服務器監控指標數據 (數據集2）與InfluxDB進行了數據壓縮效率的性能對比。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/30/30d957eafdf978e012d222563687648d.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"節約存儲成本並非只有數據壓縮一種辦法。針對時序數據越舊的數據被訪問的概率越低的特點，GaussDB（for Influx）提供了時序數據的分級存儲，支持用戶自定義冷熱數據，實現數據的冷熱分離。熱數據相對數據量小，訪問頻繁，被存儲在性能更好、成本較高的存儲介質上；冷數據相對數據量大，訪問概率低，保存時間較久，被存儲在成本較低的存儲介質上，進而達到節約存儲成本的目的。根據實際業務數據測算，相同數據量下存儲成本僅有關係型數據庫的1/20。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"高性能多維聚合查詢","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"多維聚合是時序數據庫中較爲常見，且會定期重複執行的一種查詢，例如AIOps運維監控場景中查詢CPU、內存在指定時間範圍內的平均值。","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"SELECT mean(usage_cpu), mean(usage_mem)\nFROM cpu_info \nWHERE time >= '2020-11-01T06:05:27Z' and time < '2020-11-01T18:05:27Z' \nGROUP BY time(1h), hostname\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9f/9f892c79c3008e18769267189018a7c8.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在提升聚合查詢整體性能方面，GaussDB（for Influx) 做了如下優化：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"採用MPP架構：一條查詢語句可以在多節點及多核併發執行。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"向量化查詢引擎：在查詢結果數據量很大時，傳統的火山模型每次迭代返回一條數據，存在過多的開銷導致性能瓶頸。GaussDB（for Influx）內部實現了向量化查詢引擎，每次迭代批量返回數據，大大減少了額外開銷。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"增量聚合引擎：基於滑動窗口的聚合查詢，大部分從聚合結果緩存中直接命中，僅需要聚合增量數據部分即可。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"多維倒排索引：支持多維多條件組合查詢，避免大量Scan數據。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"存儲摘要索引，加快數據查詢中過濾無關數據。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}}]}],"attrs":{}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/dc/dc319cb33c4ff6623a40f2dad8608552.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相同節點規格，GaussDB（for Influx）的聚合查詢性能是InfluxDB Enterprise的10倍，是Timescale的2到5倍。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"分鐘級彈性擴縮容","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3e/3eebbe9b8cdaf2f7ee5da1a1e37f3cdd.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在時序數據庫的運行過程中，隨着業務量的增加，常常需要對數據庫進行在線擴容，以滿足業務的要求。傳統數據庫中的數據存儲在本地，擴容後往往需要遷移數據。當數據量達到一定規模時，數據遷移所耗費的時間往往按天計算，給運維帶來了很大的困難。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上圖所示，每個Database邏輯上由多個Partition組成，每個Partition獨立存儲，且都可自描述。所有Partition數據都存儲在分佈式共享存儲上，數據庫Shard節點和數據沒有物理綁定關係。擴容時首先offload源節點Partition，再在目標節點assign即可。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"時序數據應該存儲在專門爲時序數據進行優化的時序數據庫系統中。華爲雲某業務從Cassandra切換到GaussDB（for Influx）後，計算節點從總共39個（熱集羣18個，冷集羣9個，大數據分析集羣 12個）降低到了9個節點，縮減4倍計算節點。存儲空間消耗從每天","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"1TB","attrs":{}},{"type":"text","text":"降低到","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"100GB","attrs":{}},{"type":"text","text":"以內，縮減10倍存儲空間消耗。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"GaussDB（for Influx）提供了獨特的數據存儲管理解決方案，雲原生的存儲與計算架構，可根據業務變化快速擴容縮容；高效的數據壓縮能力和數據冷熱分離設計，可大幅降低數據存儲成本；高吞吐的集羣，可滿足大規模運維監控和物聯網場景海量數據寫入和查詢性能要求。","attrs":{}}]}]}