支撐 40億數據總量、分鐘級報表：中通大數據平臺在大促中的進化

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一年一度的雙十一又雙叒叕來了，給技術人最好的禮物就是大促技術指南！ 而經過這些年的發展，大促早已不僅僅侷限於電商行業，現在各行各業其實都會採用類似方式做運營活動，汽車界有 818，電商有 618 、11.11 等等，各種各樣的大促場景，對包括數據庫在內的基礎軟件提出了很多新挑戰，同時也積累了諸多最佳實踐。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在雙十一到來前，PingCAP 與汽車之家、易車網、京東、中通等用戶展開一系列深入探討，希望爲大家揭祕逐年飆升的銷量背後隱藏着什麼樣的技術難題？用什麼技術架構才能平穩地扛住流量洪峯？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#2C2C2C","name":"user"}}],"text":"大促中，大家買買買後最期盼的事情就是收到快遞。"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"成立於 2002 年的中通快遞，是一家以快遞爲主體，以國際、快運、雲倉、商業、冷鏈、金融、智能、星聯、傳媒爲輔的綜合物流服務品牌。2020年，中通完成業務量 170 億件，市場佔有份額達到 20.4%。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"整個快遞的生命週期、轉運週期可以用五個字來概括——"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"收、發、到、派、籤"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"："}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/30\/bf\/30f96878b10646b1a891111a6fb0eabf.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"支撐整個快遞生命週期的平臺就是中通大數據平臺。"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"中通從離線到實時的數據兼容再到數倉，有着一套比較完善的大數據平臺體系。ETL 建模也會依託該大數據平臺，最終通過大數據平臺對外提供數據應用的支持以及基於離線 OLAP 分析的支持，整個數據建模的頻率可以支持到半小時級別。在這個完善的大數據平臺基礎上，中通開始更多地思考"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"如何增強實時多維分析能力"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/11\/24\/117d4964fbb4ffda7ac47cd5yy654124.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"中通與 TiDB 的結緣是在 2017 年調研分庫分表場景時開始的。當時中通分庫分表達到 16000 張表，業務上已經無法再繼續擴展下去。2018 年底，中通開始測試 TiDB 2.0，主要關注的是大數據量的存儲，以及分析性能。2019 年年初，中通上線了生產應用的支持。目前生產上穩定的版本是 TiDB 3.0.14 。2020 年底，中通開始測試 TiFlash，目標期望有兩點"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"：一是提高時效，二是降低硬件使用情況"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1.0 時代：滿足需求"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"1.0 是"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"滿足需求"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"的時代，業務需求主要包含以下幾點："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"業務發展非常快，數據量非常大，每筆訂單更新有 5-6 次，操作有峯值；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"做過調研的技術方案，很難支撐多維分析的需求；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"業務方對數據分析的週期要求比較長；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"對分析時效要求也很高；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"單機性能瓶頸，包括單點故障、風險高，這些也是在業務上不能忍受的；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"除此之外，QPS 也很高，應用要求毫秒級響應。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"技術需求方面，中通需要打通多個業務場景 + 多個業務指標；需要"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"強一致的分佈式事務"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"，在原有業務模式下切換的代價很小；還需要"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"對整個分析計算工程化"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"，下線原來的存儲過程；能夠"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"支持高併發的讀寫、更新"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"；能夠支持"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"在線的維護"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"，保證單點的故障對業務是沒有影響；同時，還要"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"與現有的大數據技術生態緊密結合"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"在一起，做到分鐘級的統計分析；最後是中通一直在探索的，即要建立 100 + 列以上的大寬表，"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"基於這張寬表，要做到多維度的查詢分析"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/de\/9f\/de06f177869915998c2d01df7a6e889f.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"目前 TiDB 在中通應用的一些落地場景"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"時效系統應用場景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"其中，時效系統是中通原有的一套系統，現在已經進行了重構。這套系統原來的存儲和計算主要是依賴 Oracle 設計的，計算依賴存儲過程。這套架構也比較簡單，一邊是消息的接入，一邊是負載。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/ee\/e1\/ee75bec1644aa9366d42329d400006e1.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"隨着業務體量的增長，這一套架構的性能已經逐漸出現瓶頸。在對這套系統進行架構升級時，中通把整個存儲遷移到 TiDB 上，整個計算遷移到 TiSpark。消息接入依賴於 Spark Link，通過消息隊列最終到 TiDB。TiSpark 會提供分鐘級的一些計算，輕度彙總會到 Hive，中度彙總會到 MySQL。基於 Hive，通過 Presto 對外提供應用的服務。"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"相較原來關係型數據庫的分表，無論是 OLTP 還是 OLAP 都極大地降低了開發的工作量，並且和現有的大數據生態技術棧相融合。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/7a\/2a\/7afa51dd0e76940d1224967a39302d2a.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"1.0 時代中通的數據庫系統架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"遷移帶來的收益有很多：第一是"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"容量的增長"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"，原來的數據中心有三倍的富餘，已有系統數據存儲週期增加到三倍以上；第二，在"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"可擴展性"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"方面，支持在線橫向擴展，運維可以隨時上下計算和存儲節點，應用的感知很小；第三，滿足了"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"高性能的 OLTP"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":" 業務需求，查詢性能雖略有降低的，但是符合業務需求；第四，數據庫單點壓力沒有了，"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"OLTP 和 OLAP 實現“分離”"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"，互不干擾；第五，"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"支持了更多維度的分析需求"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"；第六，整體架構看起來比原來更清晰，"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"可維護性增強"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"，系統的可擴展性也增強了許多。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"大寬表應用場景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"另一個場景是中通一直在做的寬表的建設與摸索"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"。其實之前中通測過很多系統，包括 Hbase、Kudu。Kudu 的寫入性能還是很不錯的，但是其社區活躍度在國內一般。同時，中通使用 impala 作爲 OLAP 查詢引擎，但主流使用的是 Presto，兼容性有待考慮，也很難滿足所有業務場景需求。此外，中通的業務特性要求系統能夠快速地計算分析幾十億的數據，並能同步到離線的集羣裏與 T+1 數據做融合，還要能提供給數據產品和數據服務直連拉取明細數據。最後是海量數據的處理，中通有很多消息源的接入，需要針對每一票進行全鏈路路由和時效的預測，定位到每一票的轉運環節，數據量很大，對時效的要求也很高。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/0a\/1b\/0ae71621c28f82b0afc99ac2cff1061b.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"中通的大寬表建設"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"目前，寬表已經建設有 150 多個字段。數據來源於 10 多個 Topic 。主要的項目接入是通過 Flink 和 Spark ，打通了各個業務產生的數據，彙總到 TiDB 形成業務寬表。額外一部分，依賴於 TiSpark，從業務寬表輸出分析結果，同步 3 億條數據到 Hive。此外，還提供了十分鐘級別的實時數據建設和離線 T+1 的整合。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#888888","name":"user"}}],"text":"中通目前的集羣規模"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"在使用過程中，中通也遇到了一些問題，總結起來就是"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"量變引起質變"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"。第一，熱點問題。索引熱點在目前情況下表現較爲突出，因爲中通的業務量規模十分大，操作存在高峯，在大時候該熱點問題表現特別明顯。第二，內存碎片化問題。在之前的低版本里，在穩定運行了一段時間後，因爲有業務特性和大量的更新和刪除，導致內存碎片化比較嚴重，這個在反饋給了 TiDB 後，已經修復了這個問題。第三，着重介紹一個參數——TiFlash 讀取 index 的參數。通過測試，當讀取的數據量\/總數據量大於 1\/10 的時候，建議該參數關閉。爲什麼這麼說？因爲 Test 數可能會變少，但是單位 Test 過渡的時間會變長。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"運維監控"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"使用 TiDB 後會發現它的監控指標特別豐富，使用了流行的 Prometheus + Grafana ，多而全"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"。之前，中通因爲在支持線上業務的同時，還會有開發人員來查數據，遇到了 SQL 把 TiKV Server 拉掛的情況。針對這個問題以及監控的問題，中通進行了一些開發定製。第一，兼容線上特殊帳號的慢 SQL，會自動殺掉，並通知到相應的應用負責人。第二，中通開發了支持 Spark SQL 去查詢 TiDB 的工具，併發和安全性在開發的過程中得到一些保障。此外，中通還會把一些額外的核心指標，接入到自研的監控體系。核心的告警會電話通知到相關的值班人員。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/d2\/9a\/d2524d8abfbe4e9fa55d633d0350969a.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"去年雙十一期間，中通訂單量突破 8.2 億，整個業務規模突破 7.6 億，雙十一當天的 QPS 峯值達到 35 萬 +。整個雙十一期間，數據的更新體量達到了數千億級別，整個集羣上運行的 TiSpark 任務是 100 多個，支持的在線應用 7 個。"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"整個分析的時效在 10 分鐘以下達到了 98% ，整個分析的數據週期達到 7-15 天。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.0 時代：HTAP 提升"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"2.0 時代的主要特點是"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":" HTAP 的提升"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"。中通應用 HTAP 主要來自於業務方需求的升級：基於業務方的需求，中通在 2.0 時代進行了一次架構再升級。首先，"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"引入了 TiFlash 和 TiCDC"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":" 。這帶來的收益其實是增強了時效，部分分析進入了分鐘級級別，降低了 Spark 集羣資源的使用情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/d5\/bf\/d512c1f9389556944b592831827e68bf.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"2.0 時代中通的數據系統架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"下圖是 TiSpark 和 TiFlash 的對比，中通線上有兩套集羣，一個基於 3.0，一個基於 5.0。簡單地對比一下 3.0 和 5.0 的情況：3.0 主要的分析是基於 TiSpark，5.0 是基於 TiFlash 。目前 3.0 集羣有 137 個物理節點， 5.0  有 97 個節點。整個運行的週期中，3.0 是 5 - 15 分鐘，基於 5.0 的 TiFlash 已經做到 1-2 分鐘，整個 TiKV 的負載降低是比較明顯的。另外， 在 3.0 上 Spark 的資源大概有 60 臺，而在 5.0 上，線上的加上在測試的，大概有 10 臺就足夠了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/62\/1a\/62680ce01e5f8821e96bd14b310f451a.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"在整個測試周期中，生產的集羣是 3.0 ，4.0 的測試周期其實是非常短的。在測試時，業務的場景有一些維表 Join 的情況，當時 4.0 對 MPP 沒有支持，對一些函數的支持可能也不是那麼完善，測試結果不是很理想。對 HTAP 的測試主要是在 5.0 階段，5.0 已經支持 MPP ，對函數的支持也越來越豐富。目前中通生產上應用的版本是 TiDB 5.1 。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/a3\/e1\/a3c367708d1d7a0352060aed1c2a60e1.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"上圖右側是整個 5.0 集羣在 618 期間的負載情況。在剛剛結束的 618 中， 5.0 上線的一些任務已經在支持 618 移動端的大促看板。中通有 6 個核心的指標是基於 TiFlash 計算的。"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"集羣響應整體平穩，報表達到了分鐘級以內的時效。"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"整體的數據體量在 40 億 - 50 億 +，報表分析數據達到 10 億 +。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.0 時代：展望未來"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/43\/10\/43b4c15bdbc06bd2c0fc2b7b93d0dc10.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"第一是監控"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"。提到監控，由於中通的集羣比較大，所以面臨的問題和遇到的問題可能會多一點。大集羣的實例多，指標加載慢，排查問題的效率得不到保障。監控雖然很全，但是出了問題的時候無法快速定位到問題；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"第二是解決"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"執行計劃"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"偶發不準的問題。這種偶發不準有時候會影響到一些線上的負載相互影響，拉高集羣的指標，導致業務相互影響。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"第三是實現"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"自動清理"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"。目前中通數據的清理是通過自己寫成 SQL 清理的，但是過期數據清理比較麻煩。希望之後可以支持舊數據自動 TTL。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"第四，隨着 5.0 "},{"type":"text","marks":[{"type":"color","attrs":{"color":"#C31A0A","name":"user"}},{"type":"strong"}],"text":"列式存儲"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":"的引入，中通計劃把 TiSpark 的任務逐漸全部切到 TiFlash 上面，期望達成提高時效和降低硬件成本的目標。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#333333","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大促對於企業而言，除了支持業務創新，也是一次對自身技術架構的大練兵和全鏈路演練。通過大促的極致考驗，企業的 IT 架構、組織流程、人才技能都獲得了大幅提升。而在大促中的經驗和思考，也會加速企業日常的業務創新節奏，提升技術驅動的創新效率，打造增長新引擎。"}]}]}