30天完成數據庫替換、頂住10倍大促流量壓力,易車是如何做到的?

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一年一度的雙十一又雙叒叕來了,給技術人最好的禮物就是大促技術指南! 而經過這些年的發展,大促早已不僅僅侷限於電商行業,現在各行各業其實都會採用類似方式做運營活動,汽車界有 818,電商有 618 、11.11 等等,各種各樣的大促場景,對包括數據庫在內的基礎軟件提出了很多新挑戰,同時也積累了諸多最佳實踐。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在雙十一到來前,PingCAP 與汽車之家、易車網、京東、中通等用戶展開一系列深入探討,希望爲大家揭祕逐年飆升的銷量背後隱藏着什麼樣的技術難題?用什麼技術架構才能平穩地扛住流量洪峯?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/f8\/50\/f8803d220107943930ac8efc16c9ab50.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"汽車界的“大促”狂歡節"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"成立於 2000 年的易車,是國內最早一批汽車互聯網平臺企業之一,爲汽車用戶提供專業、豐富的互聯網資訊服務,提升用戶在選車、購車、用車和換車過程中的全程體驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在今年“ 818 ” 期間,易車與浙江衛視聯合推出了一臺綜合汽車工藝秀、明星歌舞演出和明星綜藝秀的車界“春晚”——“易車超級 818 汽車狂歡夜”。在爲汽車用戶帶來視聽盛宴、購車福利的同時,晚會還推出超 150 臺半價車的超值福利,觀衆可邊看晚會邊搶 5 折售賣的好車,同時還有購車紅包、抵扣券、車款直降等多重優惠,得到實實在在的購車福利。截至晚會結束,全平臺觀看直播人次達2.24億,獲得線上訂單4.39萬,累計成交額(GMV)64.2億元。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"易車的大促首秀"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在易車的 818 狂歡節中,數據庫的應用場景有很多,其中實時數據看板是主要的應用業務之一。看板可以實時展示易車 818 購車節的專題、活動、流量、線索、互動等數據表現,是大數據平臺的整體數據輸出。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於易車的這場汽車狂歡夜是臺網互動的直播活動,搖一搖(紅包、半價車、易車幣)和主會場分會場直播節目的投票都是用戶參與度最高、數據流量最大的環節。在整個活動過程中,不僅要求數據庫能夠存儲海量數據,同時還要求能夠應對高併發、低延遲等場景需求。這裏的數據庫不僅會作爲數據存儲的介質,還會作爲實時計算的數據源頭,配合流量數據,實現秒級數據實時播報。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據庫和 Flink 是整個系統中非常重要的兩個組件,Flink 的數據來源包括數據庫和業務流量數據,所以數據庫不僅要滿足數據秒級實時推送,還要支持 Flink 高併發的讀寫請求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"易車數據庫負責人田震坦言,易車今年是第一次做大促,沒有太多經驗,量也不好預估,很多需求都是在最後才提出。爲了保險起見,DBA 團隊在設計大促方案時做了降級方案,但誰都不希望真的實施降級,這對用戶的體驗太不友好。所以整個 DBA 團隊將主要精力放在壓測上,並按照平時的兩個數量級(100倍)來規劃數據庫壓測方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一開始,易車考慮的首選數據庫依然是 MySQL。但在壓測過程中,爲了保證計算結果的實時性,實時任務會頻繁對數據庫進行大批量數據寫入,MySQL 主從延遲高,極端情況下引起的 MySQL 主從切換,切換時間過長,導致數據庫出現短暫不可用狀態。同時,實時任務會持續寫入大量數據,引起磁盤爆滿。在分秒必爭的直播過程中這肯定是無法容忍的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在情勢急迫下,田震想到了 TiDB。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"“在游泳中學游泳” TiDB 臨危受命"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"實際上,田震很早就接觸過 TiDB ,那時候他一度認爲 TiDB 是一款海外產品,瞭解 TiDB 主要也是從海外社區開始的。但出於謹慎的原因,田震希望將產品研究透徹再正式上線。本次大促給了雙方合作一個完美的契機,他形容這一過程就像是“在游泳中學游泳”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TiDB 社區的技術支持給了易車 DBA 們非常重要的幫助,從七月正式立項,僅用了不到一個月時間就完成了選型、方案設計、壓測、上線部署,並在“818”中有驚無險地將大促流量平穩承載過來。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/c8\/ce\/c8bdee9b092e3fbe0a6e5bae36bb62ce.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"818 汽車狂歡數據看板業務架構圖"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在整個 818 活動中,TiDB 被用作 818 汽車狂歡節數據看板的核心數據庫。易車準備了兩套 TiDB 集羣,和實時計算的主備方案一一對應。業務研發通過雙寫的方式把數據同時寫入兩個集羣,一部分業務的查詢連接集羣 1 ,另一部分業務的查詢連接集羣 2,當其中一個集羣出現問題,應用端就會切換到另外一個集羣。兩個 TiDB 集羣都是部署了 3 個 TiDB Server、3 個 PD Server、6 個 TiKV 節點、2 個 TiFlash 節點。此外,還準備了 4 臺機器做擴容以免數據量暴漲集羣支撐不了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最終,易車 818 汽車狂歡節期間數據量達到了平時的 10 倍以上,在直播最後蔡徐坤出場時,數據庫流量更是直接翻了四倍,差點啓用事先準備好兜底用的一鍵擴容方案。在整個過程中,818 汽車狂歡數據看板業務 SQL 999 始終控制在 8ms 以內,SQL 99 在 3ms 左右,QPS 達到 62k。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/1b\/5f\/1bc5e8b1c91ca9106a239baf88bff15f.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"紅包搖一搖業務架構圖"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同時,TiDB  也作爲容災方案被應用在紅包搖一搖業務中,避免由於業務流量暴漲引起 MySQL 不可用的情況。一旦發生不可用,業務方可以直接將數據庫切換到 TiDB。TiDB 在整個業務中需要作爲數據源、實時計算維表和實時計算結果存儲引擎三個角色。TiDB 通過 TiCDC 將數據實時推送到 Kafka 中,爲了保證 TiCDC 穩定高效,易車爲 TiDB 中的每個庫創建了一個 TiCDC 任務,將數據實時推送到指定 Kafka 中,然後 Flink 負責將同一個 TOPIC 中的屬於不同庫表的數據進行解析,分流到庫表對應的 TOPIC 中,提供給實時計算業務使用。實時計算任務消費 Kafka 中的 TiDB 數據進行業務邏輯計算,同時還需要從 TiDB 中查詢對應的維度數據,最終將計算結果再輸出到 TiDB 中。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"高速增長的挑戰:技術棧統一"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大促的極限場景總能發現一些平時注意不到的問題,在易車的高速發展中,很多業務爲了快速迭代、迅速上線,引入了非常多的技術棧,如 Lambda 、 Kappa 等大數據架構,Kylin、Druid、Clickhouse 等實時數倉等等。但易車 DBA 團隊卻只有 6個人,管理如此多的技術棧無疑是一個很大的挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"統一技術棧成爲易車 DBA 團隊的最佳選擇,藉着這次大促的機會,易車希望用 TiDB 上線取代 Kylin、Druid、Clickhouse ,簡化技術棧,DBA 團隊也能將注意力放回專職工作上。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TiDB  的 HTAP 架構是一個混合了交易型事務和分析處理的融合架構,由於都是在同一個架構、同一套數據中,解決了易車實時數倉數據流延遲的問題。數據不用再從 OLTP 數據庫複製出來,經過漫長的 ETL 清洗等過程進入分析工具。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而 TiDB 對 MySQL 的完美兼容,對 DBA 和開發者意味着不需要做什麼改變,只要會 SQL 就能使用。在以往應用 Hadoop 或 Spark 時,由於學習成本比較高,對使用造成了一定壁壘。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經此一役,易車的業務方對 TiDB 平添了許多期待與信任。未來,易車的廣告、媒體平臺、網站、投放數據、廣告效果都希望能夠實時看到,田震希望借用 TiDB 覆蓋易車整個混合技術棧的場景,與其他數據流進行打通,這些都需要 TiDB HTAP 對實時數倉進行支持。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大促對於企業而言,除了支持業務創新,也是一次對自身技術架構的大練兵和全鏈路演練。通過大促的極致考驗,企業的 IT 架構、組織流程、人才技能都獲得了大幅提升。而在大促中的經驗和思考,也會加速企業日常的業務創新節奏,提升技術驅動的創新效率,打造增長新引擎。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"相關文章:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/www.infoq.cn\/article\/PzNajq3us9S46jr6LSBx","title":"xxx","type":null},"content":[{"type":"text","text":"《突破極限,京東雲數據庫打造急速秒殺體驗》"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章