高達200個應用，近8000個實例的工行MySQL轉型實踐

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文根據魏亞東老師在〖2020 Gdevops全球敏捷運維峯會〗現場演講內容整理而成。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/8b\/8bc88bcc29109584828dce441d2e0e80.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"講師介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"魏亞東，中國工商銀行軟件開發中心三級經理，資深架構師，杭州研發部數據庫專家團隊牽頭人和開發中心安全團隊成員，負責技術管理、數據庫、安全相關工作。2009年加入中國工商銀行軟件開發中心，致力於推動管理創新、效能提升，提供全面技術管控，推動自動化實施，實現業務價值的高質量快速交付；同時作爲技術專家，爲生產安全提供技術支持。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、ICBC數據庫轉型背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先介紹下工行進行MySQL轉型的背景，大概有以下四方面的考量："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/44\/44164e87d15f3a77cbe64551135d0360.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、利潤提升"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大家知道工行的利潤在四大行當中是最高的，怎麼持續做到利潤提升？簡單來說就是開源節流。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"衆所周知，國有銀行以前存量核心體系核心都是DB2，OLTP系統是Oracle，這種架構類型很成熟，但是運營費用昂貴，機器和服務隨便就要幾千萬幾億資金的支出。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着大數據時代步入成熟期，我們從數據存儲從TB發展到PB，科技業務量呈指數增長，大家可以看到線上化處理已經成爲主流，特別是在疫情開始後，這對銀行的處理能力和峯值吞吐量施加了雙重壓力，傳統集中式這種方式已經不能滿足我們的需求，我們也付不起這個成本，而基於通用的廉價的X86硬件基礎設施在業界互聯網頭部公司已經廣泛使用，可行性毋庸置疑，所以我們就開始借鑑他們這種方式來做。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、同業領先"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大家都說我行是宇宙第一大行，所以我們行的科技隊伍人數最多的，我們要做到同業領先，大致有兩點訴求："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同業競爭，之前互聯網金融所謂的金融科技或者科技金融，其業務實際上是允許打擦邊球的，這就對銀行的業務量提出了一個挑戰，要求我們的業務可以做到快速上新，而業界基於DevOps的快速交付和灰度發佈常態化，也是應運而生的產物而已；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大家對商業銀行有很高的期望，特別是國有銀行，代表着國家聲譽，這就要求我們必須支持24小時處理業務，即客戶想用你的時候不能出問題，這就對高併發、高可用性、海量數據的處理提出更高的要求，不能宕機。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3、技術自研"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"全球化的趨勢要求技術無國界，但是從去年開始的貿易戰，以及美國的實體管制清單，我們可以看到技術是有國界的，比如之前HashiCorp發佈公告，其企業版產品禁止中國使用已經成爲一種徵兆。所以擁抱開源、提升技術自研能力，解決對商業產品的過度依賴其實是一種趨勢，但是也要注意到，不是所有的開源產品有沒有問題，大家一定要注意審閱開源許可證，比如GPL這種傳染性許可證其實風險很高。同時從銀行業務的特性來看，實現分佈式可擴展架構，做到在線擴縮容，滿足銀行對事務強一致性的處理要求，對我們的技術積累都是一種不小的挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4、長遠規劃"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我行數據庫產品的長遠規劃簡單來說就是2步走原則："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"擴展技術路線，探索新的技術產品，比如我行已經在部分應用落地GaussDB和OceanBase，後面還會試驗更多的產品；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"豐富技術積累，對業界主流產品技術發展路線會做一個持續的跟蹤，逐步轉化成我們自身的資產，以前有幸參加過中村修二先生座談會，其強調十年累積，所以技術積累是一個寂寞但是必須要做的東西。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、ICBC數據庫建設歷程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大家可以看下我行數據庫的建設歷程，我們最早是從2014年開始基於MySQL 5.5建立MySQL運維平臺。因爲是小打小鬧，並未成體系化，所以2016年纔是我行第一階段建設的開始，逐步完善周邊配套。從2018年中旬開始，是我行第二階段的建設。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一階段是探索MySQL的可能性，試點成功後就開始全面推廣，並持續完善研發和運維自動化體系。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二階段遵循兩步走的原則，一方面隨着數據庫越來越增多，運維成本大幅增長，資源利用率不高，所以我們將MySQL上雲，後面我會介紹成效，大概90%都是基於雲數據庫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另一方面，我們還要擴展技術路線，試點新產品GaussDB和OceanBase，落地5個產品場景，一是在物聯網裏做了一個創新，二是在基於其他的業務場景化。單純從應用場景來看，其可以滿足銀行業務要求，但是它們的生態環境不太好，就是配套周邊有待完善（這裏指除原廠以外的生態圈），相對MySQL來說還是有所欠缺。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/27\/274d182b235fcf58c61bfcd12bc8eb63.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、ICBC MySQL實施成效"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"來看看我們的成效，有以下四點："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/2a\/2af6da6d6a6acde2a7b9cfcf04a25ad8.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、高度自主自控能力"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們基於開源技術構建了企業級的分佈式解決方案，MySQL數據庫解決方案，滿足不同業務場景的需求。還自研運維管理平臺，實現規模節點集羣化、自動化、標準化管理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、廣泛運用於各業務場景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MySQL應用高達217個，MySQL數據庫實例接近8000個，爲主機應用下移提供數據服務支撐，經受了支撐雙十一、春節業務高峯萬級TPS的真實考驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3、同業領先數據庫雲化服務"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我行在同業率先達成了MySQL的大規模雲化部署，佔比達到90%以上。同時我們提供了一鍵式的快速供給，一鍵式的運維管理，簡化了運維操作流程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4、秒級恢復、分鐘級切換"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"多站點數據存儲，保證數據強一致性，實現同城雙活RPO=0。本地故障秒級恢復，無需人工干預，業務系統無感知，園區級故障分鐘級同城切換。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、ICBC技術路線"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我行技術路線其實應該跟大家玩法一致，我們最早是和騰訊做的交流，所以我們的技術路線有點接近於騰訊，使用應用架構的服務網關加上處理分組，配合運維管理平臺實現的。處理分組，也就是所謂的set實際上包含了聯機處理功能的服務器集合，既有中間件這類應用服務器，也包含了MySQL的數據庫服務器。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當交易請求過來時，首先通過SLB（或F5）進行負載均衡後，到達服務網關，服務網關會根據業務預設的一些字段，比如用戶ID、銀行卡號、帳號進行一致性Hash以後，分發到對應的處理分組上，由對應分組處理完成以後返回結果，從而完成交易鏈路的處理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同時我們會結合應用場景對數據進行分片，大概會做128個分片，並對分片進行合併部署。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後我們在每個MySQL服務器部署一個Agent，定時採集快照信息、監聽可用狀態等，統一上送給管理平臺，管理平臺可以生成AWR報告，也可以進行數據庫探活。如果某個數據庫被監測到有問題的時候，數據庫運維平臺就可以做到自動化的切換，實現了數據庫層面的高可用能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/07\/07c7c4dafe63e2cbbc77d5ac8b1418d5.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、ICBC技術路線-分佈式訪問層"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於分佈式訪問層，這個其實褒貶不一，因爲業界爭議很大，比如業界有阿里的TDDL，還有網易的DDB這種成熟的產品。但是我們爲了完整性，最後也是做了一個分佈式訪問層，我們的分佈式訪問層有兩種模式，但是相對較薄，與TDDL和DDB相比存在一定差距，但是壓力相對較輕，不像他們壓力會集中在訪問層的維護人員手中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們主推的是基於DBLE重構分佈式訪問層實現。我們擴展建立了配置中心，應用發版時把配置帶出來，訪問層自動下載配置並完成熱加載，實現自動化部署，規避開發人員在某個地方維護Server.xml等配置文件。當出現園區故障時，通過智能DNS自動路由到F5，然後再到DBLE集羣，實現了應用服務器到DBLE集羣的本園區就近訪問、負載均衡和故障自動轉移能力；DBLE做到無狀態設計，通過F5實現故障自動轉移，MySQL故障自動切換，聯動管理平臺，支持數據一致性校驗和自動補齊；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另一種模式我們稱之爲輕模式，類似於TDDL的sharding模式，隨應用進行部署，根據算法和配置路由到具體的數據庫節點，降低了運維難度。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/49\/499d8b26b0f13b679ec7a6cea89fe52e.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、ICBC技術路線-高可用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於高可用來說，我們的高可用歷程是學習和借鑑互聯網零頭公司的經驗發展而來，提供兩種模式，這個應該是5.7的固定模式。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"災備五級採用一主四從架構。1主庫、1本地半同步庫、2同城半同步庫和1異地異步庫；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"災備四級採用一主三從架構，1主庫、1本地半同步庫、2同城半同步庫，從經驗來看，大概性能會下降一倍，但是對於銀行來說這是必須要承擔的一個代價。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/fa\/fab8530cd463098516df9f29452a64c0.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"五、ICBC研發管控挑戰"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、問題"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"說了這麼多，大家只看到MySQL的好處，但是免費的午餐並不好喫，其對應用到的研發管控層面造成嚴峻挑戰，這一點大家務必注意，技術可行性不等於研發落地的有效執行，同時也不等於落地執行復雜度的降低。我們在進行研發管控的時候發現了3個層面的問題："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/bd\/bd7113935a403a3ab097928d2a0957d7.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"研發習慣層面。首先設計人員思維還是基於Oracle的思路，習慣用VARCHAR2這種萬能字段，但是MySQL要求表結構設計選擇用合適的字段；同時開發人員喜歡存過和奪標鏈接，動不動就5張表以上連接的複雜語句，而且不看執行計劃，在Oracle數據庫中沒什麼問題，但是MySQL不行，會掛掉；我們可以看到即便阿里有規約，慢SQL也是他們不可承受之痛；此外，開發人員習慣於大事務，這會導致主從同步延遲，對主備切換其實造成很大壓力；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MySQL有很多缺點和bug，耳熟能詳的解釋器弱，in、exsits語句對性能會造成很大影響，同時bug不斷，truncate阻塞同庫交易、Replace into自增列主鍵衝突、死鎖、Page Cleaner髒刷會阻塞IO等等；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"運維複雜度成指數級增長，設備量級提升10倍，擴大機房等物理設施實質會造成困擾，資源利用率低（CPU、存儲），所以上雲時一種必然趨勢；同時網絡流量併發大也是一個潛在隱患。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、ICBC研發管控層面"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了應對這些問題，我行在研發管控層面建立了一個完整的生態圈，這裏我建議大家也可以自己做一下。因爲最主要的一個問題是慢SQL的數量會隨着應用的增加呈現爆發性增長，這一點阿里也公開說過，當時他們也是因爲慢SQL的爆發式增長導致他們必須要建立SRE這個體系，所以我們是在設計、編碼、測試、交付後四個階段都做了一些處理，並建立了相關門禁，一環扣一環，確保問題及早發現和解決。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"設計"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在設計階段我們編寫併發布了設計指引，建立元數據的管理標準和能力提升課程。同時通過自動化的方式形成表結構設計工具和元數據管理系統的聯動體系，首先通過元數據管理系統，明確業務線和應用的字段、類型的標準，形成元數據字典，架構師設計表時，藉助設計工具聯動管理系系統選取所需要的元數據自動生成即可，最後還可以生成DDL語句，方便大家在開發環境下進行開發測試；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"編碼"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"編碼層面我們也做了自動化的門禁控制，基於Sonar體系使用Druid開發了一個數據庫檢查插件，對SQL 語句以及MyBatis文件進行解析並分析其是否符合我們的編碼規範，大概可以控制住60%左右的問題語句。一方面我們落實了安全檢查，比如SQL注入檢查，實現安全層面的保障。另一方面對它的寫法規則進行審視，提早發現不合理的寫法，畢竟MySQL跟Oracle不太一樣。通過質量門禁的方式可以確保開發人員在這個層面做到強遵守，嚴落實，以最小代價規避生產隱患；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在測試層面我們建立安全測試和性能測試的處理機制，基於OWASP的ZAP落實黑盒測試，同時做好重要交易的性能測試，確保重點交易滿足業務高併發要求，提升用戶滿意指數和幸福指數；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"交付後"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"交付後我行建立了SRE管理體系，對慢SQL進行監控治理，對大事務自動查殺，提前規避生產隱患，同時會把相關案例分析和生產問題總結成經典案例，讓大家學習，提升大家的意識，逐步建立對生產安全的一種敬畏感。最後我行積極踐行AIOps，基於全鏈路跟蹤的處理數據，實現根因分析，逐步向1-5-10的業界故障診斷目標看齊。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/48\/484bba6b82947bf7c34719b410f04cdf.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"六、ICBC系統運維"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、ICBC系統運維層面-自動化環境供給"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從系統運維層面，自動化環境供給，在2019年DAMS峯會已進行過分享，提升資源使用效率約4-5倍，加快資源部署速度，提升內部管理效率。這裏有兩個問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"IP漂移問題，我行有別於K8sOperator，底層擴展K8s實現容器的固定IP，SDN實現容器網絡資源自動化申請；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"二是IO徵用問題，SSD替換盤機，並做了RAID1，從性能來看，效果完全滿足需求。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/e8\/e8171578bc1dda765a382c86dbb69f16.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、ICBC系統運維層面-自動化運維"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從運維層面，自動化運維實際上是對數據庫的一個可用性、可靠性、性能等進行監控，涵蓋100多項監控項指標，同時基於AIOps實現1-5-10故障定位目標，及時告警。我們每個月都會對慢SQL進行常規性治理，可以看到下面這張模型圖，我們建立了一個三維雷達圖，按總數，慢SQL耗時以及建議優化方案和整改計劃，同時我們做了一個慢SQL自動查殺方案，確保慢SQL不會對生產造成巨大影響。因爲我們之前有個案例，一張表忘記建主鍵，備庫24小時都沒有追上主庫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"自動化安裝部署，我們對分佈式MySQL數據庫組件的安裝配置、高可用、數據災備、升級維護都進行了覆蓋，DBA現在應該是業界最少的了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"智能化高可用，集成了高可用切換功能，支持數據優先、同城優先的處理，融合智能告警和高可用切換動態管理，提供切換異常智能化告警和高可用保障建議能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/05\/05d87fd91d9f3e418c047dcc82653714.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3、ICBC實際場景-客戶信息系統"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"工行的客戶信息系統承載了我行客戶信息管理職能，涉及6億個人客戶和1千多萬對公客戶，總記錄數超過160億，每日2億次客戶信息維護與查詢，最高併發數7600TPS，平均交易耗時小於30毫秒，支持應用範圍同業最廣、日均訪問數量同業最多的處理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/01\/01bc03661fad813fd08baefbe4641819.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"七、ICBC分佈式數據庫的規劃"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"後續一些規劃實際上是這樣的："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ab\/ab405adabc589575a33e499df8735ce9.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、MySQL其他缺陷"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MySQL其實還有些其他的缺陷："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"GPL，它是GNU通用許可證，具有傳染性，存在風險。如果要是商用的話必須要第一時間購買它的商用許可，或者將自己軟件也同步開源，但這個實際上是很難做到；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"它無法支持分佈式事務，分佈式事務原則上是在應用層解決，增加了設計複雜度，我行建立了一個分佈式事務應用，支持三種模型，TCC、SAGA、還有XA模型，根據不同業務場景去規範應用使用。對於要求強一致性的話，比如賬務處理，我們原則上都要求使用TCC模型去進行處理；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"複雜SQL支持能力弱。我們做了一個基於Hive的通用查詢方案，可以減輕系統的負擔，業界同時還有ES的解決方案，都是比較好的處理方式。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2、探索新技術路線的挑戰"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"探索通用分佈式事務數據庫方案，可能數據庫自身沒有問題，但是從它的生態圈來說是有問題的，不夠成熟，比如PG、OB和TiDB等產品還是缺乏成熟豐富的生態體系。我們使用起來的話會很痛苦，同時國產數據庫蓬勃發展，技術路線差異很大，方向也沒有明確，所以後面我們還會繼續跟進分佈式的各個產品，去保證我們會有更多的積累。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Q&A"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q1：魏老師您好，我是百度數據庫團隊的，有兩個問題我想跟您探討一下。第一個關於開源數據庫的路線問題，之前我們做了大量MySQL方面的工作，對於PG這種類型的數據庫，我們目前是怎麼來考慮的？因爲它可能對Oracle的一些檢測更好一些，您覺得對於工行來說，後續對於開源的數據庫，MySQL和PG哪一種更好？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A1：因爲很多數據庫我們都看過，但是從生態圈來看，我們現在主流還是選了MySQL，PG就是高斯。但是從這個生態來說，PG的生態真是不好。從運維層面，這些工具鏈其實都偏MySQL支撐，所以我們現在PG還只是屬於推廣使用階段。對於高斯，我們大概是5個應用，但是MySQL的應用有217個，所以說後面我們肯定還是會去做國產化的這些改造，但是具體大規模的使用，這個可能還是要看生態圈的完善，因爲銀行畢竟要對大家負責。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q2：關於分佈式數據庫，剛纔更多的是儲存結構的數據庫，其實分佈式可能也有好多技術路線，您認爲中間件還有基於計算存儲這樣一些雲原生的架構，您覺得哪一種可能會更符合工行對這個數據的需求和規劃？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A2：你說的後兩種其實都可以。第一個是雲原生，工行雲是同業間最好的一個雲，所以我們雲原生數據庫肯定是第一個必然的規劃路線；第二個是分佈式數據庫，也是我們的一個考量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q3：基於中間件做分庫分表這種架構呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A3：其實我們數據庫分佈式訪問層比較弱，也不建議做強。因爲之前跟網易、阿里聊過，他們最後的問題都是集中在數據庫保密層，他們相當於保姆，而開發人員就不管，所以我們還是從設計開發人員層面入手，把數據庫先直接在應用層做到分佈式的拆分處理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q4：問一下MySQL有什麼強一致性是怎麼做到的？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A4：我們實際上是通過自己的一個分佈式的事務管理系統去做的事情。對於同一應用來說，它自身的強一致性其實可以保證，但是實際上銀行的一個系統是一個鏈路性的系統，所以每個節點其實最終都要保持強一致性，這種我們是通過TCC的模型去做，就要求每個節點都要求TCC的方式處理事務，最終達到強一致性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Q5：因爲我們目前也是金融公司，但是我確實沒有辦法重複是100%能過去。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A5：半同步這個其實MySQL機制已經保證了，你只要監測到它做信息擴展，到底有沒有重演，跟主步做比對，對比一下，做數據的一個檢查就可以了。其實我們在做切換的時候會去檢查從步跟主步ID是不是一樣的，重演到什麼地步，會逼迫它做強制性的重演。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另一方面，我們要求應用層面，就是有些要求比較高的應用，我們基於Redis去做，當應用發現這一層數據沒有的時候，可以切換以後，可以再做一個事務的補償。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自dbaplus社羣（ID：dbaplus）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/rNNNKSU9FvcpazG5B8pDoQ","title":"xxx","type":null},"content":[{"type":"text","text":"高達200個應用，近8000個實例的工行MySQL轉型實踐"}]}]}]}