一款優秀數據庫中間件的不完全解析

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d8/d800399b0223fc49fdafc49a7ce5144c.gif","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4e/4e375e843a31b6f08434a5327e576d21.png","alt":null,"title":"後臺提供高併發系列歷史文章階段總結版，歡迎關注","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"高併發系列歷史文章微信鏈接文檔","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"垂直性能提升","attrs":{}}]}]}]},{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.1. ","attrs":{}},{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/s?__biz=MzA4ODUzMDg5NQ==&mid=2650000954&idx=1&sn=a9ee98310e583b1712e1e64988d2a796&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"架構優化：集羣部署，負載均衡","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.2. ","attrs":{}},{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/s?__biz=MzA4ODUzMDg5NQ==&mid=2650000991&idx=1&sn=4cd73cc5aa4ccb97d9823db82737d14b&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"萬億流量下負載均衡的實現","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.3. ","attrs":{}},{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/s?__biz=MzA4ODUzMDg5NQ==&mid=2650001031&idx=1&sn=75b0eea86788b7b59c61875745b38c4c&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"架構優化：消息中間件的妙用","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.4. ","attrs":{}},{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/s?__biz=MzA4ODUzMDg5NQ==&mid=2650001031&idx=1&sn=75b0eea86788b7b59c61875745b38c4c&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"架構優化：用消息隊列實現存儲降級","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.5. ","attrs":{}},{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/s?__biz=MzA4ODUzMDg5NQ==&mid=2650001071&idx=1&sn=fe00cfd25ae6c8595bcc2aef84ed102f&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"存儲優化：mysql的索引原理和優化","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.6. ","attrs":{}},{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/s?__biz=MzA4ODUzMDg5NQ==&mid=2650001136&idx=1&sn=2585d7dcf8b0e4328fe07eca4e7fe085&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"索引優化補充篇：explain索引優化實戰","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.7. ","attrs":{}},{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/s?__biz=MzA4ODUzMDg5NQ==&mid=2650001108&idx=1&sn=5c246e6888438575f74147892671c2d1&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"存儲優化：詳解分庫分表","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.8. 本篇內容：詳解數據庫中間件","attrs":{}}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"Part1 數據庫中間件有啥用","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有一天，你去三亞玩耍，就想玩個衝浪，即時你不差錢，難道還要自己採買快艇、滑板等等裝備來滿足這爲數不多的心血來潮麼。租一個就行了嘛。這其實就是連接池的作用。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據庫中間件可以理解爲是一種具有連接池功能，但比連接池更高級的、帶很多附加功能的輔助組件，不僅可以租衝浪板，還可以提供地點推薦、上保險等等各類服務。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從網上的資料看，zdal應該算是半開源的，好像是之前開源過，但後續沒有準備維護，然後就刪除了，不過github被fork下來好多，隨便一搜就是一片，當前，只是老的版本。目前螞蟻內部的zdal好像已經更新到zdal5了吧，那咱可就看不到了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/1c/1c6df9159f8c2cfc7c0ef8a14b2025e4.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"越複雜的系統，數據庫中間件的作用越大。就拿zdal來說，它提供分庫分表，結果集合並，sql解析，數據庫failover動態切換等數據訪問層統一解決方案。下面就一起來看下，其內部實現是怎麼樣的。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"Part2 架構剖析之高屋建瓴","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9a/9a6ae9793a273eb2284714608c65149c.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.1整體概述","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上圖所示，zdal有四個重要的組成部分：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"價值體現--客戶端Client包","attrs":{}},{"type":"text","text":"。對外暴露基本操作接口，用於業務層簡單黑盒的操作數據源；業務只和client交互，動態切換/路由等邏輯只需要進行規則配置，相關邏輯由zdal實現。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"核心功能--連接管理datasource包","attrs":{}},{"type":"text","text":"。最核心的能力，提供多種類型數據庫的連接管理；不管功能多花哨，最終目的還是爲了解決數據庫連接的問題。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"關鍵能力--SQL解析parser包","attrs":{}},{"type":"text","text":"。基礎SQL解析能力；解析sql類型、字段名稱、數據庫等等，配合規則進行路由","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"擴展能力--庫表路由rule包","attrs":{}},{"type":"text","text":"。根據parser解析出的字段確定邏輯庫表和物理庫表。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.2組件圖看架構","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"組件圖對整體架構和各組件及相互聯繫的理解可以起到很好的幫助。一個簡版的組件圖畫了好久，還有不少錯，不過大概是這麼個意思，哎，基本功要丟~","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ec/ec8d214f4751d0cecf8da1d685ce6f1f.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對照上圖可以比較清晰的看到：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Client包對應用層暴露的數據源、負責監聽配置動態變更的監聽組件、負責加載組織各部分的配置組件、負責加載spring bean 和庫表規則的配置組件；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Client中加載了規則組件，實現邏輯表和數據庫的路由規則。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Client中的庫表配置調用datasource中的數據源管理服務並構建連接池的連接池；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Client中的SqlDispatcher服務調用SQL解析組件實現SQL解析。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"Part3細節剖析之一葉知秋","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.1配置加載和bean初始化","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大部分情況下，我們使用如mybatis這樣的ORM框架來進行數據庫操作，其實不管是ORM還是其他方式，應用層都需要對數據源進行配置。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以，client對外暴露了一個符合JDBC標準的datasource數據源，用來滿足應用層ORM等框架配置數據源的要求--","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"ZdalDataSource","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/23/2305a5987f9c359aa83941eaaea6ee84.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如圖片被壓縮看不清，後臺回覆獲取","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"//只提供了一個init方法，這也是spring啓動時時，必須要調用的初始化方法，所有功能，都從這裏開始\npublic class ZdalDataSource extends AbstractZdalDataSource implements DataSource{\n    public void init() {\n        try {\n            super.initZdalDataSource();\n        } catch (Exception e) {\n            CONFIG_LOGGER.error(\"...\");\n            throw new ZdalClientException(e);\n        }\n    }\n\n複製代碼","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"ZdalDataSource#init()","attrs":{}},{"type":"text","text":" 方法即爲配置加載的核心入口，init中負責加載spring配置，根據配置初始化數據源，並創建連接池，同時，將邏輯表和物理庫的對應關係都維護起來供後續路由調用。","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"    /*父類的init方法*/\nprotected void initZdalDataSource() {\n    /*用FileSystemXmlApplicationContext方式加載配置文件中的數據源和規則，轉化成zdalConfig對象*/\n    this.zdalConfig = ZdalConfigurationLoader.getInstance().getZdalConfiguration(appName,dbmode, appDsName, configPath);\n    this.dbConfigType = zdalConfig.getDataSourceConfigType();\n   this.dbType = zdalConfig.getDbType();\n   //初始化數據源\n   this.initDataSources(zdalConfig);\n   this.inited.set(true);\n    }\n}\n\n複製代碼","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從上面的類圖和這裏的兩個入口方法大概瞭解到zdal配置加載的啓動流程。下面我們就來詳細看一下，讀寫分離和分庫分表的規則是怎麼被加載，怎麼起作用的。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.2細說讀寫分離","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"讀寫分離配置的加載","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先，我們需要有數據源的相關配置，如下圖：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d2/d2c37efa05bd361a69679ee05eac8251.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此XML配置會在init方法被調用時，被初始化，解析成ZdalConfig類的屬性，ZdalConfig類的主要成員見下面代碼：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"public class ZdalConfig {\n    /** key=dsName;value=DataSourceParameter 所有物理數據源的配置項,比如用戶名，密碼，庫名等 */\n    private Map dataSourceParameters = new ConcurrentHashMap();\n    /** 邏輯數據源和物理數據源的對應關係:key=logicDsName,value=physicDsName */\n    private Map              logicPhysicsDsNames  = new ConcurrentHashMap();\n    /** 數據源的讀寫規則，比如只讀，或讀寫等配置*/\n    private Map              groupRules           = new ConcurrentHashMap();\n    /** 異常轉移的數據源規則*/\n    private Map              failoverRules        = new ConcurrentHashMap();\n    //一份完整的讀寫分離和分庫分表規則配置\n    private AppRule                          appRootRule;\n\n複製代碼","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到，xml中的規則，被解析到xxxRules裏。這裏以groupRules爲例，failover同理。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下一步則是通過解析得到的zdalConfig 來初始化數據源：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"protected final void initDataSources(ZdalConfig zdalConfig) {\n    //DataSourceParameter中存的是數據源參數，如用戶名密碼，最大最小連接數等\n    for (Entry entry : zdalConfig.getDataSourceParameters().entrySet()) {\n        try {\n           //初始化連接池\n           ZDataSource zDataSource = new ZDataSource(/*設置最大最小連接數*/createDataSourceDO(entry.getValue(),zdalConfig.getDbType(), appDsName + \".\" + entry.getKey()));\n           this.dataSourcesMap.put(entry.getKey(), zDataSource);\n        } catch (Exception e) {\n            //...\n        }\n   }\n  //其他分支略，只看最簡單的分組模式\n  if (dbConfigType.isGroup()) {\n       //讀寫配置賦值\n       this.rwDataSourcePoolConfig = zdalConfig.getGroupRules();\n       //初始化多份讀庫下的負載均衡\n       this.initForLoadBalance(zdalConfig.getDbType());\n  }\n  //註冊監聽：爲了滿足動態切換\n  this.initConfigListener();\n}\n\n複製代碼","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"initForLoadBalance的方法如下：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"private void initForLoadBalance(DBType dbType) {\n    Map dsSelectors = this.buildRwDbSelectors(this.rwDataSourcePoolConfig);\n    this.runtimeConfigHolder.set(new ZdalRuntime(dsSelectors));\n    this.setDbTypeForDBSelector(dbType);\n}\n\n複製代碼","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到，首先構建出了DB選擇器，然後賦值給了runtimeConfigHolder供運行時獲取。而構建DB選擇器的時候，其實是按讀寫兩個維度，把所有數據源都構建了一遍，即group_r和group_w下都包含5個數據源，只不過各自的權重不一樣：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"//比如按上面的配置寫庫只有一個，但是也會包含全數據源\n\ngroup_0_w_0 :\ngroup_0_w_1 :\ngroup_0_w_2 :\ngroup_0_w_3 :\ngroup_0_w_4 :\n\n//上述就是寫相關的DBSelecter的內容。\n\n複製代碼","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"讀寫分離怎麼起作用","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以delete爲例，更新刪除是要操作寫庫的","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":" public void delete(ZdalDataSource dataSource) {\n     String deleteSql = \"delete from test\";\n     Connection conn = null;\n     PreparedStatement pst = null;\n     try {\n        conn = dataSource.getConnection();\n        pst = conn.prepareStatement(deleteSql);\n        pst.execute();\n     } catch (Exception e) {\n            //...\n     } finally {\n           //資源關閉\n     }\n }\n\n複製代碼","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"getConnection會從上文中提到的runtimeConfigHolder中獲取DBSelecter，然後執行execute方法","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":" public boolean execute() throws SQLException {\n    SqlType sqlType = getSqlType(sql);\n    // SELECT相關的就選擇group_r對應的DBSelecter\n   if (sqlType == SqlType.SELECT || sqlType == SqlType.SELECT_FOR_UPDATE|| sqlType == SqlType.SELECT_FROM_DUAL) {\n     //略\n    return true;\n    //update/delete相關的就選擇group_w對應的DBSelecter\n  } else if (sqlType == SqlType.INSERT || sqlType == SqlType.UPDATE|| sqlType == SqlType.DELETE) {\n       if (super.dbConfigType == DataSourceConfigType.GROUP) {\n           executeUpdate0();\n       } else {\n           executeUpdate();\n      }\n      return false;\n  } \n}\n\n複製代碼","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果是讀取相關的，那就選_r的DBSelecter，如果是寫相關的，那就選_W的DBSelecter。那麼executeUpdate0中是怎麼執行區分讀寫數據源的呢，其實就是把這一組的數據源根據權重篩選一遍。","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"// WeightRandom#select(int[], java.lang.String[])\nprivate String select(int[] areaEnds, String[] keys) {\n   //這裏的areaEnds數組，是一個累加範圍值數據\n   //比如三個庫權重    10   9   8\n   //那麼areaEnds就是  10  19  27 是對每個權重的累加，最後一個值是總和\n   int sum = areaEnds[areaEnds.length - 1];\n   //這樣隨機出來的數，是符合權重分佈的\n   int rand = random.nextInt(sum);\n   for (int i = 0; i