一、数据分片概述
1.1、相关概念
1.1.1、分库/分表
将存放在一台数据库服务器中的数据，按照特定的方式进行拆分，分散存放到多台数据库服务器中，以达到分散单台服务器负载的效果。
1.1.2、水平分割
- 横向切分
按照表中指定字段的分片规则，将表记录按行切分，分散存储到多个数据库中。
1.1.3、垂直分割
- 纵向切分
将单个数据库的多个表按业务类型分类，分散存储到不同的数据库。
1.2、MyCAT介绍
1.2.1、软件介绍
MyCAT是基于java的分布式数据库系统中间件，为高并发环境的分布式存储提供解决方案
- 适合数据大量写入的存储需求，不适合大量查询的环境
- 支持MySQL、Oracle、Sqlserver、Mongodb等
- 提供数据读写分离服务
- 提供数据分片服务
- 基于阿里巴巴Cobar进行研发的开源软件
1.2.2、分片规则
MyCAT支持提供10种分片规则
1. 枚举法
sharding-by-intfile
2. 固定分片
rule1
3. 范围约定
auto-sharding-long
4. 求模法
mod-long
5. 日期列分区法
sharding-by-date
6. 通配取模
sharding-by-pattern
7. ASCII码求模通配
sharding-by-prefixpattern
8. 编程指定
sharding-by-substring
9. 字符串拆分hash解析
sharding-by-stringhash
10. 一致性hash
sharding-by-murmur
1.2.3、工作过程
当MyCAT收到一个SQL命令时
- 解析SQL命令涉及到的表
- 然后看对表的配置，如果有分片规则，则获取SQL命令里分片字段的值，并匹配分片函数，获取分片列表
- 然后将SQL命令发往对应的数据库服务器去执行
- 最后收集和处理所有分片结构数据，并返回到客户端
二、部署MyCAT服务
2.1、拓扑结构
2.2、部署MyCAT服务
2.2.1、创建数据库
- 分别在数据库1、2、3上创建数据库db1、db2、db3
数据库1
mysql> create database db1;
数据库2
mysql> create database db2;
数据库3
mysql> create database db3;
2.2.2、在分片服务器上安装JDK
[root@mycat ~]# yum -y install java-1.8.0-openjdk.x86_64
[root@mycat ~]# java -version
openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)
2.2.3、安装mycat软件
Mycat-server-1.6-RELEASE-20161028204710-linux.tar.gz
//免安装源码包
[root@mycat ~]# tar -zxf Mycat-server-1.6-RELEASE-20161028204710-linux.tar.gz
[root@mycat ~]# ls mycat/
bin catlet conf lib logs version.txt
[root@mycat ~]# mv mycat/ /usr/local/
[root@mycat ~]# ls /usr/local/mycat/
bin //mycat命令
catlet //扩展功能
conf //配置文件
lib //mycat使用的jar包
logs //mycat启动日志和运行日志
version.txt
- mycat使用命令
[root@mycat ~]# /usr/local/mycat/bin/mycat help
Usage: /usr/local/mycat/bin/mycat { console | start | stop | restart | status | dump }
2.2.4、重要的配置文件
[root@mycat ~]# cd /usr/local/mycat/conf
[root@mycat conf]# ls *.xml
ehcache.xml
log4j2.xml
rule.xml //分片规则
schema.xml //配置数据分片存储的表
server.xml //设置连接账号及逻辑库
2.3、修改配置文件
2.3.1、创建连接用户
- 使用默认配置
暂时不用配置
[root@mycat conf]# pwd
/usr/local/mycat/conf
[root@mycat conf]# vim server.xml
<user name="root"> //连接mycat服务的用户名
<property name="password">123456</property> //用户密码
<property name="schemas">TESTDB</property> //逻辑库
</user>
<user name="user"> ////连接mycat服务的用户名
<property name="password">user</property>
<property name="schemas">TESTDB</property>
<property name="readOnly">true</property> //只读权限
</user>
2.3.2、配置数据分片
- 先备份schema.xml
[root@mycat conf]# cp schema.xml /root/
- 删除无用的注释行
[root@mycat conf]# vim schema.xml
:q!
[root@mycat conf]# sed -i '56,77d' schema.xml //删除56到77行
[root@mycat conf]# sed -i '39,42d' schema.xml //删除39到42行
- 原始配置
![配置文件全貌]()
- 修改后的
- 修改后的<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">
<schema name="TESTDB" checkSQLschema="false" sqlMaxLimit="100">
<!-- auto sharding by id (long) -->
<table name="travelrecord" dataNode="dn1,dn2,dn3" rule="auto-sharding-long" />
<!-- global table is auto cloned to all defined data nodes ,so can join
with any table whose sharding node is in the same data node -->
<table name="company" primaryKey="ID" type="global" dataNode="dn1,dn2,dn3" />
<table name="goods" primaryKey="ID" type="global" dataNode="dn1,dn2" />
<!-- random sharding using mod sharind rule -->
<table name="hotnews" primaryKey="ID" autoIncrement="true" dataNode="dn1,dn2,dn3"
rule="mod-long" />
<!-- <table name="dual" primaryKey="ID" dataNode="dnx,dnoracle2" type="global"
needAddLimit="false"/> <table name="worker" primaryKey="ID" dataNode="jdbc_dn1,jdbc_dn2,jdbc_dn3"
rule="mod-long" /> -->
<table name="employee" primaryKey="ID" dataNode="dn1,dn2,dn3"
rule="sharding-by-intfile" />
<table name="customer" primaryKey="ID" dataNode="dn1,dn2,dn3"
rule="sharding-by-intfile">
<childTable name="orders" primaryKey="ID" joinKey="customer_id"
parentKey="id">
<childTable name="order_items" joinKey="order_id"
parentKey="id" />
</childTable>
<childTable name="customer_addr" primaryKey="ID" joinKey="customer_id"
parentKey="id" />
</table>
<!-- <table name="oc_call" primaryKey="ID" dataNode="dn1$0-743" rule="latest-month-calldate"
/> -->
</schema>
<!-- <dataNode name="dn1$0-743" dataHost="localhost1" database="db$0-743"
/> -->
<dataNode name="dn1" dataHost="mysql53" database="db1" />
<dataNode name="dn2" dataHost="mysql54" database="db2" />
<dataNode name="dn3" dataHost="mysql55" database="db3" />
<dataHost name="mysql53" maxCon="1000" minCon="10" balance="0"
writeType="0" dbType="mysql" dbDriver="native" switchType="1" slaveThreshold="100">
<heartbeat>select user()</heartbeat>
<writeHost host="hostM1" url="192.168.4.53:3306" user="adminabc"
password="123456">
</writeHost>
</dataHost>
<dataHost name="mysql54" maxCon="1000" minCon="10" balance="0"
writeType="0" dbType="mysql" dbDriver="native" switchType="1" slaveThreshold="100">
<heartbeat>select user()</heartbeat>
<writeHost host="hostM1" url="192.168.4.54:3306" user="adminabc"
password="123456">
</writeHost>
</dataHost>
<dataHost name="mysql55" maxCon="1000" minCon="10" balance="0"
writeType="0" dbType="mysql" dbDriver="native" switchType="1" slaveThreshold="100">
<heartbeat>select user()</heartbeat>
<writeHost host="hostM1" url="192.168.4.55:3306" user="adminabc"
password="123456">
</writeHost>
</dataHost>
</mycat:schema>
- 用户授权
[root@db1 ~]# mysql -uroot -p123456 -e 'grant all on *.* to adminabc@"%" identified by "123456"'
[root@db2 ~]# mysql -uroot -p123456 -e 'grant all on *.* to adminabc@"%" identified by "123456"'
[root@db3 ~]# mysql -uroot -p123456 -e 'grant all on *.* to adminabc@"%" identified by "123456"'
- 客户端验证登陆
mysql -h192.168.4.53 -uadminabc -p123456
mysql -h192.168.4.54 -uadminabc -p123456
mysql -h192.168.4.55 -uadminabc -p123456
- 启动服务
查看状态
[root@mycat conf]# /usr/local/mycat/bin/mycat start
Starting Mycat-server...
[root@mycat conf]# netstat -utnlp | grep 8066
[root@mycat mycat]# pwd
/usr/local/mycat
[root@mycat mycat]# ls logs/
mycat.log mycat.pid wrapper.log
2.3.3、查看配置完成后的结果
[root@client ~]# mysql -h192.168.4.56 -P8066 -uroot -p123456
mysql> show databases;
+----------+
| DATABASE |
+----------+
| TESTDB | //定义的逻辑库,实际上不存在。里面的表也是假的
+----------+
1 row in set (0.00 sec)
mysql> use TESTDB
mysql> show tables;
+------------------+
| Tables in TESTDB |
+------------------+
| company |
| customer |
| customer_addr |
| employee |
| goods |
| hotnews |
| orders |
| order_items |
| travelrecord |
+------------------+
9 rows in set (0.00 sec)
三、测试配置
3.1、分片规则
3.1.1、枚举分片规则 sharding-by-intfile
给分片字段赋值时，值必须在分片规则配置文件定义的值里选择
- 查看那张表使用sharding-by-intfile
vim schema.xml
<table name="employee" primaryKey="ID" dataNode="dn1,dn2,dn3"
rule="sharding-by-intfile" /> //表使用的分片规则
- 查看分片字段名
字段名必须在规则文件定义的值里选择
vim /usr/local/mycat/conf/rule.xml
<tableRule name="sharding-by-intfile"> //分片规则
<rule>
<columns>sharding_id</columns> //字段名
<algorithm>hash-int</algorithm> //算法
</rule>
</tableRule>
- 修改分片规则配置文件，定义值列表
vim /usr/local/mycat/conf/rule.xml
<function name="hash-int"
class="io.mycat.route.function.PartitionByFileMap"> //算法
<property name="mapFile">partition-hash-int.txt</property> //配置文件名
</function>
vim /usr/local/mycat/conf/partition-hash-int.txt
10000=0 //0表示第一台数据库服务器，对应dn1
10010=1 //1表示第二台数据库服务器，对应dn2
10020=2 //3表示第三台数据库服务器，对应dn3
- 重启mycat服务
[root@mycat local]# /usr/local/mycat/bin/mycat stop
Stopping Mycat-server...
Stopped Mycat-server.
[root@mycat local]# /usr/local/mycat/bin/mycat start
Starting Mycat-server...
[root@mycat local]# netstat -utnlp | grep 8066
tcp6 0 0 :::8066 :::* LISTEN 4600/java
- 客户端连接mycat服务器，创建employee并存储数据
mysql> create table employee( ID int primary key auto_increment ,
-> sharding_id int , name char(15) , age int );
mysql> desc employee;
+-------------+----------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+----------+------+-----+---------+----------------+
| ID | int(11) | NO | PRI | NULL | auto_increment |
| sharding_id | int(11) | YES | | NULL | |
| name | char(15) | YES | | NULL | |
| age | int(11) | YES | | NULL | |
+-------------+----------+------+-----+---------+----------------+
4 rows in set (0.04 sec)
mysql> insert into employee(sharding_id , name , age )values(10000,"bob",19),(10000,"bob",19),(10000,"bob",19);
Query OK, 3 rows affected (0.06 sec)
Records: 3 Duplicates: 0 Warnings: 0
mysql> insert into employee(sharding_id , name , age )values(10010,"bobA",19),(10010,"bobA",19),(10010,"bobA",19);
Query OK, 3 rows affected (0.11 sec)
Records: 3 Duplicates: 0 Warnings: 0
mysql> insert into employee(sharding_id , name , age )values(10020,"bobF",19),(10020,"bobF",19),(10020,"bobF",19);
Query OK, 3 rows affected (0.03 sec)
Records: 3 Duplicates: 0 Warnings: 0
- 在三台数据库服务器上查看employee表记录
[root@db1 ~]# mysql -uroot -p123456 -e 'select * from db1.employee'
mysql: [Warning] Using a password on the command line interface can be insecure.
+----+-------------+------+------+
| ID | sharding_id | name | age |
+----+-------------+------+------+
| 1 | 10000 | bob | 19 |
| 2 | 10000 | bob | 19 |
| 3 | 10000 | bob | 19 |
+----+-------------+------+------+
[root@db2 ~]# mysql -uroot -p123456 -e 'select * from db2.employee'
mysql: [Warning] Using a password on the command line interface can be insecure.
+----+-------------+------+------+
| ID | sharding_id | name | age |
+----+-------------+------+------+
| 1 | 10010 | bobA | 19 |
| 2 | 10010 | bobA | 19 |
| 3 | 10010 | bobA | 19 |
+----+-------------+------+------+
[root@db3 ~]# mysql -uroot -p123456 -e 'select * from db3.employee'
mysql: [Warning] Using a password on the command line interface can be insecure.
+----+-------------+------+------+
| ID | sharding_id | name | age |
+----+-------------+------+------+
| 1 | 10020 | bobF | 19 |
| 2 | 10020 | bobF | 19 |
| 3 | 10020 | bobF | 19 |
+----+-------------+------+------+
3.1.2、求模分片规则mod-long
- 根据字段值与设定的数字求模结果存储数据
使用分片字段的值和指定数字的值，做取余运算，根据运算结果存取数据。 - 例如
余数为0对应dn1
余数为1对应dn2
余数为2对应dn3 - 查看使用求模分别规则的表
[root@mycat local]# vim /usr/local/mycat/conf/schema.xml
<table name="hotnews" dataNode="dn1,dn2,dn3"
rule="mod-long" />
- 查看分片字段名
vim /usr/local/mycat/conf/rule.xml
<tableRule name="mod-long">
<rule>
<columns>id</columns>
<algorithm>mod-long</algorithm>
</rule>
</tableRule>
- 设置分片字段求模的数字
vim /usr/local/mycat/conf/rule.xml
<function name="mod-long" class="io.mycat.route.function.PartitionByMod">
<!-- how many data nodes -->
<property name="count">3</property>
</function>
- 重启mycat服务
[root@mycat ~]# /usr/local/mycat/bin/mycat stop
Stopping Mycat-server...
Stopped Mycat-server.
[root@mycat ~]# /usr/local/mycat/bin/mycat start
Starting Mycat-server...
[root@mycat ~]# netstat -utnlp | grep 8066
tcp6 0 0 :::8066 :::* LISTEN 4896/java
- 客户端连接mycat服务器建表 并存储数据
[root@client ~]# mysql -h192.168.4.56 -P8066 -uroot -p123456
分布式存储插入记录不能省略字段名
mysql> use TESTDB
mysql> create table hotnews( id int , title char(50) , comment char(200) ) ;
mysql> insert into hotnews(id , title , comment) values (9,"xxx","xxxx"),(9,"xxx","xxx"),(9,"xxx","xxxx");
Query OK, 3 rows affected (0.09 sec)
Records: 3 Duplicates: 0 Warnings: 0
mysql> insert into hotnews(id , title , comment) values (10,"xxxA","xxxxA"),(10,"xxxA","xxxA"),(10,"xxxA","xxxxA");
Query OK, 3 rows affected (0.12 sec)
Records: 3 Duplicates: 0 Warnings: 0
mysql> insert into hotnews(id , title , comment) values (11, "xxxB","xxxxA"),(11,"xxxB","xxxA"),(11,"xxxB","xxxxA");
Query OK, 3 rows affected (0.03 sec)
Records: 3 Duplicates: 0 Warnings: 0
- 三台数据库查看数据
[root@db1 ~]# mysql -uroot -p123456 -e 'select * from db1.hotnews'
mysql: [Warning] Using a password on the command line interface can be insecure.
+------+-------+---------+
| id | title | comment |
+------+-------+---------+
| 9 | xxx | xxxx |
| 9 | xxx | xxx |
| 9 | xxx | xxxx |
+------+-------+---------+
[root@db2 ~]# mysql -uroot -p123456 -e 'select * from db2.hotnews'
mysql: [Warning] Using a password on the command line interface can be insecure.
+------+-------+---------+
| id | title | comment |
+------+-------+---------+
| 10 | xxxA | xxxxA |
| 10 | xxxA | xxxA |
| 10 | xxxA | xxxxA |
+------+-------+---------+
[root@db3 ~]# mysql -uroot -p123456 -e 'select * from db3.hotnews'
mysql: [Warning] Using a password on the command line interface can be insecure.
+------+-------+---------+
| id | title | comment |
+------+-------+---------+
| 11 | xxxB | xxxxA |
| 11 | xxxB | xxxA |
| 11 | xxxB | xxxxA |
+------+-------+---------+
四、添加新库、新表
4.1、添加新库
- 修改server.xml文件
添加GAMEDB库
[root@mycat ~]# vim /usr/local/mycat/conf/server.xml
<user name="root">
<property name="password">123456</property>
<property name="schemas">TESTDB,GAMEDB</property>
</user>
<user name="user">
<property name="password">user</property>
<property name="schemas">TESTDB,GAMEDB</property>
<property name="readOnly">true</property>
</user>
4.2、添加新表
- 修改schema.xml
[root@mycat ~]# vim /usr/local/mycat/conf/schema.xml
<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">
<schema name="GAMEDB" checkSQLschema="false" sqlMaxLimit="100">
<table name="company2" primaryKey="ID" type="global" dataNode="dn1,dn2,dn3" />
<table name="hotnews2" rule="sharding-by-intfile" dataNode="dn1,dn2,dn3" />
</schema>
4.3、重启服务
[root@mycat ~]# /usr/local/mycat/bin/mycat stop
Stopping Mycat-server...
Stopped Mycat-server.
[root@mycat ~]# /usr/local/mycat/bin/mycat start
Starting Mycat-server...
[root@mycat ~]# netstat -utnlp | grep 8066
tcp6 0 0 :::8066 :::* LISTEN 2126/java
4.4、客户端登录查看
[root@client ~]# mysql -h192.168.4.56 -P8066 -uroot -p123456
mysql> show databases;
+----------+
| DATABASE |
+----------+
| GAMEDB |
| TESTDB |
+----------+
2 rows in set (0.00 sec)
mysql> use GAMEDB
mysql> show tables;
+------------------+
| Tables in GAMEDB |
+------------------+
| company2 |
| hotnews2 |
+------------------+
2 rows in set (0.00 sec)
mysql> create table company2(name char(20),addr char(50));
Query OK, 0 rows affected (0.01 sec)