一、背景
早期，阿里巴巴B2B公司因爲存在杭州和美國雙機房部署，存在跨機房同步的業務需求。不過早期的數據庫同步業務，主要是基於trigger的方式獲取增量變更，不過從2010年開始，阿里系公司開始逐步的嘗試基於數據庫的日誌解析，獲取增量變更進行同步，由此衍生出了增量訂閱&消費的業務，從此開啓了一段新紀元。
ps. 目前內部版本已經支持mysql和oracle部分版本的日誌解析，當前的canal開源版本支持5.7及以下的版本(阿里內部mysql 5.7.13, 5.6.10, mysql 5.5.18和5.1.40/48)
基於日誌增量訂閱&消費支持的業務：
數據庫鏡像
數據庫實時備份
多級索引 (賣家和買家各自分庫索引)
search build
業務cache刷新
價格變化等重要業務消息
二、項目介紹
名稱：canal [kə'næl]
譯意： 水道/管道/溝渠
語言： 純java開發
定位： 基於數據庫增量日誌解析，提供增量數據訂閱&消費，目前主要支持了mysql
三、工作原理
1、mysql主備複製實現
從上層來看，複製分成三步：
master將改變記錄到二進制日誌(binary log)中（這些記錄叫做二進制日誌事件，binary log events，可以通過show binlog events進行查看）；
slave將master的binary log events拷貝到它的中繼日誌(relay log)；
slave重做中繼日誌中的事件，將改變反映它自己的數據。
2、canal的工作原理：
原理相對比較簡單：
canal模擬mysql slave的交互協議，僞裝自己爲mysql slave，向mysql master發送dump協議
mysql master收到dump請求，開始推送binary log給slave(也就是canal)
canal解析binary log對象(原始爲byte流)
四、canal安裝
1、下載並解壓縮
[root@master1 canal]#wget https://github.com/alibaba/canal/releases/download/v1.0.23/canal.deployer-1.0.23.tar.gz
[root@master1 canal]#mkdir /root/canal
[root@master1 canal]#tar zxvf canal.deployer-1.0.23.tar.gz -C /root/canal
2、創建mysql的canal用戶
mysql> CREATE USER 'canal'@'localhost' IDENTIFIED BY 'canal';
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'canal'@'localhost' WITH GRANT OPTION;
Query OK, 0 rows affected (0.01 sec)
mysql> CREATE USER 'canal'@'%' IDENTIFIED BY 'canal';
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
3、修改canal配置文件(如果是訪問本機，並且用戶密碼都爲canal則不需要修改配置文件)
[root@master1 canal]# vi /root/canal/conf/example/instance.properties
#################################################
## mysql serverId
canal.instance.mysql.slaveId = 1234
# position info
canal.instance.master.address = 127.0.0.1:3306
canal.instance.master.journal.name =
canal.instance.master.position =
canal.instance.master.timestamp =
#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
# username/password
#修改這兩項
canal.instance.dbUsername = canal
canal.instance.dbPassword = canal
canal.instance.defaultDatabaseName =
canal.instance.connectionCharset = UTF-8
# table regex
canal.instance.filter.regex = .*\\..*
# table black regex
canal.instance.filter.black.regex =
#################################################
4、啓動canal
[root@master1 canal]# sh /root/canal/bin/startup.sh
cd to /root/canal/bin for workaround relative path
LOG CONFIGURATION : /root/canal/bin/../conf/logback.xml
canal conf : /root/canal/bin/../conf/canal.properties
CLASSPATH :/root/canal/bin/../conf:/root/canal/bin/../lib/zookeeper-3.4.5.jar:/root/canal/bin/../lib/zkclient-0.1.jar:/root/canal/bin/../lib/spring-2.5.6.jar:/root/canal/bin/../lib/slf4j-api-1.7.12.jar:/root/canal/bin/../lib/protobuf-java-2.6.1.jar:/root/canal/bin/../lib/oro-2.0.8.jar:/root/canal/bin/../lib/netty-3.2.5.Final.jar:/root/canal/bin/../lib/logback-core-1.1.3.jar:/root/canal/bin/../lib/logback-classic-1.1.3.jar:/root/canal/bin/../lib/log4j-1.2.14.jar:/root/canal/bin/../lib/jcl-over-slf4j-1.7.12.jar:/root/canal/bin/../lib/guava-18.0.jar:/root/canal/bin/../lib/fastjson-1.1.35.jar:/root/canal/bin/../lib/commons-logging-1.1.1.jar:/root/canal/bin/../lib/commons-lang-2.6.jar:/root/canal/bin/../lib/commons-io-2.4.jar:/root/canal/bin/../lib/commons-beanutils-1.8.2.jar:/root/canal/bin/../lib/canal.store-1.0.23.jar:/root/canal/bin/../lib/canal.sink-1.0.23.jar:/root/canal/bin/../lib/canal.server-1.0.23.jar:/root/canal/bin/../lib/canal.protocol-1.0.23.jar:/root/canal/bin/../lib/canal.parse.driver-1.0.23.jar:/root/canal/bin/../lib/canal.parse.dbsync-1.0.23.jar:/root/canal/bin/../lib/canal.parse-1.0.23.jar:/root/canal/bin/../lib/canal.meta-1.0.23.jar:/root/canal/bin/../lib/canal.instance.spring-1.0.23.jar:/root/canal/bin/../lib/canal.instance.manager-1.0.23.jar:/root/canal/bin/../lib/canal.instance.core-1.0.23.jar:/root/canal/bin/../lib/canal.filter-1.0.23.jar:/root/canal/bin/../lib/canal.deployer-1.0.23.jar:/root/canal/bin/../lib/canal.common-1.0.23.jar:/root/canal/bin/../lib/aviator-2.2.1.jar:.:/usr/java/jdk1.8.0_121/lib
cd to /root/canal for continue
5、關閉canal
[root@master1 canal]# sh /root/canal/bin/stop.sh
master1: stopping canal 16062 ...
Oook! cost:1
6、相關日誌位置
[root@master1 canal]# cat /root/canal/logs/canal/canal.log
[root@master1 canal]# cat /root/canal/logs/example/example.log
五、編寫代碼讀取binlog
1、創建maven項目
Paste_Image.png
2、pom.xml增加依賴
<dependency>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal.client</artifactId>
<version>1.0.12</version>
</dependency>
3、編寫ClientSample類訂閱binlog
>package com.chainfin.canal;
>
>import java.net.InetSocketAddress;
>import java.util.List;
>
>import com.alibaba.otter.canal.client.CanalConnector;
import com.alibaba.otter.canal.client.CanalConnectors;
import com.alibaba.otter.canal.protocol.CanalEntry.Column;
import com.alibaba.otter.canal.protocol.CanalEntry.Entry;
import com.alibaba.otter.canal.protocol.CanalEntry.EntryType;
import com.alibaba.otter.canal.protocol.CanalEntry.EventType;
import com.alibaba.otter.canal.protocol.CanalEntry.RowChange;
import com.alibaba.otter.canal.protocol.CanalEntry.RowData;
import com.alibaba.otter.canal.protocol.Message;
>
/**
*
* @author jinxiaoxin
*
*/
public class ClientSample {
public static void main(String[] args) {
// 創建鏈接
CanalConnector connector = CanalConnectors.newSingleConnector(
new InetSocketAddress("10.105.10.121", 11111), "example", "",
"");// AddressUtils.getHostIp(),
int batchSize = 1000;
int emptyCount = 0;
try {
connector.connect();
connector.subscribe("test\\..*");// .*代表database，..*代表table
connector.rollback();//
int totalEmptyCount = 120;
while (emptyCount < totalEmptyCount) {
Message message = connector.getWithoutAck(batchSize); // 獲取指定數量的數據
long batchId = message.getId();
int size = message.getEntries().size();
if (batchId == -1 || size == 0) {
emptyCount++;
// System.out.println("empty count : " + emptyCount);
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
}
} else {
emptyCount = 0;
// System.out.printf("message[batchId=%s,size=%s] \n",
// batchId, size);
printEntry(message.getEntries());
}
connector.ack(batchId); // 提交確認
// connector.rollback(batchId); // 處理失敗, 回滾數據
}
System.out.println("empty too many times, exit");
} finally {
connector.disconnect();
}
}
>
private static void printEntry(List<Entry> entrys) {
for (Entry entry : entrys) {
if (entry.getEntryType() == EntryType.TRANSACTIONBEGIN
|| entry.getEntryType() == EntryType.TRANSACTIONEND) {
continue;
}
RowChange rowChage = null;
try {
rowChage = RowChange.parseFrom(entry.getStoreValue());
} catch (Exception e) {
throw new RuntimeException(
"ERROR ## parser of eromanga-event has an error,data:"
+ entry.toString(), e);
}
EventType eventType = rowChage.getEventType();
System.out
.println(String
.format("================> binlog[%s:%s] ,name[%s,%s] , eventType : %s",
entry.getHeader().getLogfileName(), entry
.getHeader().getLogfileOffset(),
entry.getHeader().getSchemaName(), entry
.getHeader().getTableName(),
eventType));
for (RowData rowData : rowChage.getRowDatasList()) {
if (eventType == EventType.DELETE) {
printColumn(rowData.getBeforeColumnsList());
} else if (eventType == EventType.INSERT) {
printColumn(rowData.getAfterColumnsList());
} else {
System.out.println("-------> before");
printColumn(rowData.getBeforeColumnsList());
System.out.println("-------> after");
printColumn(rowData.getAfterColumnsList());
}
}
}
}
>
private static void printColumn(List<Column> columns) {
for (Column column : columns) {
System.out.println(column.getName() + " : " + column.getValue()
+ " update=" + column.getUpdated());
}
}
}
>
4、MYSQL進行增刪改查
mysql> CREATE TABLE `xdual` (
-> `ID` int(11) NOT NULL AUTO_INCREMENT,
-> `X` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
-> PRIMARY KEY (`ID`)
-> ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8 ;
Query OK, 0 rows affected (0.01 sec)
mysql> insert into xdual(id,x) values(1,now());
Query OK, 1 row affected (0.01 sec)
5、eclipse控制檯可以看到如下輸出
Paste_Image.png
六、canal集羣搭建
1、需要安裝zookeeper
自行了解安裝及其使用，此處略過
2、修改canal配置文件
修改兩個配置
[root@master1 canal]# vi /root/canal/conf/canal.properties
# 用逗號隔開且不留空格
canal.zkServers=10.105.10.123:2181,10.105.10.124:2181,10.105.10.125:2181
canal.instance.global.spring.xml = classpath:spring/default-instance.xml
3、拷貝canal到另一臺機器並解壓縮(作爲從節點)
[root@master1 canal]# scp canal.deployer-1.0.23.tar.gz [email protected]:/root
canal.deployer-1.0.23.tar.gz 100% 9586KB 9.4MB/s 00:00
[root@master2 ~]# mkdir canal
[root@master2 ~]# tar -zxvf canal.deployer-1.0.23.tar.gz -C /root/canal
4、修改slave配置文件
[root@master2 canal]# vi /root/canal/conf/example/instance.properties
## mysql serverId修改與master不同即可
canal.instance.mysql.slaveId = 1235
# position info修改成mysql地址即可
canal.instance.master.address = 10.105.10.121:3306
[root@master2 canal]# vi /root/canal/conf/canal.properties
# 用逗號隔開且不留空格
canal.zkServers=10.105.10.123:2181,10.105.10.124:2181,10.105.10.125:2181
canal.instance.global.spring.xml = classpath:spring/default-instance.xml
5、兩臺機器分別啓動canal
[root@master2 canal]# sh /root/canal/bin/startup.sh
6、查看canal在zk中的狀態
[zk: localhost:2181(CONNECTED) 11] get /otter/canal/destinations/example/running
{"active":true,"address":"10.105.10.121:11111","cid":1}
cZxid = 0x250000f037
ctime = Thu Apr 13 13:42:04 CST 2017
mZxid = 0x250000f037
mtime = Thu Apr 13 13:42:04 CST 2017
pZxid = 0x250000f037
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x15b5b11e9cd16e1
dataLength = 55
numChildren = 0
10.105.10.121爲active，所以10.105.10.122位stangby節點
7、測試HA是否配置成功
stop掉10.105.10.121這臺機器
[root@master1 canal]# sh /root/canal/bin/stop.sh
master1: stopping canal 6107 ...
Oook! cost:1
[zk: localhost:2181(CONNECTED) 28] get /otter/canal/destinations/example/running
{"active":true,"address":"10.105.10.122:11111","cid":1}
cZxid = 0x250000f0a7
ctime = Thu Apr 13 13:46:10 CST 2017
mZxid = 0x250000f0a7
mtime = Thu Apr 13 13:46:10 CST 2017
pZxid = 0x250000f0a7
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x35b5b11e9d816cb
dataLength = 55
很明顯active已經切換到10.105.10.122上了
8、客戶端代碼
//創建connector 爲集羣connector
CanalConnector connector = CanalConnectors.newClusterConnector("10.105.10.123:2181,10.105.10.124:2181,10.105.10.125:2181","example", "", "");
啓動代碼
Paste_Image.png
9、錯誤解決
當這個配置的ip不配成127.0.0.1時(也就是本地)
canal.instance.master.address = 10.105.10.121:3306
查看canal配置文件，會報如下錯誤導致，無法讀取binlog
[root@master1 example]# tail -f example.log
2017-04-13 14:29:23.646 [destination = example , address = master1/10.105.10.121:3306 , EventParser] ERROR c.a.o.canal.parse.inbound.mysql.dbsync.DirectLogFetcher - I/O error while reading from client socket
java.io.IOException: Received error packet: errno = 1236, sqlstate = HY000 errmsg = Could not find first log file name in binary log index file
at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.DirectLogFetcher.fetch(DirectLogFetcher.java:95) ~[canal.parse-1.0.23.jar:na]
at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.dump(MysqlConnection.java:121) [canal.parse-1.0.23.jar:na]
at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:209) [canal.parse-1.0.23.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
解決辦法：
登錄mysql
mysql> show variables like "%sum%";
+---------------------------+-------+
| Variable_name | Value |
+---------------------------+-------+
| binlog_checksum | CRC32 |
| innodb_checksum_algorithm | crc32 |
| innodb_checksums | ON |
| innodb_log_checksums | ON |
| master_verify_checksum | OFF |
| slave_sql_verify_checksum | ON |
+---------------------------+-------+
6 rows in set (0.00 sec)
查詢資料發現mysql版本爲5.6時，
這個錯誤一般出現在master5.6,slave在低版本的情況下。這是由於5.6使用了crc32做binlog的checksum；
當一個event被寫入binary log(二進制日誌)的時候，checksum也同時寫入binary log，然後在event通過網絡傳輸到從服務器(slave)之後，再在從服務器中對其進行驗證並寫入從服務器的relay log.
由於每一步都記錄了event和checksum，所以我們可以很快地找出問題所在。
在master1中設置binlog_checksum =none；
mysql> set global binlog_checksum='NONE';
Query OK, 0 rows affected (0.00 sec)
mysql> show variables like "%sum%";
+---------------------------+-------+
| Variable_name | Value |
+---------------------------+-------+
| binlog_checksum | NONE |
| innodb_checksum_algorithm | crc32 |
| innodb_checksums | ON |
| innodb_log_checksums | ON |
| master_verify_checksum | OFF |
| slave_sql_verify_checksum | ON |
+---------------------------+-------+
6 rows in set (0.01 sec)
這樣錯誤就可以解決了
七、properties配置文件參數解釋
properties配置分爲兩部分：
canal.properties (系統根配置文件)
instance.properties (instance級別的配置文件，每個instance一份)
1、canal.properties
canal.destinations #當前server上部署的instance列表
canal.conf.dir #conf/目錄所在的路徑
canal.auto.scan #開啓instance自動掃描
#如果配置爲true，canal.conf.dir目錄下的instance配置變化會自動觸發：
#a. instance目錄新增： 觸發instance配置載入，lazy爲true時則自動啓動
#b. instance目錄刪除：卸載對應instance配置，如已啓動則進行關閉
#c. instance.properties文件變化：reload instance配置，如已啓動自動進行重啓操作
canal.auto.scan.interval #instance自動掃描的間隔時間，單位秒
canal.instance.global.mode #全局配置加載方式
canal.instance.global.lazy #全局lazy模式
canal.instance.global.manager.address #全局的manager配置方式的鏈接信息 無
canal.instance.global.spring.xml #全局的spring配置方式的組件文件
canal.instance.example.mode
canal.instance.example.lazy
canal.instance.example.spring.xml
#instance級別的配置定義，如有配置，會自動覆蓋全局配置定義模式
#命名規則：canal.instance.{name}.xxx 無
2、instance.properties
canal.id #每個canal server實例的唯一標識，暫無實際意義
canal.ip #canal server綁定的本地IP信息，如果不配置，默認選擇一個本機IP進行啓動服務
canal.port #canal server提供socket服務的端口
canal.zkServers #canal server鏈接zookeeper集羣的鏈接信息
#例子：10.20.144.22:2181,10.20.144.51:2181
canal.zookeeper.flush.period #canal持久化數據到zookeeper上的更新頻率，單位毫秒
canal.instance.memory.batch.mode #canal內存store中數據緩存模式
#1. ITEMSIZE : 根據buffer.size進行限制，只限制記錄的數量
#2. MEMSIZE : 根據buffer.size * buffer.memunit的大小，限制緩存記錄的大小
canal.instance.memory.buffer.size #canal內存store中可緩存buffer記錄數，需要爲2的指數
canal.instance.memory.buffer.memunit #內存記錄的單位大小，默認1KB，和buffer.size組合決定最終的內存使用大小
canal.instance.transactionn.size 最大事務完整解析的長度支持
超過該長度後，一個事務可能會被拆分成多次提交到canal store中，無法保證事務的完整可見性
canal.instance.fallbackIntervalInSeconds #canal發生mysql切換時，在新的mysql庫上查找binlog時需要往前查找的時間，單位秒
#說明：mysql主備庫可能存在解析延遲或者時鐘不統一，需要回退一段時間，保證數據不丟
canal.instance.detecting.enable #是否開啓心跳檢查
canal.instance.detecting.sql #心跳檢查sql insert into retl.xdual values(1,now()) on duplicate key update x=now()
canal.instance.detecting.interval.time #心跳檢查頻率，單位秒
canal.instance.detecting.retry.threshold #心跳檢查失敗重試次數
canal.instance.detecting.heartbeatHaEnable #心跳檢查失敗後，是否開啓自動mysql自動切換
#說明：比如心跳檢查失敗超過閥值後，如果該配置爲true，canal就會自動鏈到mysql備庫獲取binlog數據
canal.instance.network.receiveBufferSize #網絡鏈接參數，SocketOptions.SO_RCVBUF
canal.instance.network.sendBufferSize #網絡鏈接參數，SocketOptions.SO_SNDBUF
canal.instance.network.soTimeout #網絡鏈接參數，SocketOptions.SO_TIMEOUT
作者：獻給記性不好的自己
鏈接：https://www.jianshu.com/p/6299048fad66
來源：簡書
簡書著作權歸作者所有，任何形式的轉載都請聯繫作者獲得授權並註明出處。