使用canal監視mysql庫表變化併發布到kafka

參考官方文檔

1. https://github.com/alibaba/canal/wiki/QuickStart
2. https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart

本地測試環境

1. mariadb: 5.5.60
2. kafka: 2.1.0
3. canal: 1.1.2

kafka安裝

測試環境爲單機版本，解壓後啓動就好。
下面爲自己寫的一個啓動腳本

#!/bin/bash
nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
if [ `ps aux|grep zookeeper|wc -l` -gt 1 ]
then
        nohup bin/kafka-server-start.sh config/server.properties &
fi

mariadb配置

修改/etc/my.cnf，添加或修改以下屬性：

log-bin=mysql-bin
binlog_format=ROW
server-id=1 #配置mysql replaction需要定義，不能和canal的slaveId重複
binlog-do-db = secure

重啓mariadb並進入mysql命令行

CREATE USER canal IDENTIFIED BY 'canal';  
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
-- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
FLUSH PRIVILEGES;

canal安裝

解壓

mkdir -p /usr/local/canal
cp   canal.deployer-1.1.2.tar.gz   /usr/local/canal
cd !$
tar -zxvf canal.deployer-1.1.2.tar.gz 

修改canal.properties

# ...
# 可選項: tcp(默認), kafka, RocketMQ
canal.serverMode = kafka
# ...
# kafka/rocketmq 集羣配置: 192.168.1.117:9092,192.168.1.118:9092,192.168.1.119:9092 
canal.mq.servers = 127.0.0.1:9092
canal.mq.retries = 0
canal.mq.batchSize = 16384
canal.mq.maxRequestSize = 1048576
canal.mq.lingerMs = 1
canal.mq.bufferMemory = 33554432
# Canal的batch size, 默認50K, 由於kafka最大消息體限制請勿超過1M(900K以下)
canal.mq.canalBatchSize = 50
# Canal get數據的超時時間, 單位: 毫秒, 空爲不限超時
canal.mq.canalGetTimeout = 100
# 是否爲flat json格式對象
canal.mq.flatMessage = true
canal.mq.compressionType = none
canal.mq.acks = all

修改conf/example/instance.properties

# 修改slaveId
canal.instance.mysql.slaveId=2
#  按需修改成自己的數據庫信息
#################################################
...
canal.instance.master.address=127.0.0.1:3306
# username/password,數據庫的用戶名和密碼
...
canal.instance.dbUsername = canal
canal.instance.dbPassword = canal
...
# mq config
canal.mq.topic=example
# 針對庫名或者表名發送動態topic, 表達式如果只有庫名則匹配庫名的數據都會發送到對應名稱topic
# 如果是庫名.表名則匹配的數據會發送到以庫名_表名爲名稱的topic
#canal.mq.dynamicTopic=mytest,.*,mytest.user,mytest\\..*,.*\\..*
canal.mq.partition=0
# hash partition config
#canal.mq.partitionsNum=3
#庫名.表名: 唯一主鍵，多個表之間用逗號分隔
#canal.mq.partitionHash=mytest.person:id,mytest.role:id
#################################################

啓動和查看日誌

sh bin/startup.sh
tailf logs/canal/canal.log