mysql主從搭建

背景

後臺工程師兼職做系統運維工程師不容易，碼字留念。入司前，研發團隊後臺開發一個都沒有，入職時跟一羣外包同事摸爬滾打3個月，算是工作交接，因爲外包朋友算是拿多少錢幹多少活的主，所以數據庫這塊一直是單點部署，有時候做夢都夢見主庫掛掉啦，數據無法恢復，直接打包走人的場景，本着數據是一個公司的生命的認知力，決定把數據庫這塊做個實時備份的從庫，然後就在網上各種搜爬，網上教程倒不少，但是跟着配了很多方案都沒有配置成功,最後還是藉助以前老哥 @威哥的淫威之下才把任務完成。下面我們開始吧。

環境配置

|名稱|版本|IP|備註|配置|

安裝mysql

解壓


tar -zxvf mysql-5.6.33-linux-glibc2.5-x86_64.tar.gz

cp -r mysql-5.6.33-linux-glibc2.5-x86_64 /usr/local/mysql

添加用戶組和用戶


#添加用戶組
groupadd mysql
#添加用戶mysql 到用戶組mysql
useradd -g mysql mysql

安裝


#創建mysql數據存儲目錄

cd /usr/local/mysql/
mkdir ./data/mysql
#修改成mysql權限
chown -R mysql:mysql ./
#
./scripts/mysql_install_db --user=mysql --datadir=/usr/local/mysql/data/mysql
#把mysqld服務加到系統服務中
cp support-files/mysql.server /etc/init.d/mysqld
#修改可執行權限
chmod 755 /etc/init.d/mysqld
#mysql啓動配置文件默認讀取路徑
cp support-files/my-default.cnf /etc/my.cnf


#修改啓動腳本
vi /etc/init.d/mysqld
#修改項：
basedir=/usr/local/mysql/

datadir=/usr/local/mysql/data/mysql
#啓動服務
service mysqld start
#測試連接
./mysql/bin/mysql -uroot
#加入環境變量，編輯 /etc/profile，這樣可以在任何地方用mysql命令了
export PATH=$PATH:/usr/local/mysql//bin
source /etc/profile
#啓動mysql
service mysqld start
#關閉mysql
service mysqld stop
#查看運行狀態
service mysqld status


#授權

use mysql;
CREATE USER canal IDENTIFIED BY 'salve';
#只讀權限從庫用戶權限
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'salve'@'%';
-- GRANT ALL PRIVILEGES ON *.* TO 'slave'@'%' ;
FLUSH PRIVILEGES;
或
GRANT ALL PRIVILEGES ON *.* TO `salve`@'%’ IDENTIFIED BY ‘youpassword’ WITH GRANT OPTION;
FLUSH PRIVILEGES;

兩臺服務器都執行以上安裝操作

主從配置

方式:GTID方式


GTID
Global transaction identifiers
可以理解爲一個事務對應一個唯一ID
一個GTID在一個服務器上只會執行一次
GTID是用來代替傳統複製的方法
MySQl-5.6.2開始支持，5.6.10後完善了
GTID的組成
Server_uuid:sequence number

配置注意

修改主庫192.168.199.129:/etc/my.cnf文件，需要加入以下參數需要放在[mysqld]下面


gtid_mode=on

enforce-gtid-consistency=on

server-id = 813316

log-bin = /usr/local/mysql/data

binlog_format = row

log-slave-updates=1

binlog-gtid-simple-recovery=1

關閉/重啓服務


[root@localhost ~]# service mysqld restart

Starting MySQL.... SUCCESS!

登錄mysql，查詢主庫配置信息是否生效


[root@localhost etc]# mysql -uroot

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 1

Server version: 5.6.33-log MySQL Community Server (GPL)



Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.



Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.



Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.



mysql> show master status;

+-------------+----------+--------------+------------------+------------------------------------------+

| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |

+-------------+----------+--------------+------------------+------------------------------------------+

| data.000004 | 191 | | | fd736651-d0a2-11e9-b357-000c293bc199:1-3 |

+-------------+----------+--------------+------------------+------------------------------------------+

1 row in set (0.00 sec)

修改從庫配置信息，192.168.199.131:/etc/my.cnf


gtid_mode=on

enforce-gtid-consistency=on

server-id = 813317

log-bin = /usr/local/mysql/data

binlog_format = row

log-slave-updates=1

binlog-gtid-simple-recovery=1

sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES

啓動後查看GTID是否生效


[root@zhanghp2 etc]# mysql -uroot

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 4

Server version: 5.6.33-log MySQL Community Server (GPL)



Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.



Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.



Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.



mysql> show master status;

+-------------+----------+--------------+------------------+------------------------------------------+

| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |

+-------------+----------+--------------+------------------+------------------------------------------+

| data.000004 | 191 | | | fd736651-d0a2-11e9-b357-000c293bc199:1-3 |

+-------------+----------+--------------+------------------+------------------------------------------+

1 row in set (0.00 sec)

以上配置結束，下面開始導出原有數據，然後倒入到從庫中。

導出數據


#-A是導出全庫， 可以把-A換成數據庫的名字， 比如liuxn liuxn3316

/usr/local/mysql/bin/mysqldump -uroot -pchenw44 -S /tmp/mysql.sock --master-data=2 --single-transaction -A >/home/mysql/db.sql

導入數據


#注意如果是一個壞掉的庫可能會報錯是一個關於GTID的值應該爲空的錯誤提示,需要在庫上執行 mysql> reset master

#查詢gtid是否開啓成功

show global variables like "%gtid%";

#導入

mysql -S /tmp/mysql3317.sock </home/mysql/db.sql

#導入後由於mysql庫也導入了，需要這樣連接進入

mysql -uroot -S /tmp/mysql3307.sock

從庫配置開始同步


#清除之前的從庫配置信息

reset slave all

#master_auto_position可以自動開啓查找日誌功能， 再自己查找了

change master to master_host='10.10.1.81',MASTER_PORT=3316,master_user='repl',master_password='repl4slave',master_auto_position=1;

查看同步配置show slave statusG;


mysql> show slave status\G;

*************************** 1. row ***************************

               Slave_IO_State: 

                  Master_Host: 192.168.199.129

                  Master_User: root

                  Master_Port: 3306

                Connect_Retry: 60

              Master_Log_File: data.000002

          Read_Master_Log_Pos: 1913

               Relay_Log_File: zhanghp2-relay-bin.000003

                Relay_Log_Pos: 4

        Relay_Master_Log_File: data.000002

             Slave_IO_Running: No

            Slave_SQL_Running: No

              Replicate_Do_DB: 

          Replicate_Ignore_DB: 

           Replicate_Do_Table: 

       Replicate_Ignore_Table: 

      Replicate_Wild_Do_Table: 

  Replicate_Wild_Ignore_Table: 

                   Last_Errno: 0

                   Last_Error: 

                 Skip_Counter: 0

          Exec_Master_Log_Pos: 1913

              Relay_Log_Space: 240

              Until_Condition: None

               Until_Log_File: 

                Until_Log_Pos: 0

           Master_SSL_Allowed: No

           Master_SSL_CA_File: 

           Master_SSL_CA_Path: 

              Master_SSL_Cert: 

            Master_SSL_Cipher: 

               Master_SSL_Key: 

        Seconds_Behind_Master: NULL

Master_SSL_Verify_Server_Cert: No

                Last_IO_Errno: 2003

                Last_IO_Error: error connecting to master '[email protected]:3306' - retry-time: 60 retries: 44

               Last_SQL_Errno: 0

               Last_SQL_Error: 

  Replicate_Ignore_Server_Ids: 

             Master_Server_Id: 0

                  Master_UUID: fd736651-d0a2-11e9-b357-000c293bc199

             Master_Info_File: /usr/local/mysql/data/mysql/master.info

                    SQL_Delay: 0

          SQL_Remaining_Delay: NULL

      Slave_SQL_Running_State: 

           Master_Retry_Count: 86400

                  Master_Bind: 

      Last_IO_Error_Timestamp: 190912 18:42:07

     Last_SQL_Error_Timestamp: 

               Master_SSL_Crl: 

           Master_SSL_Crlpath: 

           Retrieved_Gtid_Set: 

            Executed_Gtid_Set: fd736651-d0a2-11e9-b357-000c293bc199:1-3

                Auto_Position: 1

1 row in set (0.00 sec)



ERROR: 

No query specified

開始同步start slave


mysql> start slave;

Query OK, 0 rows affected (0.08 sec)

查看同步狀態


mysql> show slave status\G;

*************************** 1. row ***************************

               Slave_IO_State: Waiting for master to send event

                  Master_Host: 10.199.96.147

                  Master_User: root

                  Master_Port: 3306

                Connect_Retry: 60

              Master_Log_File: data.000001

          Read_Master_Log_Pos: 344899576

               Relay_Log_File: goodairnbapp04-relay-bin.000005

                Relay_Log_Pos: 330811707

        Relay_Master_Log_File: data.000001

             Slave_IO_Running: Yes

            Slave_SQL_Running: Yes

              Replicate_Do_DB: 

          Replicate_Ignore_DB: 

           Replicate_Do_Table: 

       Replicate_Ignore_Table: 

      Replicate_Wild_Do_Table: 

  Replicate_Wild_Ignore_Table: 

                   Last_Errno: 0

                   Last_Error: 

                 Skip_Counter: 0

          Exec_Master_Log_Pos: 344899576

              Relay_Log_Space: 330811889

              Until_Condition: None

               Until_Log_File: 

                Until_Log_Pos: 0

           Master_SSL_Allowed: No

           Master_SSL_CA_File: 

           Master_SSL_CA_Path: 

              Master_SSL_Cert: 

            Master_SSL_Cipher: 

               Master_SSL_Key: 

        Seconds_Behind_Master: 0

Master_SSL_Verify_Server_Cert: No

                Last_IO_Errno: 0

                Last_IO_Error: 

               Last_SQL_Errno: 0

               Last_SQL_Error: 

  Replicate_Ignore_Server_Ids: 

             Master_Server_Id: 813316

                  Master_UUID: 641e091b-93fb-11e8-a27d-801844ee17a0

             Master_Info_File: /export/mysql/data/mysql/master.info

                    SQL_Delay: 0

          SQL_Remaining_Delay: NULL

      Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it

           Master_Retry_Count: 86400

                  Master_Bind: 

      Last_IO_Error_Timestamp: 

     Last_SQL_Error_Timestamp: 

               Master_SSL_Crl: 

           Master_SSL_Crlpath: 

           Retrieved_Gtid_Set: 641e091b-93fb-11e8-a27d-801844ee17a0:1842-304615

            Executed_Gtid_Set: 641e091b-93fb-11e8-a27d-801844ee17a0:1-304615



#  Retrieved_Gtid_Set: 641e091b-93fb-11e8-a27d-801844ee17a0:1842-304615

表示從第1842個開始接收現在是第304615個

# Executed_Gtid_Set: 641e091b-93fb-11e8-a27d-801844ee17a0:1-304615

表示執行了304615個事務

驗證

在主庫test庫上建一個新表測試


CREATE TABLE `auth_client` (

  `id` int(11) NOT NULL AUTO_INCREMENT,

  `code` varchar(255) DEFAULT NULL COMMENT '服務編碼',

  `secret` varchar(255) DEFAULT NULL COMMENT '服務密鑰',

  `name` varchar(255) DEFAULT NULL COMMENT '服務名',

  `locked` char(1) DEFAULT NULL COMMENT '是否鎖定',

  `description` varchar(255) DEFAULT NULL COMMENT '描述',

  `crt_time` datetime DEFAULT NULL COMMENT '創建時間',

  `crt_user` varchar(255) DEFAULT NULL COMMENT '創建人',

  `crt_name` varchar(255) DEFAULT NULL COMMENT '創建人姓名',

  `crt_host` varchar(255) DEFAULT NULL COMMENT '創建主機',

  `upd_time` datetime DEFAULT NULL COMMENT '更新時間',

  `upd_user` varchar(255) DEFAULT NULL COMMENT '更新人',

  `upd_name` varchar(255) DEFAULT NULL COMMENT '更新姓名',

  `upd_host` varchar(255) DEFAULT NULL COMMENT '更新主機',

  `attr1` varchar(255) DEFAULT NULL,

  `attr2` varchar(255) DEFAULT NULL,

  `attr3` varchar(255) DEFAULT NULL,

  `attr4` varchar(255) DEFAULT NULL,

  `attr5` varchar(255) DEFAULT NULL,

  `attr6` varchar(255) DEFAULT NULL,

  `attr7` varchar(255) DEFAULT NULL,

  `attr8` varchar(255) DEFAULT NULL,

  PRIMARY KEY (`id`)

) ENGINE=InnoDB AUTO_INCREMENT=19 DEFAULT CHARSET=utf8mb4

在從庫查詢是否同步


mysql> show tables;

+----------------+

| Tables_in_test |

+----------------+

| auth_client |

+----------------+

1 row in set (0.00 sec)

bingo生效啦！！！

A & Q

mysql啓動不成功，查看mysql日誌


190910 16:55:17 mysqld_safe mysqld from pid file /export/mysql/data/mysql/goodairnbapp03.pid ended

190910 16:55:52 mysqld_safe Starting mysqld daemon with databases from /export/mysql/data/mysql

2019-09-10 16:55:53 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).

2019-09-10 16:55:53 0 [Note] /usr/local/mysql/mysql/bin/mysqld (mysqld 5.6.33-log) starting as process 33313 ...

/usr/local/mysql/mysql/bin/mysqld: File '/export/mysql/data.index' not found (Errcode: 13 - Permission denied)


# 數據存儲目錄權限問題

chown -R mysql:mysql /export/mysql

start slave報錯，開始同步不成功


show slave status \G; 

Slave_IO_Running: YesSlave_SQL_Running: No

2016-06-09 00:07:07 23352 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013)

2016-06-09 00:07:07 23352 [Note] Slave I/O thread killed while reading event

2016-06-09 00:07:07 23352 [Note] Slave I/O thread exiting, read up to log 'mysql-bin-190.000667', position 9049889

2016-06-09 00:07:07 23352 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.

2016-06-09 00:07:07 23352 [Warning] Slave SQL: If a crash happens this configuration does not guarantee that the relay log info will be consistent, Error_code: 0

2016-06-09 00:07:07 23352 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin-190.000666' at position 2935199, relay log './mysql3306-relay-bin.000002' position: 2935366

2016-06-09 00:07:07 23352 [Note] 'SQL_SLAVE_SKIP_COUNTER=1' executed at relay_log_file='./mysql3306-relay-bin.000002', relay_log_pos='2935366', master_log_name='mysql-bin-190.000666', master_log_pos='2935199' and new position at relay_log_file='./mysql3306-relay-bin.000002', relay_log_pos='2935750', master_log_name='mysql-bin-190.000666', master_log_pos='2935583' 

2016-06-09 00:07:07 23352 [Note] Slave I/O thread: connected to master '[email protected]:3306',replication started in log 'mysql-bin-190.000667' at position 9049889

2016-06-09 00:07:08 23352 [ERROR] Slave SQL: Could not execute Write_rows event on table asy.pm_camera_recordrate; Duplicate entry '913df478e36f4f888505874ddec59240' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log mysql-bin-190.000666, end_log_pos 2944382, Error_code: 1062

2016-06-09 00:07:08 23352 [Warning] Slave: Duplicate entry '913df478e36f4f888505874ddec59240' for key 'PRIMARY' Error_code: 1062

解決方式：在my.cnf文件中加入如下代碼到[mysqld]。重啓mysql


slave-skip-errors = 1062

如果能預估到錯誤數據比較少。也可以用如下代碼。每次執行只跳過一個事務。


stop slave; SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1 ; start slave ;

跳過1062和1032錯誤。可能會導致日誌中出現警告信息。所以在在my.cnf文件中加入如下代碼到[mysqld]。重啓mysql可解決問題。


binlog_format=mixed

反思問題起因

在從庫由於使用了 kill -9 殺掉了mysql線程。可能導致了mysql的事務回滾。也有可能導致了mysql中繼日誌出現問題。
在拉取備份文件恢復的時候。由於拉取了最新的備份數據。恢復數據的時候。只是重設了同步的binlog文件和pos點。並沒有修改中繼日誌的起點。導致了中繼日誌中的數據應該是比數據庫中的備份點數據更早了。然後產生了1062主鍵衝突和1032刪除數據不存在的錯誤。

背景

環境配置

安裝mysql

主從配置

驗證

A & Q

反思問題起因

SQL優化-20231016

mysql主從搭建

ELK-MACOS搭建部署（包含集羣）

電信NB平臺對接

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結