mysql 主备复制停止问题定位以及状态监控

1 主备复制停止定位

配置好主从复制后，没有进行主备复制。通过日志查看，发现从库在同步的时候跑错了。
在从库上查看日志：

vim /var/log/mysqld.log

提示error信息是函数创建失败，缺少DETERMINISTIC。并给出了修改建议：

Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql-bin.000003' position 25395

查看slave的状态：

show slave status\G;

返回结果中明确表示，slave sql不再运行。其中的错误提示，和log中看到的一致。

2 计划修复的方案

越过此条log，继续之后的执行。
在主库的机器上，查看主库的log：

show binlog events IN 'mysql-bin.000003' FROM 54911 LIMIT 50;#需要指定哪个文件（上面的log中有），起始position（依旧在上面的log中）

结果：

从log看到，下一条的position是多少。
在从库中重新定位同步的起始位置，position就是上面找到的信息。

stop slave;
change master to master_log_file='mysql-bin.000003',master_log_pos=47245;
start slave;

查看状态，确认启动同步成功：

show slave status\G;

3 线上监控脚本

如果是线上环境，需要监控slave库的状态，及时告警。这里写了一个python的脚本，很简陋。
安装python：我的环境还有py2，所以这里指定的名字中都带“3”

yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gcc make
wget https://www.python.org/ftp/python/3.6.2/Python-3.6.2.tar.xz
tar -xvJf  Python-3.6.2.tar.xz
./configure prefix=/usr/local/python3
make && make install
ln -s /usr/local/python3/bin/python3 /usr/bin/python3
export PATH=$PATH:/usr/local/python3/bin

安装setuptools：

wget https://pypi.python.org/packages/source/s/setuptools/setuptools.zip --no-check-certificate
unzip setuptools.zip
cd setuptools
python3 setup.py build
python3 setup.py install

安装pip：

wget wget https://pypi.python.org/packages/source/p/pip/pip.tar.gz
tar xvf pip.tar.gz
cd pip
python3 setup.py install

注意：各个软件的版本下载最新的就好
安装python的依赖包：

pip install PyMySQL
pip install schedule

监控脚本分为三个文件，一个main以及mysql监控和发送邮件：

发送邮件
'''
Created on 2018-3-21

@author: yunzhong
'''

import smtplib
from email.mime.text import MIMEText
from email.header import Header

class MailSender(object):
    '''
    classdocs
    '''
    mailHost="smtp.exmail.qq.com"
    mailPort=465
    mailProtocol="smtp"
    mailUsername="[email protected]"
    mailPassword="mail"
    receivers = ['[email protected]']

    def __init__(self):
        '''
        Constructor
        '''
    
    def sendEmail(self, headFrom, headTo, subject,content):
        message = MIMEText(content, 'plain', 'utf-8')
        message['From'] = Header(headFrom, 'utf-8')
        message['To'] =  Header(headTo, 'utf-8')
        
        subject = subject
        message['Subject'] = Header(subject, 'utf-8')
        
        
        try:
            smtpObj = smtplib.SMTP_SSL()
            smtpObj.connect(self.mailHost, self.mailPort)
            smtpObj.login(self.mailUsername,self.mailPassword)
            smtpObj.sendmail(self.mailUsername, self.receivers, message.as_string())
            print ("错误，发送电子邮件")
        except smtplib.SMTPException as err:
            print ("Error: 不能发送邮件",err)

mysql监控
'''
Created on 2018-3-21

@author: yunzhong
'''
import pymysql

class MysqlMonitor(object):
    '''
    classdocs
    '''
    mysqlHost = 'host.com'
    user = 'user'
    password = 'root+-*/'
    database = ''
   
    def monitor(self):
        db = pymysql.connect(self.mysqlHost,self.user,self.password,self.database )
        cursor = db.cursor()
        cursor.execute("SHOW SLAVE STATUS")
        results = cursor.fetchall()
        print(results)
        slaveIO = 0
        slaveSql = 0
        for result in results:
            print("host:",result[1])
            print("host:",result[5])
            print("host:",result[10])
            print("host:",result[11])
            if result[10].lower() == 'no' :
                slaveIO = 1
            if result[11].lower() == 'no':
                slaveSql = 10
        db.close()
        return slaveIO + slaveSql

main文件
'''
Created on 2018-3-21

@author: yunzhong
'''
import schedule
import time
from mysqlMonitor import MysqlMonitor
from mailSender import MailSender

if __name__ == '__main__':
    pass

def mo():
    print(time.strftime('%Y-%m-%d %H:%M:%S',time.localtime(time.time())),"开始获取数据库slave信息")
    monitor = MysqlMonitor()
    mResult = monitor.monitor()
    print(time.strftime('%Y-%m-%d %H:%M:%S',time.localtime(time.time())),"数据库slave监控信息",mResult)
    if mResult == 1:
        sender = MailSender()
        sender.sendEmail('mysql告警邮件', '管理员', '数据库同步监控', '主从数据库运行异常，请核查。数据库slave IO异常')
    elif mResult == 10:
        sender = MailSender()
        sender.sendEmail('mysql告警邮件', '管理员', '数据库同步监控', '主从数据库运行异常，请核查。数据库slave SQL异常')
    elif mResult > 0:
        sender = MailSender()
        sender.sendEmail('mysql告警邮件', '管理员', '数据库同步监控', '主从数据库运行异常，请核查。数据库slave SQL异常')
    else:
        print(time.strftime('%Y-%m-%d %H:%M:%S',time.localtime(time.time())),"数据库slave运行正常")

try:
    print(time.strftime('%Y-%m-%d %H:%M:%S',time.localtime(time.time())),"数据库slave监控程序开始执行")
    schedule.every(1).minutes.do(mo)
except BaseException as err:
    print ("Error: 不能发送邮件",err)
while True:
    try:
        schedule.run_pending()
        time.sleep(1)
    except BaseException as err:
        print ("Error: 不能发送邮件",err)

python脚本启动命令：-u参数，是为了print无缓存，直接输出到console

nohup python3 -u monitor.py > ./monitor.log 2>&1 &