otter學習（九）——常見報錯處理

一、binlog文件被清理：Could not find first log file name in binary log index file

1.報錯日誌

2.現象

如下圖所示，pipeline的mainstem狀態一直處於定位中狀態

3.問題排查

一般出現這個報錯，都是由於運維同學誤清空數據庫binlog文件導致，我們按下述步驟確定是否由於binlog文件被清理：

首先，查看當前同步的binlog位點：

其次：登錄數據庫查看binlog文件信息（查詢sql爲：show master logs;）：

通過上述比對，發現數據庫的binlog被清理掉了，被清理前，otter標記的位點爲mysql-bin.000245這個文件，被清理後，mysql重新從1開始生成binlog文件導致otter同步失敗

4.處理辦法：

1.清空掉otter的同步信息

2.檢查canal的同步位點配置

3.重新啓動otter同步

二、mysql大事務造成otter假死

1.報錯日誌

無報錯日誌

2.現象

channel狀態正常，mainstem狀態也是工作中，但是position信息裏，position的信息一直不更新（超過半小時以上）

3.確認是否爲大事務的方法

首先，登錄對應倉庫的數據庫，先查詢當前數據庫的binlog文件跑到哪裏了

查詢sql爲 show master logs;

然後查詢position信息裏卡住的binlog文件信息，判斷是否產生了大事務

查詢sql爲show binlog events in '你要查詢的binlog名稱';（例如：show binlog events in 'mysql-bin.000235';）

（下圖爲查詢mysql-bin.000235這個binlog文件，發現由於源庫做切表處理，產生了大事務）

4.解決方法

從我們卡住的位點，依次往後面查詢binlog,找到這個大事務終結的位點。然後手動更新canal的位點信息，刪除同步記錄，然後重啓channel

a.清空掉otter的同步信息

b.檢查canal的同步位點配置

c.重新啓動otter同步

三、node內存溢出：java.lang.OutOfMemoryError: unable to create new native thread

1.報錯日誌

2.現象

node節點均爲運行中狀態，但是涉及到問題node的channel的同步均處於掛起狀態，且無法解掛及停止

3.問題排查

一般出現這個報錯，基本上都是node服務器上內存溢出，會導致我們一些操作無法執行

首先，根據報錯log上的nid找到對應的node

其次：嘗試登錄node服務器，看是否能重啓node

如果服務器也無法登陸，說明是服務器掛掉了，問題基本上明瞭了

如果服務器可以登陸，嘗試重啓node服務

4.處理辦法：

聯繫運擁有阿里雲服務器權限的運維同事，幫忙重啓對應的服務器

四、node節點掛掉

1.現象

Channel配置顯示爲掛起或者停止狀態，並且啓動或者解掛操作爲置灰不可操作。可以判定爲node節點掛掉了

2.排查

進入node管理頁面，查看是否有node節點狀態處於未啓動狀態

3.解決方法

根據掛掉的node節點，登錄對應的服務器，進行node服務的重啓操作，node服務器信息如下

node節點名稱	服務器ip	服務器訪問賬號	node路徑
node節點名稱	服務器ip	服務器訪問賬號	node路徑
node-247	10.44.50.247	root/Geekplus@2017	/usr/local/geekplus/software/node.deployer-4.2.16
node-248	10.44.50.248	root/Geekplus@2017	/usr/local/geekplus/software/node.deployer-4.2.16
node-249	10.44.50.249	root/Geekplus@2017	/usr/local/geekplus/software/node.deployer-4.2.16

登錄服務器後，輸入停止node服務命令

sh /usr/local/geekplus/software/node.deployer-4.2.16/bin/stop.sh

待node服務關閉後，輸入node服務啓動命令

sh /usr/local/geekplus/software/node.deployer-4.2.16/bin/start.sh

重啓完node後，進入otter主頁面進行channel的解掛或者啓動操作（http://10.44.50.248:21000 用戶名/密碼：admin/admin

五、otter網絡故障

監控日誌中有類似以下內容時：檢查從阿里雲上是否能訪問到數據庫ip，ping是否通，不通可能是現場的vpn斷了，找網絡運維恢復。如果ping是通的，嘗試停止同步後恢復。

pid:14 nid:1 exception:canal:dataplatform-成都主庫-業務數據:com.alibaba.otter.canal.parse.exception.CanalParseException: java.io.IOException: connect /192.168.116.16:3306 failure
Caused by: java.io.IOException: connect /192.168.116.16:3306 failure
at com.alibaba.otter.canal.parse.driver.mysql.MysqlConnector.connect(MysqlConnector.java:77)
at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.connect(MysqlConnection.java:86)
at com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser.preDump(MysqlEventParser.java:85)
at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:175)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.alibaba.otter.canal.parse.driver.mysql.socket.BioSocketChannelPool.open(BioSocketChannelPool.java:18)
at com.alibaba.otter.canal.parse.driver.mysql.socket.SocketChannelPool.open(SocketChannelPool.java:18)
at com.alibaba.otter.canal.parse.driver.mysql.MysqlConnector.connect(MysqlConnector.java:72)
... 4 more

pid:10 nid:1 exception:setl:com.alibaba.otter.node.etl.select.exceptions.SelectException: com.google.common.collect.ComputationException: org.apache.commons.lang.exception.NestableRuntimeException: find table [wms.t_pick_wave_detail] error
at com.alibaba.otter.node.etl.select.selector.MessageParser.parse(MessageParser.java:211)
at com.alibaba.otter.node.etl.select.selector.canal.CanalEmbedSelector.selector(CanalEmbedSelector.java:308)
at com.alibaba.otter.node.etl.select.SelectTask.processSelect(SelectTask.java:236)
at com.alibaba.otter.node.etl.select.SelectTask.access$300(SelectTask.java:94)
at com.alibaba.otter.node.etl.select.SelectTask$1.run(SelectTask.java:208)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.common.collect.ComputationException: org.apache.commons.lang.exception.NestableRuntimeException: find table [wms.t_pick_wave_detail] error
at com.google.common.collect.MapMaker$ComputingMapAdapter.get(MapMaker.java:889)
at com.alibaba.otter.node.etl.common.db.dialect.AbstractDbDialect.findTable(AbstractDbDialect.java:109)
at com.alibaba.otter.node.etl.common.db.dialect.AbstractDbDialect.findTable(AbstractDbDialect.java:113)
at com.alibaba.otter.node.etl.select.selector.MessageParser.internParse(MessageParser.java:434)
at com.alibaba.otter.node.etl.select.selector.MessageParser.internParse(MessageParser.java:390)
at com.alibaba.otter.node.etl.select.selector.MessageParser.parse(MessageParser.java:183)
... 9 more
Caused by: org.apache.commons.lang.exception.NestableRuntimeException: find table [wms.t_pick_wave_detail] error
at com.alibaba.otter.node.etl.common.db.dialect.AbstractDbDialect$2.apply(AbstractDbDialect.java:188)
at com.alibaba.otter.node.etl.common.db.dialect.AbstractDbDialect$2.apply(AbstractDbDialect.java:172)
at com.google.common.collect.ComputingConcurrentHashMap$ComputingValueReference.compute(ComputingConcurrentHashMap.java:356)
at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.compute(ComputingConcurrentHashMap.java:182)
at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.getOrCompute(ComputingConcurrentHashMap.java:151)
at com.google.common.collect.ComputingConcurrentHashMap.getOrCompute(ComputingConcurrentHashMap.java:67)
at com.google.common.collect.MapMaker$ComputingMapAdapter.get(MapMaker.java:885)
... 14 more
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:331)
at com.alibaba.otter.shared.common.utils.meta.DdlUtils.findTable(DdlUtils.java:120)
at com.alibaba.otter.node.etl.common.db.dialect.AbstractDbDialect$2.apply(AbstractDbDialect.java:179)
... 20 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor134.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:341)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2251)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2284)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2083)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:806)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:410)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:328)
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)
at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:79)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
... 23 more
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
... 42 more