logstash es同步 部分數據丟失

錯誤信息

es，查詢商品數據，沒有查詢到條碼信息。mysql中條碼信息是存在的
測試同學反饋的抓包信息。商品條碼(code)不存在

排查過程

查看es中的數據，數據從identifies這個對象中獲取，該對象中數據爲空。這條數據上有tag (_jdbcstreamingfailure)，這個有點奇怪

查看對應的說明，是同步過程中出現錯誤添加的默認標籤(https://www.elastic.co/guide/en/logstash/7.4/plugins-filters-jdbc_streaming.html#plugins-filters-jdbc_streaming-tag_on_failure)

找到對應的logstash相關時間點的日誌，發現mysql連接有異常

[2019-09-23T11:40:01,523][WARN ][logstash.filters.jdbcstreaming] Exception when executing JDBC query 
{:exception=>#<Sequel::DatabaseError: Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: 
The last packet successfully received from the server was 249,761,473 milliseconds ago. 
The last packet sent successfully to the server was 249,761,473 milliseconds ago. is longer than the server configured value of 'wait_timeout'. 
You should consider either expiring and/or testing connection validity before use in your application, 
increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.>}

---

#問題原因
網上查到的資料，分析的主要原因爲MySQL斷開了空閒的連接。
詳情可參考https://blog.csdn.net/pandajava/article/details/41946251

查看了對應的環境設置

---

解決思路

1. mysql jdbc connection增加了autoReconnect配置 (網上資料說這個只對mysql 4.X生效，5之後的版本不生效。我們的是5.7，未單獨驗證)
2. 增加jdbc_validate_connection配置
   

https://www.elastic.co/guide/en/logstash/7.4/plugins-filters-jdbc_streaming.html#plugins-filters-jdbc_streaming-jdbc_validate_connection

修改配置後，觀察了1d運行情況，未出現之前的錯誤信息