使用Logstash將Mysql數據增量同步到ElasticSearch搜索引擎

下載logstash：https://www.elastic.co/cn/downloads/logstash/

解壓到目錄/usr/local下

安裝插件

bin/logstash-plugin install logstash-input-jdbc

bin/logstash-plugin install logstash-output-elasticsearch

出現錯誤，Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c5330000, 986513408, 0) failed; error='Cannot allocate memory' (errno=12)
------------------

修改JVM啓動參數  vim config/jvm.options 

-Xms256m  

-Xmx256m 

再次安裝，成功

下載jdbc驅動到目錄

/usr/local/logstash-7.0.0/es/mysql-connector-java-8.0.13.jar

編寫配置文件，使用表內自定義的date字段來設爲判斷增量

input {
  jdbc {
    jdbc_driver_library => "/usr/local/logstash-7.0.0/es/mysql-connector-java-8.0.13.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://ip:3306/realfake"
    jdbc_user => "root"
    jdbc_password => "password"
    schedule => "* * * * *"
    statement => "SELECT * FROM commodity WHERE date >= :sql_last_value"
    use_column_value => true
    tracking_column_type => "numeric"
    tracking_column => "date"
    last_run_metadata_path => "syncpoint_table"
  }
}

output {
  elasticsearch {
    hosts => ["ip1", "ip2"]
    index => "commodity"
    document_type => "commodity"
    document_id => "%{comm_id}"
  }
}

保存爲es.conf，運行，記住logstash千萬不要在es服務器上運行，否則會導致es節點崩潰

nohup bin/logstash -f es/es.conf &

查看運行狀態 jobs -l

往數據庫中添加數據，查看效果

輸入指令，查看nohup日誌

tail -fn 20 nohup.out

查看es數據

成功