logstash同步mysql數據到elasticsearch

安裝 jdbc 和 elasticsearch 插件

/usr/share/logstash/bin/logstash-plugin install logstash-input-jdbc
/usr/share/logstash/bin/logstash-plugin install logstash-output-elasticsearch

下載mysql-connector-java

cd /usr/share/logstash/logstash-core/lib/jars
wget http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.46/mysql-connector-java-5.1.46.jar

編寫配置文件 logstash.conf
tips:數據庫中刪除的數據無法同步到ES中，只能同步insert,update數據

input {
  stdin{
  }
  jdbc {
    type=>"task_desc_info"
    jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/db_dev?useUnicode=true&characterEncoding=utf-8&useSSL=false"
    jdbc_user => "root"
    jdbc_password => "147258369"
    jdbc_driver_library => ""
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_paging_enabled => true
    jdbc_page_size => "50000"
    jdbc_default_timezone =>"Asia/Shanghai"

    tracking_column => "auto_id"
    record_last_run => "true"
    use_column_value => "true"
    last_run_metadata_path => "task_desc"
    clean_run => "false"

    statement => "select * from task_description where auto_id > :sql_last_value"
    schedule => "* * * * *"
  }
}
output{
  if [type]=="task_desc_info" {
    stdout {
      codec => json_lines
    }
    elasticsearch {
      hosts => "127.0.0.1:9200"
      index => "task_desc"
      document_id => "%{task_id}"
    }
  }
}                                                                                                   

啓動

/usr/share/logstash/bin/logstash/ -f logstash.conf

同步多張表
編寫/pipelines.yml

- pipeline.id: table1
  path.config: "/path/tabale1.conf"
- pipeline.id: table2
  path.config: "/path/table2.conf"
- pipeline.id: table3
  path.config: "/path/table3.conf"

參考：https://www.cnblogs.com/qdhxhz/p/9520685.html