logstash jdbc同步mysql多表數據到elasticsearch

一、環境準備

1. elasticsearch5.5.3
2. logstash5.5.3
3. mysql5.6.37
4. mysql表中必須有時間戳字段

二、測試條件

1. 同步兩張表數據到索引
   
   • 以表名作爲es索引名稱
   • 以表的註解id作爲es的object id

三、書寫logstash配置文件,並命名爲dbToEs.conf

input {
   jdbc {
    jdbc_driver_library => "mysql-connector-java-5.1.36.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://192.168.1.151:3306/mydb"
    jdbc_user => "root"
    jdbc_password => "root@hiekn"
    schedule => "* * * * *"
    statement => "SELECT * from tb_user where lastModifyTime > :sql_last_value"
    add_field => { "[@metadata][type]" => "mydb" } 
  }
  jdbc {
    jdbc_driver_library => "mysql-connector-java-5.1.36.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://192.168.1.151:3306/mydb2"
    jdbc_user => "root"
    jdbc_password => "root@hiekn"
    schedule => "* * * * *"
    statement => "SELECT * from tb_user where lastModifyTime > :sql_last_value"
    add_field => { "[@metadata][type]" => "mydb2" } 
  }
}

filter {
    json {
        source => "message"
        remove_field => ["message"]
    }
    mutate  {
        remove_field => ["@timestamp","@version"]
    }
}

output {
    if [@metadata][type] == "mydb" {
       elasticsearch {
            hosts => [ "192.168.1.157:9200" ]
            #manage_template => false
            index => "mydb"
            document_id => "%{id}"
            document_type => "mydb_data"
        }
    } else if [@metadata][type] == "mydb2"{
        elasticsearch {
            hosts => [ "192.168.1.157:9200" ]
            index => "mydb2"
            document_id => "%{id}"
            document_type => "mydb2_data"
        }
    }
}

四、運行logstash

1. 將配置文件放在bin目錄下
2. 執行./logstash -f ./dbToEs.conf

五、說明

1. 第一次執行時間戳是以1970-01-01 00:00:00爲起點
2. logstash每間隔1分鐘執行一次sql