Logstash多表同步MySQL數據到ES

1、Logstash下載

前往官網下載logstash，下載地址https://www.elastic.co/downloads/logstash

解壓

2、下載mysql-connector-java jar包

在logstash目錄（\bin同級）新建mysql目錄，將下載好的mysql-connector-java-5.1.34.jar放入此目錄；

在mysql目錄新建jdbc.conf文件，此文件將配置數據庫連接信息、查詢數據sql、分頁信息、同步頻率等核心信息。

3、配置文件

多表同步時，多個jdbc{}，output中多個判斷。下例爲全量更新

注意：在elasticsearch6.0中，一個索引下只能有一個類型。

input {
    stdin {}
    jdbc {
        jdbc_driver_library => "mysql/mysql-connector-java-6.0.6.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        jdbc_connection_string => "jdbc:mysql://sandboxo2o.mysql.rds.aliyuncs.com:3306/food"
        jdbc_user => "petkitfood"
        jdbc_password => "petkit123"
        jdbc_paging_enabled => "true"
        jdbc_page_size => "100000"
        clean_run => true
        use_column_value => true
        tracking_column => "id"
        last_run_metadata_path => "/Users/menglinjie/ES-node/testdata.text"
        statement => "select * from user"
        schedule => "0 1 * * * *"
        type => "food_user"
   }
   jdbc {
        jdbc_driver_library => "mysql/mysql-connector-java-6.0.6.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        jdbc_connection_string => "jdbc:mysql://sandboxo2o.mysql.rds.aliyuncs.com:3306/food"
        jdbc_user => "petkitfood"
        jdbc_password => "petkit123"
        jdbc_paging_enabled => "true"
        jdbc_page_size => "100000"
        clean_run => true
        use_column_value => true
        tracking_column => "id"
        last_run_metadata_path => "/Users/menglinjie/ES-node/testdata.text"
        statement => "select * from store_vip_pet"
        schedule => "0 1 * * * *"
        type => "food_pet"
   }
}
output {
   stdout {
        codec => json_lines
   }
   if[type] == "food_user"{
        elasticsearch {
        hosts => ["192.168.7.95:9200"]
        index => "food_user_index"
        document_id => "%{id}"
        document_type => "user"
   }
    }
   if[type] == "food_pet"{
        elasticsearch {
        hosts => ["192.168.7.95:9200"]
        index => "food_pet_index"
        document_id => "%{id}"
        document_type => "pet"
   }
    }
}

4、啓動Logstash

進入 Logstash的bin目錄下：

./logstash -f mysql/jdbc.conf

5、查看kibana中索引和文檔

Kibana安裝與漢化