利用LogStash同步ORACLE數據至ES

1、LogStash

1.1下載Logstash

Wget https://artifacts.elastic.co/downloads/logstash/logstash-6.5.4.tar.gz

1.2項目結構

解壓縮後項目結構爲：

其中我們需要手動添加oracle的jar包作爲logstash的驅動。

1.3修改配置文件

作用是：每秒增量更新，每分鐘進行一次全量更新
刪除的話，目前logstash沒有策略實現。不過我們可以通過代碼實現。比如說每隔一段時間，更換一次index。刪除舊index，從而實現另類的刪除同步。

input {
  jdbc {//每秒更新新增數據
    jdbc_driver_library => "/data/elasticsearch/logstash/ojdbc6-11.2.0.3.jar"
    jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
    jdbc_connection_string => "jdbc:oracle:thin:@192.168.54.15:1521:ORCL"
    jdbc_user => "oracle"
    jdbc_password => "oracle"
    schedule => "* * * * * *"
    statement_filepath => "/data/elasticsearch/logstash/sql/test_oracle_one.sql"
    record_last_run => true
    use_column_value => true
    tracking_column => rn
    codec => plain { charset => "UTF-8"}
    tracking_column_type => numeric
    last_run_metadata_path => "/data/elasticsearch/logstash/metadata-path/test_oracle_one_last_id"
    clean_run => "false"
    jdbc_paging_enabled => true
    jdbc_page_size => 10000
  }
  jdbc {//每分鐘進行全量更新
    jdbc_driver_library => "/data/elasticsearch/logstash/ojdbc6-11.2.0.3.jar"
    jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
    jdbc_connection_string => "jdbc:oracle:thin:@192.168.54.15:1521:ORCL"
    jdbc_user => "oracle"
    jdbc_password => "oracle"
    schedule => "* * * * *"
    statement_filepath => "/data/elasticsearch/logstash/sql/test_oracle_one_all.sql"
    use_column_value => true
    tracking_column_type => numeric
    tracking_column => rn
    codec => plain { charset => "UTF-8"}
    jdbc_paging_enabled => true
    jdbc_page_size => 10000
  }
}
filter {
  date {
    match => [ "timestamp" , "yyyy/MM/dd HH:mm:ss Z" ]
  }
}
output {
  elasticsearch {
    hosts => ["192.168.54.11:9200"]
    index => "test_oracle_one"
    document_type => "_doc"
    document_id => "%{rn}"
  }
}

1.4 sql語句示例

test_oracle_one.sql:
SELECT * FROM(SELECT TL.*,ROWNUM rn FROM (SELECT * FROM TEST_LOGSTASH) TL) WHERE RN > :sql_last_value
test_oracle_one_all.sql:
SELECT TL.*,ROWNUM rn FROM (SELECT * FROM TEST_LOGSTASH) TL

1.5 pipeline.yml配置

作用是將配置文件加載進來，可寫多個

 - pipeline.id: test_oracle
  path.config: "/data/elasticsearch/logstash/conf/test_oracle.conf"

1.6 啓動

sh bin/logstash