利用LogStash同步ORACLE数据至ES

1、LogStash

1.1下载Logstash

Wget https://artifacts.elastic.co/downloads/logstash/logstash-6.5.4.tar.gz

1.2项目结构

解压缩后项目结构为：

其中我们需要手动添加oracle的jar包作为logstash的驱动。

1.3修改配置文件

作用是：每秒增量更新，每分钟进行一次全量更新
删除的话，目前logstash没有策略实现。不过我们可以通过代码实现。比如说每隔一段时间，更换一次index。删除旧index，从而实现另类的删除同步。

input {
  jdbc {//每秒更新新增数据
    jdbc_driver_library => "/data/elasticsearch/logstash/ojdbc6-11.2.0.3.jar"
    jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
    jdbc_connection_string => "jdbc:oracle:thin:@192.168.54.15:1521:ORCL"
    jdbc_user => "oracle"
    jdbc_password => "oracle"
    schedule => "* * * * * *"
    statement_filepath => "/data/elasticsearch/logstash/sql/test_oracle_one.sql"
    record_last_run => true
    use_column_value => true
    tracking_column => rn
    codec => plain { charset => "UTF-8"}
    tracking_column_type => numeric
    last_run_metadata_path => "/data/elasticsearch/logstash/metadata-path/test_oracle_one_last_id"
    clean_run => "false"
    jdbc_paging_enabled => true
    jdbc_page_size => 10000
  }
  jdbc {//每分钟进行全量更新
    jdbc_driver_library => "/data/elasticsearch/logstash/ojdbc6-11.2.0.3.jar"
    jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
    jdbc_connection_string => "jdbc:oracle:thin:@192.168.54.15:1521:ORCL"
    jdbc_user => "oracle"
    jdbc_password => "oracle"
    schedule => "* * * * *"
    statement_filepath => "/data/elasticsearch/logstash/sql/test_oracle_one_all.sql"
    use_column_value => true
    tracking_column_type => numeric
    tracking_column => rn
    codec => plain { charset => "UTF-8"}
    jdbc_paging_enabled => true
    jdbc_page_size => 10000
  }
}
filter {
  date {
    match => [ "timestamp" , "yyyy/MM/dd HH:mm:ss Z" ]
  }
}
output {
  elasticsearch {
    hosts => ["192.168.54.11:9200"]
    index => "test_oracle_one"
    document_type => "_doc"
    document_id => "%{rn}"
  }
}

1.4 sql语句示例

test_oracle_one.sql:
SELECT * FROM(SELECT TL.*,ROWNUM rn FROM (SELECT * FROM TEST_LOGSTASH) TL) WHERE RN > :sql_last_value
test_oracle_one_all.sql:
SELECT TL.*,ROWNUM rn FROM (SELECT * FROM TEST_LOGSTASH) TL

1.5 pipeline.yml配置

作用是将配置文件加载进来，可写多个

 - pipeline.id: test_oracle
  path.config: "/data/elasticsearch/logstash/conf/test_oracle.conf"

1.6 启动

sh bin/logstash