SpringBoot -- ES從mysql中獲取數據/Elasticsearch-jdbc

前置工作

• 當前服務器爲CentOS6.5+ 64bit

• elasticsearch與springboot已經正常運行，可以參考上一篇 SpringBoot集成ES

• 獲取Elasticsearch-jdbc以及support ES版本

• 獲取地址爲：http://xbib.org/repository/org/xbib/elasticsearch/importer/elasticsearch-jdbc/2.3.4.0/elasticsearch-jdbc-2.3.4.0-dist.zip
• 如果是在release裏獲取的版本，則沒有 lib目錄，在後面的腳本運行中會報錯

服務器安裝 dos2unix

• 編寫shell腳本後上傳服務器，如果不進行轉換則會出現各種錯誤

安裝Elasticsearch-jdbc

wget http://xbib.org/repository/org/xbib/elasticsearch/importer/elasticsearch-jdbc/2.3.4.0/elasticsearch-jdbc-2.3.4.0-dist.zip 

unzip elasticsearch-jdbc-2.3.4.0-dist.zip 

安裝dos2unix

yum install dos2unix

創建shell腳本

全量獲取數據

mysql_accountinfo.sh

#!/bin/sh

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
bin=${DIR}/../bin
lib=${DIR}/../lib

echo '{
    "type":"jdbc",
    "jdbc":{
        "url":"jdbc:mysql://localhost:3306/kakme",
        "user":"root",
        "password":"123456",
        "sql":"select id as _id , account_name as accountName , nick_name nickName from account_info",
        "elasticsearch" : {
            "cluster" : "nini",
            "host" : "localhost",
            "port" : 9300
        },
        "index":"cwenao",
        "type":"accountinfo",

        "type_mapping" :{
            "account_info": {
                "properties": {
                    "id":{
                        "type":"string",
                        "index":"not_analyzed"
                    },
                    "accountName":{
                        "type":"string"
                    },
                    "nickName":{
                        "type":"string"
                    }
                }
            }
        }
    }
}' | java \
    -cp "${lib}/*" \
    -Dlog4j.configurationFile=${bin}/log4j2.xml \
    org.xbib.tools.Runner \
    org.xbib.tools.JDBCImporter

上傳服務器並用dos2unix進行格式轉換

• 運行前需要更改權限
• chmod 700 mysql_accountinfo.sh

dos2unix mysql_accountinfo.sh
chmod 700 mysql_accountinfo.sh
sh mysql_accountinfo.sh

查看導入結果

curl -XGET 'http://localhost:9200/cwenao/accountinfo/_search?pretty'

查詢結果

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "cwenao",
      "_type" : "accountinfo",
      "_id" : "a63126d074f04db587cd76a48c817509",
      "_score" : 1.0,
      "_source" : {
        "accountName" : "cwenao",
        "nickName" : null
      }
    }, {
      "_index" : "cwenao",
      "_type" : "accountinfo",
      "_id" : "a63126d074f04db587cd76a48c817510",
      "_score" : 1.0,
      "_source" : {
        "accountName" : "nini",
        "nickName" : "啦啦啦啦啦啦啦阿里"
      }
    } ]
  }
}

增量數據

• 增加增量條件存儲文件： statefile
• 執行時間： schedule
• 增量查詢sql

mysql_accountinfo_time.sh

#!/bin/sh

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
bin=${DIR}/../bin
lib=${DIR}/../lib

echo '{
    "type":"jdbc",
    "jdbc":{
        "url":"jdbc:mysql://localhost:3306/kakme",
        "user":"root",
        "password":"123456",        
        "statefile":"statefile_account.json",
        "schedule" : "0 0-59 0-23 ? * *",
        "sql": [{
            "statement":"select id as _id , account_name as accountName , nick_name nickName from account_info where update_time > ?",
            "parameter" : ["$metrics.lastexecutionstart"]
        }],

        "elasticsearch" : {
            "cluster" : "nini",
            "host" : "localhost",
            "port" : 9300
        },
        "index":"cwenao",
        "type":"accountinfo",
        "type_mapping" :{
            "account_info": {
                "properties": {
                    "id":{
                        "type":"string",
                        "index":"not_analyzed"
                    },
                    "accountName":{
                        "type":"string"
                    },
                    "nickName":{
                        "type":"string"
                    }
                }
            }
        }
    }
}' | java \
    -cp "${lib}/*" \
    -Dlog4j.configurationFile=${bin}/log4j2.xml \
    org.xbib.tools.Runner \
    org.xbib.tools.JDBCImporter

上傳服務器並用dos2unix進行格式轉換

• 運行前需要更改權限
• chmod 700 mysql_accountinfo.sh

dos2unix mysql_accountinfo_time.sh
chmod 700 mysql_accountinfo_time.sh
sh mysql_accountinfo_time.sh

數據庫插入數據

INSERT INTO `kakme`.`account_info` (
    `id`,
    `account_name`,
    `nick_name`,
    `mail`,
    `m_tel`,
    `land_tel`,
    `is_vip`,
    `password`,
    `salt`,
    `head_image`,
    `back_image`,
    `account_type`,
    `source`,
    `audit_status`,
    `create_time`,
    `update_time`,
    `status`
)
VALUES
    (
        'a63126d074f04db587cd76a48c817512',
        'jintiantianq',
        '你說天氣如何',
        NULL,
        NULL,
        NULL,
        NULL,
        '12b66f880fddbbabf279e64076e288fc',
        '112233',
        NULL,
        NULL,
        NULL,
        NULL,
        NULL,
        '2017-02-09 11:32:14',
        '2017-02-09 11:32:14',
        '1'
    );

查看導入結果

curl -XGET 'http://localhost:9200/cwenao/accountinfo/_search?pretty'

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "cwenao",
      "_type" : "accountinfo",
      "_id" : "a63126d074f04db587cd76a48c817509",
      "_score" : 1.0,
      "_source" : {
        "accountName" : "cwenao",
        "nickName" : null
      }
    }, {
      "_index" : "cwenao",
      "_type" : "accountinfo",
      "_id" : "a63126d074f04db587cd76a48c817510",
      "_score" : 1.0,
      "_source" : {
        "accountName" : "nini",
        "nickName" : "啦啦啦啦啦啦啦阿里"
      }
    } ]
  }
}

---

代碼

代碼請移步 Github參考地址

如有疑問請加公衆號（K171），如果覺得對您有幫助請 github start