ElasticSearch 單機間索引數據遷移筆記

本文記錄的是在兩個單機ES服務之間的數據遷移。

索引數據遷移主要步驟

1、再舊服務上備份索引數據
2、在新服務上恢復索引數據

在舊服務上備份索引數據

第一步，先查看舊服務上的快照倉庫信息

獲取所有已註冊快照倉庫，省略倉庫名或者使用_all

curl -X GET "localhost:9200/_snapshot?pretty"

返回

{
  "elasticsearch_backup" : {
    "type" : "fs",
    "settings" : {
      "location" : "/usr/local/elasticsearch_backup"
    }
  }
}

查看倉庫中所有快照信息

curl -X GET "localhost:9200/_snapshot/elasticsearch_backup/_all?pretty"

返回

{
  "snapshots" : [ ]
}

第二步，創建最新快照備份

創建新快照前，先執行以下命令，確保緩存中的索引落到磁盤中：

curl -X POST "localhost:9200/_flush?pretty"

返回

{
  "_shards" : {
    "total" : 270,
    "successful" : 135,
    "failed" : 0
  }
}

創建快照

curl -X PUT "localhost:9200/_snapshot/elasticsearch_backup/snapshot-2018.09.26-01?wait_for_completion=true" -H 'Content-Type: application/json' -d'
{
  "ignore_unavailable": true,  
  "include_global_state": false
}                              
'

設值選項 ignore_unavailable=true，快照過程中會忽略不存在的索引，默認沒有設值，遇到不存在的索引，快照過程將失敗。設值 include_global_state 爲 false，可以阻止集羣全局狀態信息被保存爲快照的一部分。默認情況下，如果如果一個快照中的一個或者多個索引沒有所有主分片可用，整個快照創建會失敗，該情況可以通過設置 partial 爲 true 來改變。
上面命令返回

{
    "snapshot":{
        "snapshot":"snapshot-2018.09.26-01",
        "uuid":"Gs4xr5ErSE22sGcSlfpXAQ",
        "version_id":5020299,
        "version":"5.2.2",
        "indices":[
            "60cbd2e0f2c4410eaa7cde7f0f735487",
           .......
            "e39555dec34c4492a6a6ece0daadac19",
            "11d66120782e4583976d5bc235c29d29",
            "27cffb9f205b49b7aa48c6011c5fe1e4"
        ],
        "state":"SUCCESS",
        "start_time":"2018-09-26T03:24:42.052Z",
        "start_time_in_millis":1537932282052,
        "end_time":"2018-09-26T03:24:55.397Z",
        "end_time_in_millis":1537932295397,
        "duration_in_millis":13345,
        "failures":[

        ],
        "shards":{
            "total":135,
            "failed":0,
            "successful":135
        }
    }
}

查看備份狀態

curl -X GET "localhost:9200/_snapshot/elasticsearch_backup/snapshot-2018.09.26-01/_status?pretty"

第三步，在新服務上重複在就服務上創建倉庫的操作

創建倉庫

curl -X PUT "192.168.0.106:9200/_snapshot/my_backup?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "fs",
  "settings": {
    "location": "/usr/local/elasticsearch_backup_location"
  }
}
'

返回錯誤

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[my_backup] location [/usr/local/elasticsearch_backup_location] doesn't match any of the locations specified by path.repo because this setting is empty"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[my_backup] failed to create repository",
    "caused_by" : {
      "type" : "repository_exception",
      "reason" : "[my_backup] location [/usr/local/elasticsearch_backup_location] doesn't match any of the locations specified by path.repo because this setting is empty"
    }
  },
  "status" : 500
}

在新ES配置文件 elasticsearch.yml 上添加 path.repo 配置

path.repo: /usr/local/elasticsearch_backup_location

重啓ES服務，報錯

[2018-09-26T14:49:49,092][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.repo' (/usr/local/elasticsearch_backup_location)
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.2.2.jar:5.2.2]
Caused by: java.lang.IllegalStateException: Unable to access 'path.repo' (/usr/local/elasticsearch_backup_location)
	at org.elasticsearch.bootstrap.Security.addPath(Security.java:379) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:267) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:215) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Security.configure(Security.java:121) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:236) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.2.2.jar:5.2.2]
	... 6 more
Caused by: java.nio.file.AccessDeniedException: /usr/local/elasticsearch_backup_location
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_121]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_121]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_121]
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_121]
	at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_121]
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_121]
	at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_121]
	at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:421) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Security.addPath(Security.java:377) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:267) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:215) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Security.configure(Security.java:121) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:236) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.2.2.jar:5.2.2]
	... 6 more

創建目錄 /usr/local/elasticsearch_backup_location，修改權限

chmod -R go+w /usr/local/elasticsearch_backup_location

得到如下

drwxrwxrwx   2 root       root               6 9月  26 14:53 elasticsearch_backup_location

ES重啓成功
再次執行創建倉庫操作

curl -X PUT "192.168.0.106:9200/_snapshot/my_backup?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "fs",
  "settings": {
    "location": "/usr/local/elasticsearch_backup_location"
  }
}
'

返回成功

{
  "acknowledged" : true
}

查看創建倉庫

curl -X GET "192.168.0.106:9200/_snapshot/my_backup?pretty"

返回

{
  "my_backup" : {
    "type" : "fs",
    "settings" : {
      "location" : "/usr/local/elasticsearch_backup_location"
    }
  }
}

第四步，將舊服務上的快照備份拷貝到新服務倉庫目錄下

將舊服務快照備份文件壓縮拷貝到新服務倉庫目錄下解壓，然後可以查看快照內容

# 壓縮
tar -zcf index.src.tar.gz ./*
#解壓縮
tar -zxf index.src.tar.gz
# 查看快照
curl -X GET "192.168.0.106:9200/_snapshot/my_backup/_all?pretty"

第五步，在新服務上執行快照備份恢復

curl -X POST "192.168.0.106:9200/_snapshot/my_backup/snapshot-2018.09.26-01/_restore?pretty"

返回

{
  "accepted" : true
}

查看索引列表和數據目錄，均顯示恢復最新數據

---

參考

自動備份並壓縮腳本

#!/bin/bash
filename=`date +%Y%m%d%H`
backesFile=es$filename.tar.gz
cd /home/elasticsearch/back
mkdir es_dump
cd es_dump
curl -X DELETE "10.17.4.200:9200/_snapshot/backup/$filename?pretty"
echo 'sleep 30'
sleep 30
curl -X PUT "10.17.4.200:9200/_snapshot/backup/$filename?wait_for_completion=true&pretty"
echo 'sleep 30'
sleep 30
cp /home/elasticsearch/snapshot/* /home/elasticsearch/back/es_dump -rf
cd ..
tar czf $backesFile  es_dump/
rm es_dump -rf

自動解壓縮並恢復腳本

#!/bin/bash
filename='2018092609'
backesFile=es$filename.tar.gz
cd /home/elasticsearch/back
tar zxvf $backesFile
rm /home/elasticsearch/snapshot/* -rf
cp /home/elasticsearch/back/es_dump/* /home/elasticsearch/snapshot -rf
curl -X POST "10.17.4.200:9200/users/_close"
curl -X POST "10.17.4.200:9200/products/_close"
echo 'sleep 5'
sleep 5
curl -X POST "10.17.4.200:9200/_snapshot/backup/$filename/_restore?pretty" -d '{
    "indices":"users"
}' 
echo 'sleep 5'
sleep 5
curl -X POST "10.17.4.200:9200/_snapshot/backup/$filename/_restore?pretty" -d '{
    "indices":"products"
}'
echo 'sleep 5'
sleep 5
curl -X POST "10.17.4.200:9200/users/_open"
curl -X POST "10.17.4.200:9200/products/_open" 
rm es_dump -rf