1、說明
集羣本來有三個節點,但是異常情況導致兩個節點安裝es的磁盤丟失了,之後恢復了磁盤,然後恢復集羣,恢復集羣我是把好的es的整個目錄拷貝到剛恢復的節點上,然後修改配置文件。
好的節點的配置文件如下所示:
$ egrep -v "^#|^$" elasticsearch.yml
cluster.name: elasticsearch
node.name: "node 14.69"
bootstrap.mlockall: true
network.host: 192.168.14.69
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.timeout: 60s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["192.168.14.40","192.168.14.177","192.168.14.69"]
拷貝後修改剛恢復的兩個節點的配置文件,但是由於疏忽,忘記修改node.name這個參數,所以在集羣起來後所有節點的node.name都是node 14.69,之後又新的數據進來後,就出現了 unassigned 的分片,在head插件的頁面上就會在最上面出現一行 unassigned 的分片,集羣的狀態也變爲red。
上面的情況是怎麼出現的?下面分析下:
新建索引,默認是5個分片,1個副本,副本分片的主要目的就是爲了故障轉移,如果持有主分片的節點掛掉了,一個副本分片就會晉升爲主分片的角色。
副本分片和主分片是不能放到一個節點上面的,當副本分片沒有辦法分配到其他的節點上,所以出現所有副本分片都unassigned得情況。因爲配置失誤,所以集羣被識別爲只有一個節點。
2、解決辦法:
查看節點的狀態
$ curl -XGET http://192.168.14.69:9200/_cluster/health\?pretty
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 56,
"active_shards" : 112,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 52,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
}
可以看到集羣狀態時red,未賦值的分片數是52個。
首先設置副本數爲0
$ curl -XPUT "http://192.168.14.69:9200/_settings" -d'
{
"number_of_replicas" : 0
}'
再次查看節點的狀態
$ curl -XGET http://192.168.14.69:9200/_cluster/health\?pretty
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 56,
"active_shards" : 56,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 26,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
}
可以看到集羣未賦值的分片數是26,少了一半,說明少的是正常節點上的副本分片。
查看所有的分片信息
$ curl http://192.168.14.40:9200/_cat/shards
voicereprot 2 p UNASSIGNED
voicereprot 0 p UNASSIGNED
voicereprot 3 p STARTED 15301 5.8mb 192.168.14.40 node 14.40
voicereprot 1 p STARTED 15461 5.8mb 192.168.14.177 node 14.177
voicereprot 5 p STARTED 14540 5.3mb 192.168.14.40 node 14.40
voicereprot 4 p STARTED 16375 6.2mb 192.168.14.69 node 14.69
spipe 2 p UNASSIGNED
spipe 0 p STARTED 0 144b 192.168.14.69 node 14.69
spipe 3 p STARTED 0 144b 192.168.14.177 node 14.177
...........此處省略
查看所有的節點信息
$ curl http://192.168.14.40:9200/_nodes/process?pretty
{
"cluster_name" : "elasticsearch",
"nodes" : {
"mc3rloswRgqUJ5VkL4nxBw" : {
"name" : "node 14.177",
"transport_address" : "inet[/192.168.14.177:9300]",
"host" : "SZB-L0019761",
"ip" : "192.168.14.177",
"version" : "1.7.3",
"build" : "05d4530",
"http_address" : "inet[/192.168.14.177:9200]",
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 21695,
"max_file_descriptors" : 65536,
"mlockall" : false
}
},
"bICASdrQSe2ddNLhHw0Vyw" : {
"name" : "node 14.40",
"transport_address" : "inet[/192.168.14.40:9300]",
"host" : "DEV-L0003234",
"ip" : "192.168.14.40",
"version" : "1.7.3",
"build" : "05d4530",
"http_address" : "inet[/192.168.14.40:9200]",
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 12999,
"max_file_descriptors" : 8192,
"mlockall" : false
}
},
"kKzGBiXXTICg6f0UrT9_BA" : {
"name" : "node 14.69",
"transport_address" : "inet[/192.168.14.69:9300]",
"host" : "DEV-L0000155",
"ip" : "192.168.14.69",
"version" : "1.7.3",
"build" : "05d4530",
"http_address" : "inet[/192.168.14.69:9200]",
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 29986,
"max_file_descriptors" : 65535,
"mlockall" : false
}
}
}
}
查看節點信息,其中第一行 mc3rloswRgqUJ5VkL4nxBw、bICASdrQSe2ddNLhHw0Vyw、kKzGBiXXTICg6f0UrT9_BA 分別是節點的唯一標識。
一個一個轉移未賦值的分片
$ curl -XPOST '192.168.14.40:9200/_cluster/reroute' -d '{
"commands" : [ {
"allocate" : {
"index" : "voicereprot",
"shard" : 0,
"node" : "bICASdrQSe2ddNLhHw0Vyw",
"allow_primary" : true
}
}
]
}'
之後的只需要修改四個變量, 節點IP、index(索引名)、shard(分片標記)、node(節點唯一標識)
這裏只轉移到14.40這一個節點上
再次再次查看節點的狀態,可以發現已經轉移了一個分片
$ curl http://192.168.14.40:9200/_cluster/health\?pretty
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 57,
"active_shards" : 57,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 25,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
}
陸續轉移其他分片
$ curl -XPOST '192.168.14.40:9200/_cluster/reroute' -d '{
"commands" : [ {
"allocate" : {
"index" : "voicereprot",
"shard" : 2,
"node" : "bICASdrQSe2ddNLhHw0Vyw",
"allow_primary" : true
}
}
]
}'
...............此處省略
最後執行完畢可以發現所有的未賦值分片都已經轉移完畢
$ curl http://192.168.14.40:9200/_cat/shards
voicereprot 4 p STARTED 16375 6.2mb 192.168.14.69 node 14.69
voicereprot 0 p STARTED 0 144b 192.168.14.177 node 14.177
voicereprot 3 p STARTED 15301 5.8mb 192.168.14.40 node 14.40
voicereprot 1 p STARTED 15461 5.8mb 192.168.14.177 node 14.177
voicereprot 5 p STARTED 14540 5.3mb 192.168.14.40 node 14.40
......此處省略
$ curl http://192.168.14.40:9200/_cluster/health\?pretty
{
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 82,
"active_shards" : 82,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
}
最後恢復副本爲1
$ curl -XPUT "http://192.168.14.69:9200/_settings" -d'
{
"number_of_replicas" : 1
}'
可以把轉移分片的過程寫入到一個腳本里邊,把索引名和分片標記定義爲變量,把變量的值過濾到一個文檔中,腳本通過循環執行對對應的索引和分片標記做處理。
參考:
https://blog.csdn.net/x4609883/article/details/79926267
https://blog.csdn.net/u013905744/article/details/81508029
https://cloud.tencent.com/info/527778d6d1b6588d0c27d9622a6fa7b6.html
https://blog.csdn.net/envinfo2012/article/details/82767674