elasticsearch 生命週期 冷熱數據分離

官方文檔地址：

https://www.elastic.co/guide/en/elasticsearch/reference/7.4/index-lifecycle-management.html

需求：
1.5天后，將舊索引移至暖機階段，將其標記爲只讀，然後將其縮小爲單個碎片。
2.30天后，將索引移至冷態，然後將其移至較便宜的硬件上。
3.達到所需的90天保留期後，刪除索引。

實施

1.爲所有data節點打tag

node.attr.box_type: hot
node.attr.box_type: warm
node.attr.box_type: cold

2.滾動重啓data節點

1）關閉數據均衡：

curl -XPUT /_cluster/settings
{
  "transient" : {
    "cluster.routing.allocation.enable " : "none"
  }
}

2）刷新緩存：

curl -XPOST /_flush/synced

3）重啓節點

4）打開數據均衡

curl -XPUT /_cluster/settings
{
  "transient" : {
    "cluster.routing.allocation.enable " : "all"
  }
}

5）等待集羣變green

6. 滾動重啓其他節點

3.設置index生命週期

1）定義生命週期策略

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 97
          }
        }
      },
      "warm": {
        "min_age": "5d",
        "actions": {
          "allocate": {
            "include": {},
            "exclude": {},
            "require": {
              "box_type": "warm"
            }
          },
          "forcemerge": {
            "max_num_segments": 1
          },
          "set_priority": {
            "priority": 50
          }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "include": {},
            "exclude": {},
            "require": {
              "box_type": "cold"
            }
          },
          "set_priority": {
            "priority": 1
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

注：
1）一旦complete了，這個policy就相當於執行完了，即該階段不存在了，如果後續觸發相反操作，也不會觸發該策略，如將cold手動移回hot，不會自動移回
2）segment合併後，數據減少量約在1G/200G左右，segment merge後會提高搜索速度

3）priority優先級：一旦節點發生崩潰，優先級高的，最先被恢復

4）hot：只能配置rollover和優先級策略，配置其他action會報錯

2.將生命週期策略，應用到所有已存在index

PUT _all/_settings
{
  "index.routing.allocation.require.box_type": "hot",
  "index.lifecycle.name": "my_policy"
}
注：默認所有index放在hot狀態

3.修改template

PUT _template/datastream_template
{
  "index_patterns": ["datastream-*"],                 
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 2,
    "index.lifecycle.name": "my_policy",      
    "index.routing.allocation.require.box_type": "hot"   
  }
}

4.查看目前index狀態

GET datastream-20191005/_ilm/explain

5.其他命令

強制段合併：
POST /datastream-20191005/_forcemerge?only_expunge_deletes=false&max_num_segments=100&flush=true

查看index的segment狀態:GET /_cat/segments?datastream-20191005

查看節點的tag狀態：GET /_cat/nodeattrs

配置exclude_ip：
PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.exclude._ip": "xx.xx.x.xx,xx.xx.x.xx"
  }
}

取消exclude_ip配置：
PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.exclude._ip": null
  }
}

手動觸發遷移：
PUT datastream-20191005/_settings
{
  "index": {
    "routing": {
      "allocation": {
        "require": {
          "box_type": "cold"
        }
      }
    }
  }
}

查看節點設置：
GET datastream-20191005/_settings

查看policy設置：
GET _ilm/policy/my_policy

6.其他node tag方式

節點tag
node.zone: hot
node.zone: warm
node.zone: cold

上述box_type均換爲zone
PUT /datastream-20191005/_settings
{
  "index.routing.allocation.include.zone": "cold"
}

注：index.routing.allocation配置可以配置include/require/exclude,但最好不要都配，容易造成不識別