elasticsearch 生命周期 冷热数据分离

官方文档地址：

https://www.elastic.co/guide/en/elasticsearch/reference/7.4/index-lifecycle-management.html

需求：
1.5天后，将旧索引移至暖机阶段，将其标记为只读，然后将其缩小为单个碎片。
2.30天后，将索引移至冷态，然后将其移至较便宜的硬件上。
3.达到所需的90天保留期后，删除索引。

实施

1.为所有data节点打tag

node.attr.box_type: hot
node.attr.box_type: warm
node.attr.box_type: cold

2.滚动重启data节点

1）关闭数据均衡：

curl -XPUT /_cluster/settings
{
  "transient" : {
    "cluster.routing.allocation.enable " : "none"
  }
}

2）刷新缓存：

curl -XPOST /_flush/synced

3）重启节点

4）打开数据均衡

curl -XPUT /_cluster/settings
{
  "transient" : {
    "cluster.routing.allocation.enable " : "all"
  }
}

5）等待集群变green

6. 滚动重启其他节点

3.设置index生命周期

1）定义生命周期策略

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 97
          }
        }
      },
      "warm": {
        "min_age": "5d",
        "actions": {
          "allocate": {
            "include": {},
            "exclude": {},
            "require": {
              "box_type": "warm"
            }
          },
          "forcemerge": {
            "max_num_segments": 1
          },
          "set_priority": {
            "priority": 50
          }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "include": {},
            "exclude": {},
            "require": {
              "box_type": "cold"
            }
          },
          "set_priority": {
            "priority": 1
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

注：
1）一旦complete了，这个policy就相当于执行完了，即该阶段不存在了，如果后续触发相反操作，也不会触发该策略，如将cold手动移回hot，不会自动移回
2）segment合并后，数据减少量约在1G/200G左右，segment merge后会提高搜索速度

3）priority优先级：一旦节点发生崩溃，优先级高的，最先被恢复

4）hot：只能配置rollover和优先级策略，配置其他action会报错

2.将生命周期策略，应用到所有已存在index

PUT _all/_settings
{
  "index.routing.allocation.require.box_type": "hot",
  "index.lifecycle.name": "my_policy"
}
注：默认所有index放在hot状态

3.修改template

PUT _template/datastream_template
{
  "index_patterns": ["datastream-*"],                 
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 2,
    "index.lifecycle.name": "my_policy",      
    "index.routing.allocation.require.box_type": "hot"   
  }
}

4.查看目前index状态

GET datastream-20191005/_ilm/explain

5.其他命令

强制段合并：
POST /datastream-20191005/_forcemerge?only_expunge_deletes=false&max_num_segments=100&flush=true

查看index的segment状态:GET /_cat/segments?datastream-20191005

查看节点的tag状态：GET /_cat/nodeattrs

配置exclude_ip：
PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.exclude._ip": "xx.xx.x.xx,xx.xx.x.xx"
  }
}

取消exclude_ip配置：
PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.exclude._ip": null
  }
}

手动触发迁移：
PUT datastream-20191005/_settings
{
  "index": {
    "routing": {
      "allocation": {
        "require": {
          "box_type": "cold"
        }
      }
    }
  }
}

查看节点设置：
GET datastream-20191005/_settings

查看policy设置：
GET _ilm/policy/my_policy

6.其他node tag方式

节点tag
node.zone: hot
node.zone: warm
node.zone: cold

上述box_type均换为zone
PUT /datastream-20191005/_settings
{
  "index.routing.allocation.include.zone": "cold"
}

注：index.routing.allocation配置可以配置include/require/exclude,但最好不要都配，容易造成不识别