Elasticsearch[7.6]移除type遷移/改造方案

google譯自：https://www.elastic.co/guide/en/elasticsearch/reference/7.6/removal-of-types.html#removal-of-types

方案一：`include_type_name參數設爲true`

設置爲true後即可用原有方式put。

對於我們的用戶來說，這是一個很大的變化，因此我們試圖使其儘可能輕鬆。更改將按以下方式推出：

Elasticsearch 5.6.0

index.mapping.single_type: true在索引上設置將啓用按索引的單一類型行爲，該行爲將在6.0中強制執行。
在5.6中創建的索引上可以使用父代的join字段替換。

Elasticsearch 6.x

在5.x中創建的索引將像在5.x中一樣繼續在6.x中運行。
在6.x中創建的索引僅允許每個索引使用一種類型。該類型可以使用任何名稱，但是只能有一個。首選的類型名稱是_doc，因此索引API具有與7.0中相同的路徑： PUT {index}/_doc/{id}和POST {index}/_doc
該_type名稱不能再與組合_id以形成_uid 字段。該_uid字段已成爲該_id字段的別名。
新索引不再支持父/子的舊樣式，而應使用該join字段。
的_default_映射類型已棄用。
在6.8中，索引創建，索引模板和映射API支持查詢字符串參數（include_type_name），該參數指示請求和響應是否應包含類型名稱。它默認爲true，並且應設置爲明確的值以準備升級到7.0。不設置include_type_name 將導致棄用警告。沒有顯式類型的索引將使用虛擬類型名稱_doc。

Elasticsearch 7.x

不建議在請求中指定類型。例如，爲文檔建立索引不再需要document type。新索引API適用PUT {index}/_doc/{id}於顯式ID和POST {index}/_doc 自動生成的ID。請注意，在7.0中，它_doc是路徑的永久部分，並且表示端點名稱而不是文檔類型。
將include_type_name在創建索引，索引模板，地圖API參數默認爲false。完全設置該參數將導致棄用警告。
的_default_映射類型被去除。

Elasticsearch 8.x

不再支持在請求中指定類型。
該include_type_name參數已刪除。

方案二：自定義類型字段

當然，一個集羣中可以存在多少個主要分片是有限制的，因此您可能不想只花幾千個文檔就浪費整個分片。在這種情況下，您可以實現自己的自定義type 字段，該字段的工作方式與舊的類似_type。

You can achieve the same thing by adding a custom type field as follows:

假如索引twitter下，原有user和tweet兩個type。可以改成twitter里加一個屬性flag，flag有兩個值，用於搜索時區分user和tweet。

PUT twitter
{
  "mappings": {
    "_doc": {
      "properties": {
        "type": { "type": "keyword" }, //type 即上文提到的flag屬性。
        "name": { "type": "text" },
        "user_name": { "type": "keyword" },
        "email": { "type": "keyword" },
        "content": { "type": "text" },
        "tweeted_at": { "type": "date" }
      }
    }
  }
}

PUT twitter/_doc/user-kimchy
{
  "type": "user", // user類型
  "name": "Shay Banon",
  "user_name": "kimchy",
  "email": "[email protected]"
}

PUT twitter/_doc/tweet-1
{
  "type": "tweet", // tweet類型
  "user_name": "kimchy",
  "tweeted_at": "2017-10-24T09:00:00Z",
  "content": "Types are going away"
}

GET twitter/_search
{
  "query": {
    "bool": {
      "must": {
        "match": {
          "user_name": "kimchy"
        }
      },
      "filter": {
        "match": {
          "type": "tweet" // 過濾
        }
      }
    }
  }
}

以下爲原文：

Removal of mapping types

Indices created in Elasticsearch 7.0.0 or later no longer accept a _default_ mapping. Indices created in 6.x will continue to function as before in Elasticsearch 6.x. Types are deprecated in APIs in 7.0, with breaking changes to the index creation, put mapping, get mapping, put template, get template and get field mappings APIs.

What are mapping types?

Since the first release of Elasticsearch, each document has been stored in a single index and assigned a single mapping type. A mapping type was used to represent the type of document or entity being indexed, for instance a twitter index might have a user type and a tweet type.

Each mapping type could have its own fields, so the user type might have a full_name field, a user_name field, and an email field, while the tweet type could have a content field, a tweeted_at field and, like the user type, a user_name field.

Each document had a _type meta-field containing the type name, and searches could be limited to one or more types by specifying the type name(s) in the URL:

GET twitter/user,tweet/_search
{
  "query": {
    "match": {
      "user_name": "kimchy"
    }
  }
}

The _type field was combined with the document’s _id to generate a _uid field, so documents of different types with the same _id could exist in a single index.

Mapping types were also used to establish a parent-child relationship between documents, so documents of type question could be parents to documents of type answer.

Why are mapping types being removed?edit

Initially, we spoke about an “index” being similar to a “database” in an SQL database, and a “type” being equivalent to a “table”.

This was a bad analogy that led to incorrect assumptions. In an SQL database, tables are independent of each other. The columns in one table have no bearing on columns with the same name in another table. This is not the case for fields in a mapping type.

In an Elasticsearch index, fields that have the same name in different mapping types are backed by the same Lucene field internally. In other words, using the example above, the user_name field in the user type is stored in exactly the same field as the user_name field in the tweet type, and both user_name fields must have the same mapping (definition) in both types.

This can lead to frustration when, for example, you want deleted to be a date field in one type and a boolean field in another type in the same index.

On top of that, storing different entities that have few or no fields in common in the same index leads to sparse data and interferes with Lucene’s ability to compress documents efficiently.

For these reasons, we have decided to remove the concept of mapping types from Elasticsearch.

Alternatives to mapping typesedit

Index per document typeedit

The first alternative is to have an index per document type. Instead of storing tweets and users in a single twitter index, you could store tweets in the tweets index and users in the user index. Indices are completely independent of each other and so there will be no conflict of field types between indices.

This approach has two benefits:

Data is more likely to be dense and so benefit from compression techniques used in Lucene.
The term statistics used for scoring in full text search are more likely to be accurate because all documents in the same index represent a single entity.

Each index can be sized appropriately for the number of documents it will contain: you can use a smaller number of primary shards for users and a larger number of primary shards for tweets.

Custom type fieldedit

Of course, there is a limit to how many primary shards can exist in a cluster so you may not want to waste an entire shard for a collection of only a few thousand documents. In this case, you can implement your own custom type field which will work in a similar way to the old _type.

Let’s take the user/tweet example above. Originally, the workflow would have looked something like this:

PUT twitter
{
  "mappings": {
    "user": {
      "properties": {
        "name": { "type": "text" },
        "user_name": { "type": "keyword" },
        "email": { "type": "keyword" }
      }
    },
    "tweet": {
      "properties": {
        "content": { "type": "text" },
        "user_name": { "type": "keyword" },
        "tweeted_at": { "type": "date" }
      }
    }
  }
}

PUT twitter/user/kimchy
{
  "name": "Shay Banon",
  "user_name": "kimchy",
  "email": "[email protected]"
}

PUT twitter/tweet/1
{
  "user_name": "kimchy",
  "tweeted_at": "2017-10-24T09:00:00Z",
  "content": "Types are going away"
}

GET twitter/tweet/_search
{
  "query": {
    "match": {
      "user_name": "kimchy"
    }
  }
}

You can achieve the same thing by adding a custom type field as follows:

PUT twitter
{
  "mappings": {
    "_doc": {
      "properties": {
        "type": { "type": "keyword" }, 
        "name": { "type": "text" },
        "user_name": { "type": "keyword" },
        "email": { "type": "keyword" },
        "content": { "type": "text" },
        "tweeted_at": { "type": "date" }
      }
    }
  }
}

PUT twitter/_doc/user-kimchy
{
  "type": "user", 
  "name": "Shay Banon",
  "user_name": "kimchy",
  "email": "[email protected]"
}

PUT twitter/_doc/tweet-1
{
  "type": "tweet", 
  "user_name": "kimchy",
  "tweeted_at": "2017-10-24T09:00:00Z",
  "content": "Types are going away"
}

GET twitter/_search
{
  "query": {
    "bool": {
      "must": {
        "match": {
          "user_name": "kimchy"
        }
      },
      "filter": {
        "match": {
          "type": "tweet" 
        }
      }
    }
  }
}

The explicit type field takes the place of the implicit _type field.

Parent/Child without mapping typesedit

Previously, a parent-child relationship was represented by making one mapping type the parent, and one or more other mapping types the children. Without types, we can no longer use this syntax. The parent-child feature will continue to function as before, except that the way of expressing the relationship between documents has been changed to use the new join field.

Schedule for removal of mapping typesedit

This is a big change for our users, so we have tried to make it as painless as possible. The change will roll out as follows:

Elasticsearch 5.6.0

Setting index.mapping.single_type: true on an index will enable the single-type-per-index behaviour which will be enforced in 6.0.
The join field replacement for parent-child is available on indices created in 5.6.

Elasticsearch 6.x

Indices created in 5.x will continue to function in 6.x as they did in 5.x.
Indices created in 6.x only allow a single-type per index. Any name can be used for the type, but there can be only one. The preferred type name is _doc, so that index APIs have the same path as they will have in 7.0: PUT {index}/_doc/{id} and POST {index}/_doc
The _type name can no longer be combined with the _id to form the _uid field. The _uid field has become an alias for the _id field.
New indices no longer support the old-style of parent/child and should use the join field instead.
The _default_ mapping type is deprecated.
In 6.8, the index creation, index template, and mapping APIs support a query string parameter (include_type_name) which indicates whether requests and responses should include a type name. It defaults to true, and should be set to an explicit value to prepare to upgrade to 7.0. Not setting include_type_name will result in a deprecation warning. Indices which don’t have an explicit type will use the dummy type name _doc.

Elasticsearch 7.x

Specifying types in requests is deprecated. For instance, indexing a document no longer requires a document type. The new index APIs are PUT {index}/_doc/{id} in case of explicit ids and POST {index}/_doc for auto-generated ids. Note that in 7.0, _doc is a permanent part of the path, and represents the endpoint name rather than the document type.
The include_type_name parameter in the index creation, index template, and mapping APIs will default to false. Setting the parameter at all will result in a deprecation warning.
The _default_ mapping type is removed.

Elasticsearch 8.x

Specifying types in requests is no longer supported.
The include_type_name parameter is removed.

Migrating multi-type indices to single-typeedit

The Reindex API can be used to convert multi-type indices to single-type indices. The following examples can be used in Elasticsearch 5.6 or Elasticsearch 6.x. In 6.x, there is no need to specify index.mapping.single_type as that is the default.

Index per document typeedit

This first example splits our twitter index into a tweets index and a users index:

PUT users
{
  "settings": {
    "index.mapping.single_type": true
  },
  "mappings": {
    "_doc": {
      "properties": {
        "name": {
          "type": "text"
        },
        "user_name": {
          "type": "keyword"
        },
        "email": {
          "type": "keyword"
        }
      }
    }
  }
}

PUT tweets
{
  "settings": {
    "index.mapping.single_type": true
  },
  "mappings": {
    "_doc": {
      "properties": {
        "content": {
          "type": "text"
        },
        "user_name": {
          "type": "keyword"
        },
        "tweeted_at": {
          "type": "date"
        }
      }
    }
  }
}

POST _reindex
{
  "source": {
    "index": "twitter",
    "type": "user"
  },
  "dest": {
    "index": "users",
    "type": "_doc"
  }
}

POST _reindex
{
  "source": {
    "index": "twitter",
    "type": "tweet"
  },
  "dest": {
    "index": "tweets",
    "type": "_doc"
  }
}

Custom type fieldedit

This next example adds a custom type field and sets it to the value of the original _type. It also adds the type to the _id in case there are any documents of different types which have conflicting IDs:

PUT new_twitter
{
  "mappings": {
    "_doc": {
      "properties": {
        "type": {
          "type": "keyword"
        },
        "name": {
          "type": "text"
        },
        "user_name": {
          "type": "keyword"
        },
        "email": {
          "type": "keyword"
        },
        "content": {
          "type": "text"
        },
        "tweeted_at": {
          "type": "date"
        }
      }
    }
  }
}


POST _reindex
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter"
  },
  "script": {
    "source": """
      ctx._source.type = ctx._type;
      ctx._id = ctx._type + '-' + ctx._id;
      ctx._type = '_doc';
    """
  }
}

Typeless APIs in 7.0edit

In Elasticsearch 7.0, each API will support typeless requests, and specifying a type will produce a deprecation warning.

Typeless APIs work even if the target index contains a custom type. For example, if an index has the custom type name my_type, we can add documents to it using typeless index calls, and load documents with typeless get calls.

Index APIsedit

Index creation, index template, and mapping APIs support a new include_type_name URL parameter that specifies whether mapping definitions in requests and responses should contain the type name. The parameter defaults to true in version 6.8 to match the pre-7.0 behavior of using type names in mappings. It defaults to false in version 7.0 and will be removed in version 8.0.

It should be set explicitly in 6.8 to prepare to upgrade to 7.0. To avoid deprecation warnings in 6.8, the parameter can be set to either true or false. In 7.0, setting include_type_name at all will result in a deprecation warning.

See some examples of interactions with Elasticsearch with this option set to false:

PUT index?include_type_name=false
{
  "mappings": {
    "properties": { 
      "foo": {
        "type": "keyword"
      }
    }
  }
}

Copy as cURL View in Console

Mappings are included directly under the mappings key, without a type name.

PUT index/_mappings?include_type_name=false
{
  "properties": { 
    "bar": {
      "type": "text"
    }
  }
}

Copy as cURL View in Console

Mappings are included directly under the mappings key, without a type name.

GET index/_mappings?include_type_name=false

Copy as cURL View in Console

The above call returns

{
  "index": {
    "mappings": {
      "properties": { 
        "foo": {
          "type": "keyword"
        },
        "bar": {
          "type": "text"
        }
      }
    }
  }
}

Mappings are included directly under the mappings key, without a type name.

Document APIsedit

In 7.0, index APIs must be called with the {index}/_doc path for automatic generation of the _id and {index}/_doc/{id} with explicit ids.

PUT index/_doc/1
{
  "foo": "baz"
}

Copy as cURL View in Console

{
  "_index": "index",
  "_id": "1",
  "_type": "_doc",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 1
}

Similarly, the get and delete APIs use the path {index}/_doc/{id}:

GET index/_doc/1

Copy as cURL View in Console

In 7.0, _doc represents the endpoint name instead of the document type. The _doc component is a permanent part of the path for the document index, get, and delete APIs going forward, and will not be removed in 8.0.

For API paths that contain both a type and endpoint name like _update, in 7.0 the endpoint will immediately follow the index name:

POST index/_update/1
{
    "doc" : {
        "foo" : "qux"
    }
}

GET /index/_source/1

Copy as cURL View in Console

Types should also no longer appear in the body of requests. The following example of bulk indexing omits the type both in the URL, and in the individual bulk commands:

POST _bulk
{ "index" : { "_index" : "index", "_id" : "3" } }
{ "foo" : "baz" }
{ "index" : { "_index" : "index", "_id" : "4" } }
{ "foo" : "qux" }

Copy as cURL View in Console

Search APIsedit

When calling a search API such _search, _msearch, or _explain, types should not be included in the URL. Additionally, the _type field should not be used in queries, aggregations, or scripts.

Types in responsesedit

The document and search APIs will continue to return a _type key in responses, to avoid breaks to response parsing. However, the key is considered deprecated and should no longer be referenced. Types will be completely removed from responses in 8.0.

Note that when a deprecated typed API is used, the index’s mapping type will be returned as normal, but that typeless APIs will return the dummy type _doc in the response. For example, the following typeless get call will always return _doc as the type, even if the mapping has a custom type name like my_type:

PUT index/my_type/1
{
  "foo": "baz"
}

GET index/_doc/1

Copy as cURL View in Console

{
    "_index" : "index",
    "_type" : "_doc",
    "_id" : "1",
    "_version" : 1,
    "_seq_no" : 0,
    "_primary_term" : 1,
    "found": true,
    "_source" : {
        "foo" : "baz"
    }
}

Index templatesedit

It is recommended to make index templates typeless by re-adding them with include_type_name set to false. Under the hood, typeless templates will use the dummy type _doc when creating indices.

In case typeless templates are used with typed index creation calls or typed templates are used with typeless index creation calls, the template will still be applied but the index creation call decides whether there should be a type or not. For instance in the below example, index-1-01 will have a type in spite of the fact that it matches a template that is typeless, and index-2-01 will be typeless in spite of the fact that it matches a template that defines a type. Both index-1-01 and index-2-01 will inherit the foo field from the template that they match.

PUT _template/template1
{
  "index_patterns":[ "index-1-*" ],
  "mappings": {
    "properties": {
      "foo": {
        "type": "keyword"
      }
    }
  }
}

PUT _template/template2?include_type_name=true
{
  "index_patterns":[ "index-2-*" ],
  "mappings": {
    "type": {
      "properties": {
        "foo": {
          "type": "keyword"
        }
      }
    }
  }
}

PUT index-1-01?include_type_name=true
{
  "mappings": {
    "type": {
      "properties": {
        "bar": {
          "type": "long"
        }
      }
    }
  }
}

PUT index-2-01
{
  "mappings": {
    "properties": {
      "bar": {
        "type": "long"
      }
    }
  }
}

Copy as cURL View in Console

In case of implicit index creation, because of documents that get indexed in an index that doesn’t exist yet, the template is always honored. This is usually not a problem due to the fact that typeless index calls work on typed indices.

Mixed-version clustersedit

In a cluster composed of both 6.8 and 7.0 nodes, the parameter include_type_name should be specified in index APIs like index creation. This is because the parameter has a different default between 6.8 and 7.0, so the same mapping definition will not be valid for both node versions.

Typeless document APIs such as bulk and update are only available as of 7.0, and will not work with 6.8 nodes. This also holds true for the typeless versions of queries that perform document lookups, such as terms.

Elasticsearch[7.6]移除type遷移/改造方案

方案一：`include_type_name參數設爲true`

方案二：自定義類型字段

以下爲原文：

What are mapping types?

Why are mapping types being removed?edit

Alternatives to mapping typesedit

Schedule for removal of mapping typesedit

Migrating multi-type indices to single-typeedit

Typeless APIs in 7.0edit

Apache Flink 1.9.0做了這些修改

springboot2|springboot啓動流程源碼分析:tomcat啓動原理

消息隊列rocketmq：Topic 與 Tag 最佳實踐

目前主流的 Java 虛擬機有哪些?

螞蟻金服 Service Mesh 實踐探索 | Qcon 實錄

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結

Elasticsearch[7.6]移除type遷移/改造方案

方案一：include_type_name參數設爲true

方案二：自定義類型字段

以下爲原文：

What are mapping types?

Why are mapping types being removed?edit

Alternatives to mapping typesedit

Schedule for removal of mapping typesedit

Migrating multi-type indices to single-typeedit

Typeless APIs in 7.0edit

方案一：`include_type_name參數設爲true`