Es 會跟蹤每個活動分片的索引,分片在五分鐘內如果沒有收到任何有關索引的操作,則認爲分片不活動了;這將給Es機會去減小分片的資源,同時執行一種特殊刷新,成爲:同步刷新;同步刷新也是執行普通的刷新,然後給所有的分片添加一個不同的標記;
當沒有索引操作時,id 標記會添加到分片上。標記可以作爲一個快速的方式來檢查兩個分片的lucene索引一致是否一致;這種快速的id 比較主要用於 數據恢復或者重啓後跳過第一個也是成本最高的階段;這種情況下,segment 不需要copy,事務日誌的重演會立即執行;當id 標記和flush 一起用的時候,事物日誌很可能爲空,更加加速了數據的恢復;
這對於具有很多很少更新或者不變的索引的情況(如基於時間的數據),將會更有用;查看分片是否被id 標記,用如下命令:
curl -X GET "dmp9:9201/wzp_test/_stats/commit?level=shards"
{"_shards":{"total":10,"successful":10,"failed":0},"_all":{"primaries":{},"total":{}},"indices":{"wzp_test":{"primaries":{},"total":{},"shards":{"0":[{"routing":{"state":"STARTED","primary":true,"node":"8hkDNjDQRrGMWmhUFbhPrA","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAQg==","generation":12,"user_data":{"translog_uuid":"OHqJo56NQoqD0vivoRE_-w","sync_id":"AWdPVMGxhBFTwa1Fywjs","translog_generation":"1543159976461"},"num_docs":411901},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}},{"routing":{"state":"STARTED","primary":false,"node":"aagBamD6QZWOm-GF5QPNdg","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAQg==","generation":12,"user_data":{"translog_uuid":"OHqJo56NQoqD0vivoRE_-w","sync_id":"AWdPVMGxhBFTwa1Fywjs","translog_generation":"1543159976461"},"num_docs":411901},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}}],"1":[{"routing":{"state":"STARTED","primary":true,"node":"8hkDNjDQRrGMWmhUFbhPrA","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAQQ==","generation":13,"user_data":
{"translog_uuid":"r5RMt3ZDS5ip1Il2cldOKA","sync_id":"AWdPVMGxhBFTwa1Fywjr","translog_generation":"1543160040431"},"num_docs":383147},"shard_path":
{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}},{"routing":{"state":"STARTED","primary":false,"node":"aagBamD6QZWOm-GF5QPNdg","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAQQ==","generation":13,"user_data":{"translog_uuid":"r5RMt3ZDS5ip1Il2cldOKA","sync_id":"AWdPVMGxhBFTwa1Fywjr","translog_generation":"1543160040431"},"num_docs":383147},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}}],"2":[{"routing":{"state":"STARTED","primary":true,"node":"8hkDNjDQRrGMWmhUFbhPrA","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAPw==","generation":13,"user_data":{"translog_uuid":"EYkaGyrTTPCtgKArFf0Qyw","sync_id":"AWdPVMGxhBFTwa1Fywjp","translog_generation":"1543159976463"},"num_docs":383110},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}},{"routing":{"state":"STARTED","primary":false,"node":"aagBamD6QZWOm-GF5QPNdg","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAPw==","generation":13,"user_data":{"translog_uuid":"EYkaGyrTTPCtgKArFf0Qyw","sync_id":"AWdPVMGxhBFTwa1Fywjp","translog_generation":"1543159976463"},"num_docs":383110},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}}],"3":[{"routing":{"state":"STARTED","primary":true,"node":"8hkDNjDQRrGMWmhUFbhPrA","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAQw==","generation":13,"user_data":{"translog_uuid":"livjpLxlRSakLFnCp8n9Jw","sync_id":"AWdPVMGxhBFTwa1Fywjo","translog_generation":"1543160040432"},"num_docs":410859},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}},{"routing":{"state":"STARTED","primary":false,"node":"aagBamD6QZWOm-GF5QPNdg","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAQw==","generation":13,"user_data":{"translog_uuid":"livjpLxlRSakLFnCp8n9Jw","sync_id":"AWdPVMGxhBFTwa1Fywjo","translog_generation":"1543160040432"},"num_docs":410859},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}}],"4":[{"routing":{"state":"STARTED","primary":true,"node":"8hkDNjDQRrGMWmhUFbhPrA","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAQA==","generation":13,"user_data":{"translog_uuid":"US1IS1LaR5qkwlJwLLs7Vw","sync_id":"AWdPVMGxhBFTwa1Fywjq","translog_generation":"1543159976461"},"num_docs":416518},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}},{"routing":{"state":"STARTED","primary":false,"node":"aagBamD6QZWOm-GF5QPNdg","relocating_node":null},"commit":{"id":"uSW9jUn1xqqvWjl0/SiAQA==","generation":13,"user_data":{"translog_uuid":"US1IS1LaR5qkwlJwLLs7Vw","sync_id":"AWdPVMGxhBFTwa1Fywjq","translog_generation":"1543159976461"},"num_docs":416518},"shard_path":{"state_path":"/data2/es_test/dmp_test/nodes/0","data_path":"/data2/es_test/dmp_test/nodes/0","is_custom_data_path":false}}]}}}}
官方文檔給的結果示例:
curl -X GET "localhost:9200/twitter/_stats/commit?level=shards"
{
...
"indices": {
"twitter": {
"primaries": {},
"total": {},
"shards": {
"0": [
{
"routing": {
...
},
"commit": {
"id": "te7zF7C4UsirqvL6jp/vUg==",
"generation": 2,
"user_data": {
"sync_id": "AU2VU0meX-VX2aNbEUsD" ,
...
},
"num_docs": 0
}
}
...
],
...
}
}
}
}
同步刷新 API
同步刷新API 允許管理員手動刷新。這對於計劃的(回滾)集羣重啓特別有用,您可以停止索引,並且不希望等待默認的5分鐘來自動同步刷新空閒索引;
儘管很方便,但是有一些警告:
-
同步刷新是個盡最大努力的操作:任何正在執行索引的操作都會造成同步刷新失敗;這就一位置有些分片可能刷新成功,而有些分片則刷新失敗;
-
同步id 的標記在再次刷新是會移除掉;
這是因爲刷新會替換低級別的儲存標記的lucene 提交點;事物日誌中未提交的操作不會移除id 標記;現實中,在任何時間,應該把索引操作視觸發的移除標記爲刷新一樣;
note:有索引操作時,請求同步刷新是無害的;當索引空閒時同步刷新成功,否則失敗;任何請求成功的同步刷新在數據恢復時更加迅速;
curl -X POST "localhost:9200/twitter/_flush/synced"返回值將顯示有多少分片同步刷新成功,及失敗原因;
1)、以下是同步刷新成功的情況;
{
"_shards": {
"total": 4,
"successful": 4,
"failed": 0
},
"twitter": {
"total": 4,
"successful": 4,
"failed": 0
}
}
2)、以下是失敗情況及其原因;
{
"_shards": {
"total": 4,
"successful": 2,
"failed": 2
},
"twitter": {
"total": 4,
"successful": 2,
"failed": 2,
"failures": [
{
"shard": 1,
"reason": "[2] ongoing operations on primary"
}
]
}
}
note:當並行索引操作時,以上錯誤會出現,且http 狀態碼將是 409 ;
3)、有時候分片copy的時候錯誤很特殊;copy失敗對於數據的快速恢復沒有用,但是如果成功了依然有用;具體錯誤如下:
{
"_shards": {
"total": 4,
"successful": 1,
"failed": 1
},
"twitter": {
"total": 4,
"successful": 3,
"failed": 1,
"failures": [
{
"shard": 1,
"reason": "unexpected error",
"routing": {
"state": "STARTED",
"primary": false,
"node": "SZNr2J_ORxKTLUCydGX4zA",
"relocating_node": null,
"shard": 1,
"index": "twitter"
}
}
]
}
}
同步刷新時copy失敗,http 返回碼將是409 ;
4)、同步刷新可以再單一索引上用,也可以在全部索引上用;
單一索引:
POST /kimchy,elasticsearch/_flush/synced
全部索引:
POST /_flush/synced