案發現場
測試同學壓測 接口,導致es瘋狂超時
排查
登錄es服務器,打開日誌,發現全部是超時日誌,再往上看發現全是debug級別的日誌,而且連每條query語句都打印了出來。
....
....
...
"max_expansions" : 50,
"fuzzy_transpositions" : true,
"lenient" : false,
"zero_terms_query" : "NONE",
"boost" : 0.1
}
}
}
],
"disable_coord" : false,
"adjust_pure_negative" : true,
"boost" : 1.0
}
},
"_source" : {
"includes" : [
"aId",
"an"
],
"excludes" : [ ]
},
"sort" : [
{
"_score" : {
"order" : "desc"
}
},
{
"dn" : {
"order" : "desc"
}
}
]
:
一次請求1000個
導致了拒絕接受新的請求
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.transport.TransportService$7@1ad069ff on EsThr
eadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@41328b39[Running, pool size = 13, active threads
= 13, queued tasks = 1000, completed tasks = 48895]]
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:50) ~[elasticsearch-5.4.1.jar:5.4.1]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) ~[?:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) ~[?:1.8.0_45]
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:94) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:89) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:623) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.transport.TransportService.access$000(TransportService.java:73) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:133) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:569) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.transport.TransportService$$Lambda$1050/1614238.sendRequest(Unknown Source) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:502) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:529) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:520) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:146) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:74) [elasticsearch-5.4.1.
jar:5.4.1]
查看當前線程池配置:
curl -XGET 'http://10.20.30.40:9200/_nodes/thread_pool?pretty'
從ElasticSearch5.0 開始,無法通過api更改線程池,需要更改elasticsearch.yml並重啓才能生效配置
thread_pool.search.queue_size: 500
#queue_size允許控制沒有線程執行它們的掛起請求隊列的初始大小。
thread_pool.search.size: 200
#size參數控制線程數,默認爲核心數乘以5。
thread_pool.search.min_queue_size:10
#min_queue_size設置控制queue_size可以調整到的最小量。
thread_pool.search.max_queue_size: 1000
#max_queue_size設置控制queue_size可以調整到的最大量。
thread_pool.search.auto_queue_frame_size: 2000
#auto_queue_frame_size設置控制在調整隊列之前進行測量的操作數。它應該足夠大,以便單個操作不會過度偏向計算。
thread_pool.search.target_response_time: 6s
#target_response_time是時間值設置,指示線程池隊列中任務的目標平均響應時間。如果任務通常超過此時間,則將調低線程池隊列以拒絕任務。
修改爲warn級別的
Elasticsearch 會輸出很多日誌,都放在 ES_HOME/logs 目錄下。默認的日誌記錄等級是 INFO 。它提供了適度的信息,但是又設計好了不至於讓你的日誌太過龐大。
當調試問題的時候,特別是節點發現相關的問題(因爲這個經常依賴於各式過於繁瑣的網絡配置),提高日誌記錄等級到 DEBUG 是很有幫助的。
你 可以 修改log4j2.properties 文件然後重啓你的節點——但是這樣做即繁瑣還會導致不必要的宕機時間。作爲替代,你可以通過 cluster-settings API 更新日誌記錄級別,就像我們前面剛學過的那樣。
要實現這個更新,選擇你感興趣的日誌器,然後在前面補上 logger. 。對根日誌器你可以用 logger._root 來表示。
PUT /_cluster/settings
{
"transient": {
"<name of logging hierarchy>": "<level>"
}
}
例如:
curl -XPUT http://10.20.30.40:9200/_cluster/settings -d'
{
"transient" : {
"logger.discovery ": "WARN"
}
}'
設置生效,Elasticsearch 將開始輸出 discovery 模塊的 WARN級別的日誌。
{"acknowledged":true,"persistent":{},"transient":{"logger":{"org":{"elasticsearch":{"indices":{"recovery":"WARN"}}}}}}
參考
https://www.elastic.co/guide/en/elasticsearch/reference/5.1/breaking_50_settings_changes.html#_forbid_changing_of_thread_pool_types
http://doc.codingdict.com/elasticsearch/437/
https://elasticsearch.cn/article/32