ELK學習筆記之No shard available報錯

0x00 概述

研發反饋skywalking突然無法使用,查看日誌發現

2020-10-19 14:25:32,712 - org.apache.skywalking.apm.collector.cache.caffeine.service.ServiceNameCacheCaffeineService -82494825 [grpc-default-executor-149] ERROR [] - No shard available for [get [service_name][type][-87]: routing [null]]
org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [service_name][type][-87]: routing [null]]
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:209) ~[elasticsearch-5.5.0.jar:5.5.0]
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.start(TransportSingleShardAction.java:186) ~[elasticsearch-5.5.0.jar:5.5.0]
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:95) ~[elasticsearch-5.5.0.jar:5.5.0]
    at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:59) ~[elasticsearch-5.5.0.jar:5.5.0]
    at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:146) ~[elasticsearch-5.5.0.jar:5.5.0]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.5.0.jar:5.5.0]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.5.0.jar:5.5.0]

錯誤關鍵字 No shard available for

 

0x01  磁盤寫滿

經過搜索發現,該類報錯一般是磁盤慢導致的,通過命令df -h發現,存儲數據/data目錄使用量正常,但是es7的安裝目錄/app已經寫滿;

排查發現,安裝目錄內es7的日誌文件過多,導致/app目錄被寫滿,刪除部分過期日誌後,集羣恢復正常;

 

0x02 總結

除了ES數據存儲目錄需要保持足夠的使用空間,ES自己的運行日誌也需要有足夠的磁盤空間,以上兩個任意1個磁盤空間滿了,都會包磁盤不足的錯誤;

注意在es有大量業務的情況下,es自身也會產生大量日誌,需要注意磁盤清理或者配置監控。

 

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章