裝完elk跑起來之後，我的內心幾乎是崩潰的，16G內存16核cpu還經常報錯。
一、logstash和elasticsearch同時報錯
logstash出現大量報錯，可能是es佔用heap太多，沒有優化es導致的
retrying failed action with response code: 503 {:level=>:warn}
too many attempts at sending event. dropping: 2016-06-16T05:44:54.464Z %{host} %{message} {:level=>:error}
elasticsearch出現大量報錯
too many open files
是這個值太小了"max_file_descriptors" : 2048,
# curl http://localhost:9200/_nodes/process\?pretty
{
"cluster_name" : "elasticsearch",
"nodes" : {
"ZLgPzMqBRoyDFvxoy27Lfg" : {
"name" : "Mass Master",
"transport_address" : "inet[/192.168.153.200:9301]",
"host" : "localhost",
"ip" : "127.0.0.1",
"version" : "1.6.0",
"build" : "cdd3ac4",
"http_address" : "inet[/192.168.153.200:9200]",
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 943,
"max_file_descriptors" : 2048,
"mlockall" : true
解決辦法：
設置文件打開數
# ulimit -n 65535
設置開機自啓動
# vi /etc/profile
在es啓動文件裏面添加，然後重新啓動elasticsearch
# vi /home/elk/elasticsearch-1.6.0/bin/elasticsearch
ulimit -n 65535
# curl http://localhost:9200/_nodes/process\?pretty
{
"cluster_name" : "elasticsearch",
"nodes" : {
"_QXVsjL9QOGMD13Eb6t7Ag" : {
"name" : "Ocean",
"transport_address" : "inet[/192.168.153.200:9301]",
"host" : "localhost",
"ip" : "127.0.0.1",
"version" : "1.6.0",
"build" : "cdd3ac4",
"http_address" : "inet[/192.168.153.200:9200]",
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 1693,
"max_file_descriptors" : 65535,
"mlockall" : true
}
}
二、out of memory內存溢出
優化後的es配置文件內容：
# egrep -v '^$|^#' /home/elk/elasticsearch-1.6.0/config/elasticsearch.yml
bootstrap.mlockall: true
http.max_content_length: 2000mb
http.compression: true
index.cache.field.type: soft
index.cache.field.max_size: 50000
index.cache.field.expire: 10m
針對bootstrap.mlockall: true還要設置
# ulimit -l unlimited
# vi /etc/sysctl.conf
vm.max_map_count=262144
vm.swappiness = 1
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 127447
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 127447
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
# vi /etc/security/limits.d/90-nproc.conf
* soft nproc 320000
root soft nproc unlimited
三、es狀態是yellow
es中用三種顏色狀態表示:green，yellow，red.
green：所有主分片和副本分片都可用
yellow：所有主分片可用，但不是所有副本分片都可用
red：不是所有的主分片都可用
# curl -XGET http://localhost:9200/_cluster/health\?pretty
{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 1,
"active_primary_shards" : 161,
"active_shards" : 161,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 161,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
解決辦法：建立elasticsearch集羣(下篇博客寫)
四、kibana not indexed錯誤
https://rafaelmt.net/en/2015/09/01/kibana-tutorial/#refresh-fields
kibana的索引根據事件會經常更新，所以kibana圖有時候會出現 not indexed的報錯：
解決辦法：
我們訪問kibana，然後選擇settings，點擊indices，點擊logstash-*。點擊刷新的圖標就ok了