第1章 日誌分析系統需求
1.1 運維痛點
1.運維要不停的查看各種日誌。
2.故障已經發生了纔看日誌(時間問題) 。
3.節點多，日誌分散，收集日誌成了問題。
4.運行日誌，錯誤等日誌等，沒有規範目錄，收集困難。
1.2 環境痛點
1.開發人員不能登陸線上服務器查看詳細日誌。
2.各個系統都有日誌，日誌數據分散難以查找。
3.日誌數據量大，查詢速度慢，數據不夠實時
1.3 解決痛點
1.收集（Logstash）
2.存儲（Elasticsearch、Redis、Kafka）
3.搜索+統計+展示（Kibana)
4.報警，數據分析（Zabbix）
第2章 ElkStack介紹
對於日誌來說，最常見的需求就是收集、存儲、查詢、展示，開源社區正好有相對應的開源項目：logstash（收集）、elasticsearch（存儲+搜索）、kibana（展示），我們將這三個組合起來的技術稱之爲ELKStack，所以說ELKStack指的是Elasticsearch、Logstash、Kibana技術棧的結合，一個通用的架構如下圖所示：
第3章 ElkStack環境
node1和node2爲elasticsearch集羣
2.node3收集對象,Nginx、java、tcp、syslog等日誌
3.node4將logstash日誌寫入Redis,減少程序對elasticsearch依賴性，同時實現程序解耦以及架構擴展。
4.被收集主機需要部署Logstash。
主機 | ip | 服務 | 系統 |
linux-node1 | 192.168.21.133 | redis elasticsearch logstash kibana | CentOS release 6.5 (Final) |
linux-node2 | 192.168.21.134 | redis elasticsearch logstash kibana nginx | CentOS release 6.5 (Final) |
第4章 ElkStack部署
4.1 java
[root@ELK-server data]# java -version
java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 1.8.0_112-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixedmode)
4.2 下載解壓即可
tar xf elasticsearch-5.0.2.tar.gz -C /usr/local/elasticsearch
4.3 具體的配置
cd /usr/local/elasticsearch/config
[root@ELK-server config]# grep '^[a-z]' elasticsearch.yml
cluster.name: es-log
node.name: log-1
path.data: /usr/local/elasticsearch/data
path.logs:/var/log/elasticsearch/elasticsearch.log
bootstrap.memory_lock: true
network.host: 192.168.21.133
http.port: 9200
discovery.zen.ping.unicast.hosts:["192.168.21.133", "192.168.21.134"]
discovery.zen.minimum_master_nodes: 1
4.4 其他系統設置
[root@ELK-server config]# echo " vm.max_map_count = 262144
" >>/etc/sysctl.conf
[root@ELK-server config]# sysctl -p
[root@ELK-server config]# tail -2 /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
[root@ELK-server config]# vim /etc/security/limits.d/90-nproc.conf
* soft nproc 2048
4.5 報錯總結
1、can not run elasticsearch as root
切換到非root用戶
2、main ERROR Could not register mbeansjava.security.AccessControlException: access denied("javax.management.MBeanTrustPermission" "register")
改變elasticsearch文件夾所有者到當前用戶
sudo chown -R noroot:noroot elasticsearch
3、max virtual memory areas vm.max_map_count [65530] is too low,increase to at least [262144]
sudo vi /etc/sysctl.conf
添加下面配置：
vm.max_map_count=655360
並執行命令：
sudo sysctl -p
4、max file descriptors [4096] for elasticsearch process is toolow, increase to at least [65536]
sudo vi /etc/security/limits.conf
添加如下內容:
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
sudo vi /etc/pam.d/common-session
添加 session required pam_limits.so
sudo vi/etc/pam.d/common-session-noninteractive
添加 session required pam_limits.so
ack or non-link-local address, enforcing bootstrap checks
ERROR: bootstrap checks failed
memory locking requested for elasticsearch process but memoryis not locked
[root@ELK-server config]# vimelasticsearch.yml
#bootstrap.memory_lock: true #註釋
[root@ELK-server elasticsearch]# ./bin/elasticsearch
Can't start up: not enough memory
[root@ELK-server elasticsearch]# java -version
java version "1.5.0"
...
解決:
[root@ELK-server elasticsearch]# java -version
java version "1.8.0_112"
...
4.6 運行elasticsearch
[root@ELK-server elasticsearch]# ./bin/elasticsearch
...
[root@ELK-server config]# lsof -i:9200
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 3342 elk 109u IPv6 132688 0t0 TCP ELK-server:9200 (LISTEN)
[root@ELK-serverconfig]# curl -I '192.168.21.133:9200'
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 318
4.7 測試部署
[root@ELK-serverconfig]# curl -XGET '192.168.21.133:9200'
{
"name" :"log-1",
"cluster_name" : "es-log",
"cluster_uuid" : "HXIBVdzHTJqi5lexARIgGw",
"version" :{
"number": "5.0.2",
"build_hash" : "f6b4951",
"build_date" : "2016-11-24T10:07:18.101Z",
"build_snapshot" : false,
"lucene_version" : "6.2.1"
},
"tagline" :"You Know, for Search"
}
4.8 配置詳解
cd /usr/local/elasticsearch/config
[root@ELK-server config]# grep '^[a-z]' elasticsearch.yml
cluster.name: es-log #集羣名稱
node.name: log-1 #節點，保持唯一性
path.data: /usr/local/elasticsearch/data #數據存放目錄
path.logs: /var/log/elasticsearch/elasticsearch.log #日誌存放目錄
bootstrap.memory_lock: true #不使用swap分區,鎖住內存
network.host: 192.168.21.133 #允許訪問的IP
http.port: 9200 #訪問端口
discovery.zen.ping.unicast.hosts:["192.168.21.133", "192.168.21.134"] #多播地址
discovery.zen.minimum_master_nodes: 1 #生產環境配置至少兩臺
4.9 Elasticsearch插件
/usr/local/elasticsearch/bin/plugin -ielasticsearch/marcel/latest
/usr/local/elasticsearch/bin/plugin installmobz/elasticsearch-head
重新創建一個行新的node：
只需要改變：
[root@jenkins elasticsearch]# grep node.nameconfig/elasticsearch.yml
node.name: "linux-node2"
集羣管理插件：
訪問head集羣插件:http://ES_IP:9200/_plugin/head/
集羣健康值顏色詳解：
黃色：所有的主分片都是正常運行，副本是有數據的丟失；
綠色：所有的節點正常；
紅色：主節點有數據丟失，很嚴重。
zabbix監控狀態：
curl -XGET 'http://192.168.21.134:9200/_cluster/health?pretty'2>/dev/null|awk -F'"' 'NR==3{print $4}'
[root@jenkins elasticsearch]# curl -XGET 'http://192.168.21.134:9200/_cluster/health'
{"cluster_name":"zhangyiling","status":"green","timed_out":false,"number_of_nodes":2,"number_of_data_nodes":2,"active_primary_shards":10,"active_shards":20,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0}
[root@jenkins elasticsearch]# curl -XGET 'http://192.168.21.134:9200/_cluster/health?pretty'
{
"cluster_name" : "zhangyiling",
"status" :"green", #監控這個值
"timed_out": false,
"number_of_nodes" : 2, #節點數
"number_of_data_nodes" : 2, #數據節點
"active_primary_shards" : 10, #主分片
"active_shards" : 20, #所有的分片
"relocating_shards" : 0, #正在遷移的分片
"initializing_shards" : 0, #正在初始化的分片
"unassigned_shards" : 0, #沒有分配的節點
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
}
4.10 elasticsearch服務管理
https://github.com/elastic/elasticsearch-servicewrapper
cp -r service//usr/local/elasticsearch/bin/
[root@ELK-server bin]# /usr/local/elasticsearch/bin/service/elasticsearch
Usage: /usr/local/elasticsearch/bin/service/elasticsearch [console | start | stop | restart | condrestart | status | install | remove |dump ]
Commands:
console Launch in the current console.
start Start in the background as a daemonprocess.
stop Stop if running as a daemon or inanother console.
restart Stop if running and then start.
condrestart Restart only if already running.
status Query the current status.
install Install to start automatically when systemboots.
remove Uninstall.
dump Request a Java thread dump if running.
[root@ELK-server bin]# /usr/local/elasticsearch/bin/service/elasticsearchinstall
Detected RHEL or Fedora:
Installing the Elasticsearch daemon..
4.11 官方文檔
https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html
https://www.elastic.co/guide/en/elasticsearch/guide/current/administration.html