ELK(Elasticsearch + Logstash + Kibana) 是一套開源的日誌管理方案
Elasticsearch:負責日誌檢索和分析
Logstash:負責日誌的收集,處理和儲存
Kibana:負責日誌的可視化
Logstash: The server component of Logstash that processes incoming logs
Elasticsearch: Stores all of the logs
Kibana 4: Web interface for searching and visualizing logs, which will be proxied through Nginx
Logstash Forwarder: Installed on servers that will send their logs to Logstash, Logstash Forwarder serves as a log forwarding agent that utilizes the lumberjack networking protocol to communicate with Logstash
Reference:
JDK - http://www.oracle.com/technetwork/java/javase/downloads/index.html
Elasticsearch - https://www.elastic.co/downloads/elasticsearch
Logstash - https://www.elastic.co/downloads/logstash
Kibana - https://www.elastic.co/downloads/kibana
redis - http://redis.io/download
數據流流向如下
Logstash-forwarder--->Logstash--->Elasticsearch--->kibana--->nginx--->客戶瀏覽器
其中Logstash-forwarder是客戶端的日誌收集工具將日誌發送給服務端Logstash後
Logstash通過使用grok匹配規則對日誌進行匹配切割
然後保存在Elasticsearch中
最後通過kibana從Elasticsearch中讀取數據並轉交給nginx來處理後返回給客戶。
好了下面就是ELK系統的安裝過程了。
首先安裝JAVA環境
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u65-b17/jdk-8u65-linux-x64.rpm" rpm -Uvh jdk-8u65-linux-x64.rpm
或者直接yum安裝jdk也行不過要保證安裝好對應的版本。
安裝好jdk環境之後需要安裝Elasticsearch
rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch rpm -ivh https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.noarch.rpm
修改配置文件如下
grep -v "^.*#\|^$" /etc/elasticsearch/elasticsearch.yml network.host: localhost path.data: /data/elasticsearch chown -R elasticsearch:elasticsearch /data/elasticsearch
安裝Elasticsearch插件
cd /usr/share/elasticsearch/ && ./bin/plugin -install mobz/elasticsearch-head && ./bin/plugin -install lukas-vlcek/bigdesk
啓動Elasticsearch
service elasticsearch start chkconfig elasticsearch on
測試Elasticsearch
curl http://localhost:9200 { "status" : 200, "name" : "Black Goliath", "cluster_name" : "elasticsearch", "version" : { "number" : "1.7.2", "build_hash" : "e43676b1385b8125d647f593f7202acbd816e8ec", "build_timestamp" : "2015-09-14T09:49:53Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search" }
然後開始安裝kibana
去https://www.elastic.co/downloads/kibana 找合適的版本
每個版本下面有這麼一行內容一定要注意這些內容Compatible with Elasticsearch x.x -- x.x
這裏選擇的是kibana-4.1.3-linux-x64.tar.gz
wget https://download.elastic.co/kibana/kibana/kibana-4.1.3-linux-x64.tar.gz tar xf kibana-4.1.3-linux-x64.tar.gz mv kibana-4.1.3-linux-x64 /usr/local/kibana cd !$ grep -v "^.*#\|^$" /usr/local/kibana/config/kibana.yml port: 5601 host: "localhost" elasticsearch_url: "http://localhost:9200" elasticsearch_preserve_host: true kibana_index: ".kibana" default_app_id: "discover" request_timeout: 300000 shard_timeout: 0 verify_ssl: true bundled_plugin_ids: - plugins/dashboard/index - plugins/discover/index - plugins/doc/index - plugins/kibana/index - plugins/markdown_vis/index - plugins/metric_vis/index - plugins/settings/index - plugins/table_vis/index - plugins/vis_types/index - plugins/visualize/index
配置文件中指明kibana偵聽5601端口並且通過9200端口從elasticsearch裏面獲取數據
啓動 Kibana
nohup /usr/local/kibana/bin/kibana -l /var/log/kibana.log &
或者也可以看看下面兩個腳本
cd /etc/init.d && curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-init cd /etc/default && curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-default ln -s /usr/local/kibana /opt/kibana groupadd -g 1005 kibana useradd -u 1005 -g 1005 kibana chown -R kibana:kibana /usr/local/kibana service kibana start chkconfig kibana on
安裝nginx
yum -y install epel-release yum -y install nginx httpd-tools grep -v "^.*#\|^$" /etc/nginx/nginx.conf user nginx; worker_processes 1; error_log /var/log/nginx/error.log; pid /var/run/nginx.pid; events { worker_connections 1024; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $upstream_response_time $request_time $body_bytes_sent ' '"$http_referer" "$http_user_agent" "$http_x_forwarded_for" "$request_body" ' '$scheme $upstream_addr'; # 修改日誌格式是爲了匹配後面的Logstash的grok匹配規則 access_log /var/log/nginx/access.log main; sendfile on; keepalive_timeout 65; include /etc/nginx/conf.d/*.conf; }
grep -v "^.*#\|^$" /etc/nginx/conf.d/kibana.conf server { listen 80; server_name ocean-lab.ocean.org; location / { proxy_pass http://localhost:5601; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } }
啓動nginx
service nginx start chkconfig nginx on
之後就需要安裝Logstash了
rpm --import https://packages.elasticsearch.org/GPG-KEY-elasticsearch vi /etc/yum.repos.d/logstash.repo [logstash-1.5] name=Logstash repository for 1.5.x packages baseurl=http://packages.elasticsearch.org/logstash/1.5/centos gpgcheck=1 gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch enabled=1 yum -y install logstash
Logstash 安裝成功了 但是仍需要配置
生成 SSL 證書
logstash和logstash-forwarder通信需要使用tls證書認證。
Logstash Forwarder上面只需公鑰logstash需要配置公鑰、私鑰。
在logstash服務器上生成ssl證書。
創建ssl證書有兩種方式一種指定IP地址一種指定fqdn(dns)。
1、指定IP地址方式 [本實驗使用此種方式]
vi /etc/pki/tls/openssl.cnf
在[ v3_ca ]下面配置
subjectAltName = IP:172.16.7.11
# 切記這條很重要因爲logstash-forwarder.conf 還需要 如果配置錯誤 就會一直無法實現認證
cd /etc/pki/tls
openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
# 注意將-days設置大點以免證書過期。
*** 如果logstash服務端的IP地址變換了證書不可用了 ***
like that:
2015/11/29 16:23:48.274974 Failed to tls handshake with 127.0.0.1 x509: certificate is valid for 172.16.7.11, not 127.0.0.1
2、使用 FQDN 方式
不需要修改openssl.cnf文件。
cd /etc/pki/tls
** CN=FQDN **
openssl req -subj '/CN=elk.suzf.net/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
# 將 elk.suzf.net 換成你自己的域名。同時到域名解析那添加 elk.suzf.net 的A記錄。
配置logstash
logstash配置文件是以json格式設置參數的
配置文件位於/etc/logstash/conf.d目錄下配置包括三個部分 輸入/輸出和過濾器
首先創建一個01-lumberjack-input.conf文件
設置lumberjack輸入Logstash-Forwarder使用的協議。
cat /etc/logstash/conf.d/01-lumberjack-input.conf input { lumberjack { port => 5043 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" } }
再來創建一個02-nginx.conf用於過濾nginx日誌
cat /etc/logstash/conf.d/02-nginx.conf filter { if [type] == "nginx" { grok { match => { "message" => "%{IPORHOST:clientip} - %{NOTSPACE:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:method} %{NOTSPACE:request}(?: %{URIPROTO:proto}/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:status} (?:%{NUMBER:upstime}|-) %{NUMBER:reqtime} (?:%{NUMBER:size}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{QS:reqbody} %{WORD:scheme} (?:%{IPV4:upstream}(:%{POSINT:port})?|-)" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } date { match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ] } geoip { source => "clientip" add_tag => [ "geoip" ] fields => ["country_name", "country_code2","region_name", "city_name", "real_region_name", "latitude", "longitude"] remove_field => [ "[geoip][longitude]", "[geoip][latitude]" ] } } }
這個過濾器會尋找被標記爲“nginx”類型Logstash-forwarder定義的的日誌嘗試使用“grok”來分析傳入的nginx日誌使之結構化和可查詢。
type要與logstash-forwarder相匹配。
同時注意將nginx日誌格式設置成上面的。
日誌格式不對grok匹配規則要重寫。
可以通過http://grokdebug.herokuapp.com/ 在線工具進行調試。多半ELK沒數據錯誤在此處。
grok 匹配日誌不成功不要往下看了。搞對爲止先。
同時多看看http://grokdebug.herokuapp.com/patterns# grok匹配模式對後面寫規則匹配很受益的。
最後創建一文件來定義輸出。
cat /etc/logstash/conf.d/30-lumberjack-output.conf output { if "_grokparsefailure" in [tags] { file { path => "/var/log/logstash/grokparsefailure-%{type}-%{+YYYY.MM.dd}.log" } } elasticsearch { host => "127.0.0.1" protocol => "http" index => "logstash-%{type}-%{+YYYY.MM.dd}" document_type => "%{type}" workers => 5 template_overwrite => true } #stdout { codec =>rubydebug } }
定義結構化的日誌存儲到elasticsearch對於不匹配grok的日誌寫入到文件。
注意後面添加的過濾器文件名要位於01-99之間。因爲logstash配置文件有順序的。
在調試時候先不將日誌存入到elasticsearch而是標準輸出以便排錯。
同時多看看日誌很多錯誤在日誌裏有體現也容易定位錯誤在哪。
在啓動logstash服務之前最好先進行配置文件檢測如下
/opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/* Configuration OK
也可指定文件名檢測直到OK才行。不然logstash服務器起不起來。
最後就是啓動logstash服務了。
service logstash start chkconfig logstash on
然後就是配置Logstash-forwarder客戶端了。
安裝logstash-forwarder
wget https://download.elastic.co/logstash-forwarder/binaries/logstash-forwarder-0.4.0-1.x86_64.rpm rpm -ivh logstash-forwarder-0.4.0-1.x86_64.rpm
需要將在安裝logstash時候創建的ssl證書的公鑰拷貝到每臺logstash-forwarder服務器上。
scp 172.16.7.11:/etc/pki/tls/certs/logstash-forwarder.crt /etc/pki/tls/certs/
配置logstash-forwarder
grep -v "^.*#\|^$" /etc/logstash-forwarder.conf { "network": { "servers": [ "172.16.7.11:5043" ], "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt", "timeout": 15 }, "files": [ { "paths": [ "/var/log/messages", "/var/log/secure" ], "fields": { "type": "syslog" } },{ "paths": [ "/var/log/nginx/access.log" ], "fields": { "type": "nginx" } } ] }
這也是個json個是的配置文件
json格式不對logstash-forwarder服務是啓動不起來的
service logstash-forwarder start
連接到 Kibana
創建index
當上面的所有都配置正確的話就可以訪問kibana來查看數據了。
訪問效果如下所示
參考文檔:
https://www.digitalocean.com/community/tutorials/how-to-troubleshoot-common-elk-stack-issues
http://xianglinhu.blog.51cto.com/5787032/1716274
http://www.wklken.me/posts/2015/04/26/elk-for-nginx-log.html