環境說明
架構說明及架構圖
filebeat部署在客戶端用於收集日誌並把收集到的日誌發送到logstash.
logstash把收集到的日誌處理之後交給elasticsearch.
kibana從elasticsearch中提取數據並進行展示.
之所以使用filebeat進行日誌收集是因爲filebeat不會像logstash使用大量的資源,影響業務服務器.
環境需求
需要java環境和redis
yum install java yum install redis
使用版本
java 1.8.0_111
redis 2.8.16
filebeat 5.1.2
logstash 5.1.2
elasticsearch 5.1.1
kibana 5.1.1
安裝配置
filebeat安裝及配置
安裝filebeat
rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch cat > /etc/yum.repos.d/elk.repo <<EOF [elasticsearch-5.x] name=Elasticsearch repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md EOF yum clean all yum install filebeat -y
配置filebeat
vim /etc/filebeat/filebeat.yml paths: - /var/log/nginx/access.log tags: ["nginx"] output.logstash: hosts: ["1.8.101.53:5044"]
其中path選項爲filebeat發送給logstash的路徑, 多個日誌可以使用*.log通配. file不會自動遞歸日誌目錄下的子目錄, 如果需要遞歸子目錄可以使用類似 /var/log/*/*.log 的結構. tags選項會向log中添加一個標籤, 此標籤可以提供給logstash用於區分不同客戶端不同業務的log. output指定發送log到哪臺服務器的哪個服務, 默認輸出到elasticsearch. 本例使用logstash, 所以需要註釋掉髮送到elasticsearch的配置, 並啓用發送到logstash的配置.
啓動filebeat
/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -e &
logstash安裝及配置
安裝logstash
rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch cat > /etc/yum.repos.d/elk.repo <<EOF [elasticsearch-5.x] name=Elasticsearch repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md EOF yum clean all yum install logstash ln -s /usr/share/logstash/bin/logstash /usr/bin/logstash
配置logstash
vim /etc/logstash/conf.d/nginx.conf
input {
beats {
port => 5044
}
}
filter {
if "nginx" in [tags] {
grok {
match => [ "message","%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}"]
overwrite => [ "message"]
}
mutate {
convert => ["response","integer"]
convert => ["bytes","integer"]
convert => ["responsetime","float"]
}
geoip {
source=>"clientip"
target => "geoip"
}
date {
match => [ "timestamp","dd/MMM/YYYY:HH:mm:ss Z"]
remove_field => [ "timestamp"]
}
useragent {
source=>"agent"
}
}
}
output {
if "nginx" == [tags][0] {
elasticsearch {
hosts => ["1.8.101.53:9200"]
index => "access-%{+YYYY.MM.dd}"
}
}
}啓動logstash
logstash --path.settings /etc/logstash
elasticsearch安裝及配置
安裝elasticsearch
yum install elasticsearch
啓動elasticsearch
service elasticsearch start
elasticsearch本身不需要過多配置, 用包管理工具啓動即可
kibana安裝及配置
安裝kibana
yum intall kibana
啓動kibana
/usr/share/kibana/bin/kibana &