一、ELKStack簡介
Elstaicsearch:存儲和搜索 logstash：收集 kibana：展示
二、ELK----之Logstash安裝
環境準備
IP 主機名 操作系統 192.168.56.11 linux-node1 centos7 192.168.56.12 linux-node2 centos7
在兩臺服務器上都安裝logstash
1、JDK安裝
安裝JDK
[root@linux-node1 ~]# yum install -y java [root@linux-node1 ~]# java -version openjdk version "1.8.0_91" OpenJDK Runtime Environment (build 1.8.0_91-b14) OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
2、LogStash安裝
[root@linux-node1 ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch [root@linux-node1 ~]# cat /etc/yum.repos.d/logstash.repo [logstash-2.3] name=Logstash repository for 2.3.x packages baseurl=https://packages.elastic.co/logstash/2.3/centos gpgcheck=1 gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch enabled=1
三、Logstash配置
1、概念理解
input: 標準輸入 output: 標準輸出 filter: 過濾
2、命令行輸出
[root@linux-node1 ~]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'
Settings: Default pipeline workers: 1
Pipeline main started
hehe
2016-08-20T05:58:42.683Z linux-node1 hehe
world
2016-08-20T05:58:52.057Z linux-node1 world3、以json的形式在命令行輸出
在標準輸出中添加codec=>rubydebug功能，這樣能以json的格式顯示出來
[root@linux-node2 ~]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug } }'
Settings: Default pipeline workers: 1
Pipeline main started
四世同堂
{
"message" => "四世同堂",
"@version" => "1",
"@timestamp" => "2016-08-27T18:37:04.702Z",
"host" => "linux-node2"
}4、把日誌寫入到elasticsearch裏面
這裏我們用到output的插件
hosts是個數組類型，在集羣狀態下，可以寫多個地址
index是一個字符串類型，可以按照年月日生成索引
[root@linux-node2 ~]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["192.168.56.11:9200"] index => "logstash-%{+YYYY.MM.dd}" } }'
Settings: Default pipeline workers: 1
Pipeline main started
天蒼蒼，野茫茫
風吹草地見牛羊在es上訪問http://192.168.56.11:9200/_plugin/head/ 在數據瀏覽頁面找到當天的logstash日誌，
比如logstash-2016.08.30，查看其message信息。
5、把日誌輸出到不同位置
我們有時候會有這樣的需求，即把日誌輸出到標準輸出，又輸出到es
[root@linux-node2 ~]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout { codec => rubydebug } elasticsearch { hosts => ["192.168.56.11:9200"] index => "logstash-%{+YYYY.MM.dd}" } }'
Settings: Default pipeline workers: 1
Pipeline main started
菊花殘，滿地傷
{
"message" => "菊花殘，滿地傷",
"@version" => "1",
"@timestamp" => "2016-08-30T15:36:22.606Z",
"host" => "linux-node2"
}
你的笑容已泛黃
{
"message" => "你的笑容已泛黃",
"@version" => "1",
"@timestamp" => "2016-08-30T15:36:31.630Z",
"host" => "linux-node2"
}在實際生產中，每個需要收集日誌的服務器都安裝一個logstash 通過網絡傳輸的不需要安裝，比如syslog，可以通過網絡發送
6、編寫logstash的輸入輸出文件
logstash的輸入輸出文件都存放在
/etc/logstash/conf.d
這是在啓動腳本中定義了
PATH=/sbin:/usr/sbin:/bin:/usr/bin
export PATH
if [ `id -u` -ne 0 ]; then
echo "You need root privileges to run this script"
exit 1
fi
name=logstash
pidfile="/var/run/$name.pid"
LS_USER=root
LS_GROUP=root
LS_HOME=/var/lib/logstash
LS_HEAP_SIZE="1g"
LS_LOG_DIR=/var/log/logstash
LS_LOG_FILE="${LS_LOG_DIR}/$name.log"
LS_CONF_DIR=/etc/logstash/conf.d
......編寫一個demo.conf文件
[root@linux-node2 conf.d]# vim demo.conf
input{
stdin{}
}
filter{
}
output{
elasticsearch{
hosts => ["192.168.56.11:9200"]
index => "logstash-%{+YYYY.MM.dd}"
}
stdout{
codec => rubydebug
}
}在終端執行一下，查看結果
[root@linux-node2 conf.d]# /opt/logstash/bin/logstash -f demo.conf
Settings: Default pipeline workers: 1
Pipeline main started
誰的江山
{
"message" => "誰的江山",
"@version" => "1",
"@timestamp" => "2016-08-30T15:59:48.446Z",
"host" => "linux-node2"
}
馬蹄聲狂亂
{
"message" => "馬蹄聲狂亂",
"@version" => "1",
"@timestamp" => "2016-08-30T15:59:59.306Z",
"host" => "linux-node2"7、收取操作系統日誌並寫入elasticsearch
需求分析：
將"/var/log/message","/var/log/secure"寫入到ES中
自定義一個名爲file.conf的文件
[root@linux-node2 /etc/logstash/conf.d]# cat file.conf
input{
file{
path => ["/var/log/message","/var/log/secure"]
type => "system-log"
start_position => "beginning" #從文件頭開始收取日誌
}
}
filter{}
output{
elasticsearch{
hosts => ["192.168.56.11:9200"]
index => "system-log-%{+YYYY.MM}"
}
}啓動logstash
[root@linux-node2 conf.d]# /opt/logstash/bin/logstash -f file.conf Settings: Default pipeline workers: 1 Pipeline main started 注意： 注意： 注意： logstash在後臺啓動時會加載/etc/logstash/conf.d/目錄下的所有conf文件
在es上訪問http://192.168.56.11:9200/_plugin/head/ 查看syslog-log的日誌
