Elasticsearch + Logstash + Kibana（ELK）安裝部署方法

原創 zzxtbl 2018-09-11 08:54

18741457600589.png

Elasticsearch + Logstash + Kibana（ELK）是一套開源的日誌管理方案，分析網站的訪問情況時我們一般會藉助Google/百度/CNZZ等方式嵌入JS做數據統計，但是當網站訪問異常或者被***時我們需要在後臺分析如Nginx的具體日誌，而Nginx日誌分割/GoAccess/Awstats都是相對簡單的單節點解決方案，針對分佈式集羣或者數據量級較大時會顯得心有餘而力不足，而ELK的出現可以使我們從容面對新的挑戰。

 

Logstash：負責日誌的收集，處理和儲存

Elasticsearch：負責日誌檢索和分析

Kibana：負責日誌的可視化


學習網站 elk
https://www.elastic.co/
http://kibana.logstash.es/content/logstash/examples/nginx-access.html
https://www.gitbook.com/book/fuxiaopang/learnelasticsearch/details


國內的klb

http://blog.csdn.net/ebw123/article/details/46707559

http://www.tuicool.com/articles/m6RNBrY

http://baidu.blog.51cto.com/71938/1676798

http://dl528888.blog.51cto.com/2382721/1703059

http://dl528888.blog.51cto.com/2382721/1703059

http://caochun.blog.51cto.com/4497308/1715462


http://www.linuxdiyf.com/linux/15787.html

http://www.linuxidc.com/Linux/2015-12/126587.htm

http://blog.csdn.net/super_scan/article/details/45694289


http://www.cnblogs.com/xiaouisme/p/3977721.html

http://blog.sina.com.cn/s/blog_7b837d030101ckia.html

http://www.tuicool.com/articles/bUvmUrb

http://www.360doc.com/content/15/0512/09/1073512_469853970.shtml


博客寫的好

http://www.360doc.com/userhome/1073512

http://www.wklken.me/


國外的網站

https://github.com/instruct-br/webinar-elk   

https://github.com/instruct-br/webinar-elk/tree/master/puppet/environments/production/modules


1. 主機規劃

14981457600589.png

(1)要求兩臺主機的時間同步

(2)ssl 設置

 

2.修改兩臺主機的hostname

17261457600589.png

3.服務器端安裝elk

81001457600589.png

（2）安裝Elasticsearch


#下載安裝

wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.noarch.rpm

yum localinstall elasticsearch-1.7.1.noarch.rpm

 

#啓動相關服務

service elasticsearch start

service elasticsearch status

 

#查看Elasticsearch的配置文件

rpm -qc elasticsearch

 

/etc/elasticsearch/elasticsearch.yml

/etc/elasticsearch/logging.yml

/etc/init.d/elasticsearch

/etc/sysconfig/elasticsearch

/usr/lib/sysctl.d/elasticsearch.conf

/usr/lib/systemd/system/elasticsearch.service

/usr/lib/tmpfiles.d/elasticsearch.conf

#查看端口使用情況

netstat -nltp

80481457600589.png

#測試訪問

curl -X GET http://localhost:9200/

56311457600589.png

（3）安裝Kibana

#下載tar包

wget  https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz

#解壓

tar  zxf  kibana-4.1.1-linux-x64.tar.gz -C /usr/local/

cd /usr/local/

mv  kibana-4.1.1-linux-x64 kibana


#創建kibana服務
vi /etc/rc.d/init.d/kibana

#!/bin/bash
### BEGIN INIT INFO
# Provides:          kibana
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Runs kibana daemon
# Description: Runs the kibana daemon as a non-root user
### END INIT INFO

# Process name
NAME=kibana
DESC="Kibana4"
PROG="/etc/init.d/kibana"

# Configure location of Kibana bin
KIBANA_BIN=/usr/local/kibana/bin

# PID Info
PID_FOLDER=/var/run/kibana/
PID_FILE=/var/run/kibana/$NAME.pid
LOCK_FILE=/var/lock/subsys/$NAME
PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN
DAEMON=$KIBANA_BIN/$NAME

# Configure User to run daemon process
DAEMON_USER=root
# Configure logging location
KIBANA_LOG=/var/log/kibana.log

# Begin Script
RETVAL=0

if [ `id -u` -ne 0 ]; then
        echo "You need root privileges to run this script"
        exit 1
fi

# Function library
. /etc/init.d/functions
 
start() {
        echo -n "Starting $DESC : "

pid=`pidofproc -p $PID_FILE kibana`
        if [ -n "$pid" ] ; then
                echo "Already running."
                exit 0
        else
        # Start Daemon
if [ ! -d "$PID_FOLDER" ] ; then
                        mkdir $PID_FOLDER
                fi
daemon --user=$DAEMON_USER --pidfile=$PID_FILE $DAEMON 1>"$KIBANA_LOG" 2>&1 &
                sleep 2
                pidofproc node > $PID_FILE
                RETVAL=$?
                [[ $? -eq 0 ]] && success || failure
echo
                [ $RETVAL = 0 ] && touch $LOCK_FILE
                return $RETVAL
        fi
}

reload()
{
    echo "Reload command is not implemented for this service."
    return $RETVAL
}

stop() {
        echo -n "Stopping $DESC : "
        killproc -p $PID_FILE $DAEMON
        RETVAL=$?
echo
        [ $RETVAL = 0 ] && rm -f $PID_FILE $LOCK_FILE
}
 
case "$1" in
  start)
        start
;;
  stop)
        stop
        ;;
  status)
        status -p $PID_FILE $DAEMON
        RETVAL=$?
        ;;
  restart)
        stop
        start
        ;;
  reload)
reload
;;
  *)
# Invalid Arguments, print the following message.
        echo "Usage: $0 {start|stop|status|restart}" >&2
exit 2
        ;;
esac

#修改啓動權限

chmod +x /etc/rc.d/init.d/kibana

 

#啓動kibana服務

service kibana start

service kibana status


#查看端口

netstat -nltp

9621457600589.png



（4）設置ssl，之前設置的FQDN是elk.zzxtbl.com

 

openssl req -subj '/CN=elk.zzxtbl.com/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.cr


（5） Logstash

#下載rpm包

wget  https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.4-1.noarch.rpm

#安裝

yum  localinstall logstash-1.5.4-1.noarch.rpm

#創建一個01-logstash-initial.conf文件

#創建一個01-logstash-initial.conf文件
cat > /etc/logstash/conf.d/01-logstash-initial.conf << EOF
input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}


filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}
EOF

#啓動logstash服務

service logstash start

service logstash status

 

#查看5000端口

netstat -nltp

20121457600589.png

#增加節點和客戶端配置一樣，注意同步證書

scp /etc/pki/tls/certs/logstash-forwarder.crt  node:/opt

4.客戶端安裝Logstash Forwarder

#登陸到客戶端，安裝Logstash Forwarder
wget https://download.elastic.co/logstash-forwarder/binaries/logstash-forwarder-0.4.0-1.x86_64.rpm
yum localinstall logstash-forwarder-0.4.0-1.x86_64.rpm

#查看logstash-forwarder的配置文件位置
rpm -qc logstash-forwarder
/etc/logstash-forwarder.conf

#備份配置文件
cp /etc/logstash-forwarder.conf /etc/logstash-forwarder.conf.save

#編輯 /etc/logstash-forwarder.conf，需要根據實際情況進行修改

cat > /etc/logstash-forwarder.conf << EOF
{
  "network": {
    "servers": [ "elk.zzxtbl.com:5000" ],

    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",

    "timeout": 15
  },

  "files": [
    {
      "paths": [
        "/var/log/messages",
        "/var/log/secure"
      ],

      "fields": { "type": "syslog" }
    }
  ]
}
EOF


#啓動服務

service logstash-forwarder start

service logstash-forwarder status

 

#訪問Kibana，Time-field name 選擇 @timestamp

http://localhost:5601/

59161457600589.png

配置Nginx日誌策略

（1）修改客戶端的配置

#修改客戶端配置
vi /etc/logstash-forwarder.conf

{
  "network": {
    "servers": [ "elk.zzxtbl.com:5000" ],

    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",

    "timeout": 15
  },

  "files": [
    {
      "paths": [
        "/var/log/messages",
        "/var/log/secure"
      ],
      "fields": { "type": "syslog" }
    }, {
      "paths": [
        "/app/local/nginx/logs/access.log"
      ],
      "fields": { "type": "nginx" }
    }
  ]
}

（2）在服務器端加patterns



#服務端增加patterns
mkdir /opt/logstash/patterns
vi /opt/logstash/patterns/nginx

NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:remote_addr} - - \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATH:path}(?:%{URIPARAM:param})? HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}

#官網pattern的debug在線工具
https://grokdebug.herokuapp.com/

#修改logstash權限
chown -R logstash:logstash /opt/logstash/patterns

#修改服務端配置
vi /etc/logstash/conf.d/01-logstash-initial.conf 

input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}


filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
  if [type] == "nginx" {
    grok {
       match => { "message" => "%{NGINXACCESS}" }
    }
  }
}

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

 

重啓服務端elk服務

63111457600589.png

顯示頁面

73431457600590.png

以上就是簡單的elk分享！！


發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章

記一次被劫持掛馬經歷--Elasticsearch的遠程執行漏洞

xxrenzhe11 2019-02-22 23:58:35

Elastic Stack實戰學習教程~日誌數據的收集、分析與可視化

碼農阿呆 2019-02-22 22:21:43

Elasticsearch 評分排序

王清培 2019-02-24 12:50:09

Elasticsearch 評分排序

王清培 2019-02-24 10:58:01

Flask 教程 第十九章：Docker容器上的部署

天降攻城獅 2019-02-23 10:17:16

Flask 教程 第十八章：Heroku上的部署

天降攻城獅 2019-02-23 10:17:16

Perl Search::Elasticsearch模塊使用經驗總結

老鷹X 2019-02-22 23:53:40

ElasticSearch中的一些基本概念

zhulinu 2019-02-22 23:12:26

Elasticsearch進化史

zhulinu 2019-02-22 23:12:22

《從Lucene到Elasticsearch：全文檢索實戰》學習筆記二

柯之夢 2019-02-22 22:21:25

Elasticsearch集羣配置

cexpert 2019-02-22 22:13:11

Elasticsearch集羣的安裝

ly36843 2019-02-22 21:56:31

Elasticsearch的CURD操作

ly36843 2019-02-22 21:56:26

ELK(ElasticSearch, Logstash, Kibana)搭建實時日誌分析平臺

轉身撞牆角 2019-02-22 22:32:33

Logstash6.1 手動安裝插件

轉身撞牆角 2019-02-22 22:32:33