ELK之filebeat收集多日誌並自定義索引

需求說明

1、在《ELK收集Apache的json格式訪問日誌並按狀態碼繪製圖表》中，收集了Apache的json格式日誌，在此實驗基礎上，增加nginx的json日誌收集，並自定義filebeat的索引。本次實驗也是基於《ELK收集Apache的json格式訪問日誌並按狀態碼繪製圖表》；
2、將nginx和Apache的日誌按照狀態碼繪製柱狀圖，並將其添加到dashboard；

環境說明

10.0.0.101（test101）——部署apache、nginx、filebeat
10.0.0.102（test102）——部署elasticsearch、kibana
系統：centos7.3
備註：本次實驗的重點在於怎樣用filebeat收集多日誌的json格式日誌，並自定義索引，因此也沒有裝logstash。日誌數據從filebeat——elasticsearch——kibana展示

操作步驟

1、在test101服務器部署nginx並配置日誌格式

1.1 安裝nginx

[root@test101 conf]#yum -y install nginx

1.2 修改配置文件將端口改成8080（因爲80端口已經被Apache佔用）

[root@test101 conf]# vim /etc/nginx/conf.d/default.conf 

server {
    listen       8080;      #將默認的80端口改成80
    server_name  localhost;

    #charset koi8-r;
    #access_log  /var/log/nginx/host.access.log  main;
        ......   #省略若干行
[root@test101 conf]#

1.3 編寫index.html
在/usr/share/nginx/html/目錄下編寫了一個測試的index.html文件，以提供訪問，生成日誌：

當前生成的日誌格式爲：

[root@test101 nginx]# tailf /var/log/nginx/access.log 

10.0.0.1 - - [17/Dec/2018:11:25:11 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36" "-"
10.0.0.1 - - [17/Dec/2018:11:25:11 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36" "-"
10.0.0.1 - - [17/Dec/2018:11:25:11 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36" "-"

1.4 修改nginx輸出格式爲json格式
修改/etc/nginx/nginx.conf，註釋掉文件中log_format和access_log，添加新的日誌輸出格式：

[root@test101 nginx]# cat -n nginx.conf
     1  
     2  user  nginx;
     3  worker_processes  1;
     4  
     5  error_log  /var/log/nginx/error.log warn;
     6  pid        /var/run/nginx.pid;
     7  
     8  
     9  events {
    10      worker_connections  1024;
    11  }
    12  
    13  
    14  http {
    15      include       /etc/nginx/mime.types;
    16      default_type  application/octet-stream;

    17  #註釋掉下面部分內容（18-21行）：
    18      #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    19      #                 '$status $body_bytes_sent "$http_referer" '
    20      #                  '"$http_user_agent" "$http_x_forwarded_for"';
    21      #access_log  /var/log/nginx/access.log  main;

    22  #添加新的日誌輸出格式（23-33行）
    23      log_format main_json '{"@timestamp":"$time_local",'
    24      '"N_client_ip": "$remote_addr",'
    25      '"N_request": "$request",'
    26      '"N_request_time": "$request_time",'
    27      '"N_status": "$status",'
    28      '"N_bytes": "$body_bytes_sent",'
    29      '"N_user_agent": "$http_user_agent",'
    30      '"N_x_forwarded": "$http_x_forwarded_for",'
    31      '"N_referer": "$http_referer"'
    32      '}';
    33       access_log  /var/log/nginx/access.log main_json;
    34  
    35  
    36      sendfile        on;
    37      #tcp_nopush     on;
    38  
    39      keepalive_timeout  65;
    40  
    41      #gzip  on;
    42  
    43      include /etc/nginx/conf.d/*.conf;
    44  }
[root@test101 nginx]# 
}

1.5 重啓nginx，檢查日誌輸出格式
重啓nginx後，再訪問nginx首頁http://10.0.0.101:8080 ，可以看到日誌格式已經變成了json格式：

[root@test101 nginx]# tailf /var/log/nginx/access.log 

{"@timestamp":"17/Dec/2018:11:33:01 +0800","N_client_ip": "10.0.0.1","N_request": "GET / HTTP/1.1","N_request_time": "0.000","N_status": "304","N_bytes": "0","N_user_agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36","N_x_forwarded": "-","N_referer": "-"}
{"@timestamp":"17/Dec/2018:11:33:02 +0800","N_client_ip": "10.0.0.1","N_request": "GET / HTTP/1.1","N_request_time": "0.000","N_status": "304","N_bytes": "0","N_user_agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36","N_x_forwarded": "-","N_referer": "-"}
{"@timestamp":"17/Dec/2018:11:33:03 +0800","N_client_ip": "10.0.0.1","N_request": "GET / HTTP/1.1","N_request_time": "0.000","N_status": "304","N_bytes": "0","N_user_agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36","N_x_forwarded": "-","N_referer": "-"}

2、配置test101服務器filebeat日誌採集

2.1 修改test101的filebeat配置文件，同時收集Apache和nginx的json日誌，同時自定義索引
filebeat.yml文件修改兩個地方：
1）修改 Filebeat inputs部分，增加nginx的日誌採集

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log
  enabled: true
  paths:
    - /var/log/httpd/access_log
  json.keys_under_root: true
  json.overwrite_keys: true

- type: log          #增加nginx的日誌收集內容
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

2）修改Elasticsearch output 部分，增加索引配置

#-------------------------- Elasticsearch output ------------------------------
setup.template.name: "test101_web"     #增加索引
setup.template.pattern: "test101_web-"    #增加索引
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["10.0.0.102:9200"]
  index: "test101_web-%{+yyyy.MM.dd}"    #增加索引
  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

3）刪除kibana界面創建的索引和test102服務器上elasticsearch的索引：

[root@test102 ~]# curl 10.0.0.102:9200/_cat/indices
green open .kibana_1 udOUvbprSnKWUJISwD0r_g 1 0 3 0 62.8kb 62.8kb
[root@test102 ~]#

4）重啓test101的filebeat，生成新的索引：

[root@test102 filebeat]# curl 10.0.0.102:9200/_cat/indices
yellow open test101_web-2018.12.17 Rg31xncWSAm4oLER8DO5yg 5 1 45 0 589.4kb 589.4kb   #新的索引
green  open .kibana_1              udOUvbprSnKWUJISwD0r_g 1 0  6 0  34.5kb  34.5kb
[root@test102 filebeat]#