Beats輕量級日誌採集工具

Beats 平臺集合了多種單一用途數據採集器。這些採集器安裝後可用作輕量型代理，從成百上千或成千上萬臺機器向 Logstash 或 Elasticsearch 發送數據。常用的Beats有Filebeat(收集文件)、Metricbeat(收集服務、系統的指標數據)、Packetbeat(收集網絡包)等。這裏主要介紹Filebeat插件。

一、架構圖

二、安裝Filebeat

官網地址： https://www.elastic.co/cn/products/beats

1、下載並安裝Filebeat

wget  https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.2-linux-x86_64.tar.gz
tar -xzf filebeat-6.3.2-linux-x86_64.tar.gz -C /usr/local/
cd /usr/local/
ln -s filebeat-6.3.2-linux-x86_64 filebeat

2、自定義配置文件
① 簡單版本的配置文件

cd /usr/local/filebeat/
cat > test.yml << END
filebeat.inputs:
- type: stdin
  enabled: true
setup.template.settings:
  index.number_of_shards: 3
output.console:
  pretty: true
  enable: true

END

#啓動filebeat，啓動filebeat的時候用戶需要用filebeat用戶或者root用戶
./filebeat -e -c test.yml

#測試
啓動好後輸入任意字符串，如hello，即可輸出對應信息。

#啓動參數說明：./filebeat -e -c test.yml
-e：輸出到標準輸出，默認輸出到syslog和logs下
-c：指定配置文件

②收集日誌文件

cd /usr/local/filebeat/
cat > test.yml << END
filebeat.inputs:
- type: log
  enabled: true
  paths:
   - /var/log/*.log
   - /var/log/messages
  exclude_lines: ['^DBG',"^$",".gz$"]
setup.template.settings:
  index.number_of_shards: 3
output.console:
  pretty: true
  enable: true
END

#啓動filebeat
./filebeat -e -c test.yml

③自定義字段收集日誌文件

cd /usr/local/filebeat/
cat > test.yml << END
filebeat.inputs:
- type: log
  enabled: true
  paths:
   - /var/log/*.log
   - /var/log/messages
  exclude_lines: ['^DBG',"^$",".gz$"]
  tags: ["web","item"]       #自定義tags
  fields:                    #添加自定義字段
    from: itcast_from        #值隨便寫
  fields_under_root: true    #true爲添加到根節點中，false爲添加到子節點中
setup.template.settings:
  index.number_of_shards: 3
output.console:
  pretty: true
  enable: true
END

#啓動filebeat
./filebeat -e -c test.yml

#如果有tags字段在logstash中的書寫格式
if "web" in [tags] {  }

④收集nginx日誌文件輸出到ES或者logstash中

cd /usr/local/filebeat/
cat > nginx.yml << END
filebeat.inputs:
- type: log
  enabled: true
  paths:
   - /usr/local/nginx/access/*.log
  exclude_lines: ['^DBG',"^$",".gz$"]
  document_type: filebeat-nginx_accesslog
  tags: ["web","nginx"] 
  fields:
    from: nginx 
  fields_under_root: true 
setup.template.settings:
  index.number_of_shards: 3
output.elasticsearch:
  hosts: ["192.168.0.117:9200","192.168.0.118:9200","192.168.0.119:9200"]
#output.logstash:
#  hosts: ["192.168.0.117:5044"]
END

#啓動filebeat
./filebeat -e -c nginx.yml

三、Filebeat收集各個日誌到logstash，然後由logstash將日誌寫到redis，然後再寫入到ES

1、filebeat配置文件

cat > dashboard.yml << END
filebeat.inputs:
- input_type: log
  paths:
    - /var/log/*.log
    - /var/log/messages
  exclude_lines: ['^DBG',"^$",".gz$"]
  document_type: filebeat-systemlog
- input_type: log
  paths:
    - /usr/local/tomcat/logs/tomcat_access_log.*.log
  exclude_lines: ['^DBG',"^$",".gz$"]
  document_type: filebeat-tomcat-accesslog
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

- type: log
  enabled: true
  paths:
   - /usr/local/nginx/access/*.log
  exclude_lines: ['^DBG',"^$",".gz$"]
  document_type: filebeat-nginx-accesslog
output.logstash:
  hosts: ["192.168.0.117:5044"]
  enabled: true
  worker: 3
  compression_level: 3

END

##啓動
./filebeat -e -c dashboard.yml

2、logstash配置文件
①將beats收集的日誌寫入到logstash中

cat > beats.conf << END
input {
  beats {
    port => "5044"
    #host => "192.168.0.117"
  }
}
output {
  if [type] == "filebeat-systemlog" {
    redis {
      data_type => "list"
      host => "192.168.0.119"
      db => "3"
      port => "6379"
      password => "123456"
      key => "filebeat-systemlog"
    }
  }
  if [type] == "filebeat-tomcat-accesslog" {
    redis {
      data_type => "list"
      host => "192.168.0.119"
      db => "3"
      port => "6379"
      password => "123456"
      key => "filebeat-tomcat-accesslog"
    }
  }
  if [type] == "filebeat-nginx-accesslog" {
    redis {
      data_type => "list"
      host => "192.168.0.119"
      db => "3"
      port => "6379"
      password => "123456"
      key => "filebeat-nginx-accesslog"
    }
  }
}
END

②從redis中讀取日誌寫入ES

cat > redis-es.conf << END
input {
  redis {
    data_type => "list"
    host => "192.168.0.119"
    db => "3"
    port => "6379"
    password => "123456"
    key => "filebeat-systemlog"
    type => "filebeat-systemlog"
  }
  redis {
    data_type => "list"
    host => "192.168.0.119"
    db => "3"
    port => "6379"
    password => "123456"
    key => "filebeat-tomcat-accesslog"
    type => "filebeat-tomcat-accesslog"
  }
  redis {
    data_type => "list"
    host => "192.168.0.119"
    db => "3"
    port => "6379"
    password => "123456"
    key => "filebeat-nginx-accesslog"
    type => "filebeat-nginx-accesslog"
  }
}
 
output {
  if [type] == "filebeat-systemlog" {
    elasticsearch {
      hosts => ["192.168.0.117:9200","192.168.0.118:9200","192.168.0.119:9200"]
      index => "logstash-systemlog-%{+YYYY.MM.dd}"
    }
  }
  if [type] == "filebeat-tomcat-accesslog" {
    elasticsearch {
      hosts => ["192.168.0.117:9200","192.168.0.118:9200","192.168.0.119:9200"]
      index => "logstash-tomcat-accesslog-%{+YYYY.MM.dd}"
    }
  }
  if [type] == "filebeat-nginx-accesslog" {
    elasticsearch {
      hosts => ["192.168.0.117:9200","192.168.0.118:9200","192.168.0.119:9200"]
      index => "logstash-nginx-accesslog-%{+YYYY.MM.dd}"
    }
  }
}
END

備註：使用negate: true和match: after設置來指定任何不符合指定模式的行都屬於上一行。更多多行匹配配置請參考

四、通過kibana展示即可