filebeat 收集mysql、nginx、tomcat、docker日誌

1、filebeat自定義索引名稱
2、filebeat收集tomcatjson格式日誌
3、filebeat收集mysql慢日誌查詢
4、filebeat模塊方式收集nginxmain格式日誌
5、filebeat收集java多行匹配模式

filebeat自定義索引名稱
1、理想中的索引名稱

nginx-6.6.0-2019.11.15

2、filebeat配置

cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF 

filebeat按照服務類型拆分索引
1、第一種寫法

cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        source: "/var/log/nginx/access.log"
    - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        source: "/var/log/nginx/error.log"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF 

2、第二種寫法

cat >/etc/filebeat/filebeat.yml<<EOF 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "access"
    - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "error"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

filebeat收集tomcat的json日誌
1.安裝tomcat

 yum install tomcat tomcat-webapps

2、配置tomcat日誌格式爲json

[root@web01 ~]# /opt/tomcat/bin/shutdown.sh
[root@web01 ~]# sed -n '162p' /opt/tomcat/conf/server.xml 
	       pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u","AccessTime":"%t","method":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}"/>

3、啓動tomcat

/opt/tomcat/bin/startup.sh

4、配置filebeat

cat >/etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /opt/tomcat/logs/localhost_access_log.*.txt
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["tomcat"]

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "tomcat_access-%{[beat.version]}-%{+yyyy.MM}"

setup.template.name: "tomcat"
setup.template.pattern: "tomcat_*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

5、重啓filebeat

systemctl restart filebeat

6、訪問tomcat查看是否有數據生成

filebeat收集java多行匹配模式
1、filebeat配置文件

[root@db01 ~]# cat /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/elasticsearch/elasticsearch.log
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "es-%{[beat.version]}-%{+yyyy.MM}"
setup.template.name: "es"
setup.template.pattern: "es-*"
setup.template.enabled: false
setup.template.overwrite: true

filebeat使用模塊收集nginx日誌
1、刪除以前的ES索引和kibana索引
2、確認nginx日誌是否爲普通格式

systemctl stop nginx 
rm -rf /var/log/nginx/* 
自己修改日誌格式爲main的普通格式
systemctl start nginx

3、安裝nginx模塊所需的插件

cd /usr/share/elasticsearch/
./bin/elasticsearch-plugin install file:///root/ingest-geoip-6.6.0.zip 
./bin/elasticsearch-plugin install file:///root/ingest-user-agent-6.6.0.zip
systemctl restart elasticsearch

4、檢查filebeat配置文件是否包含模塊相關參數

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true 
  reload.period: 10s

5、激活filebeat模塊並查看

filebeat modules list 
filebeat enable nginx 

6、配置filebeat的nginx模塊

[root@web01 ~]# cat /etc/filebeat/modules.d/nginx.yml 
- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/*.log"]

  error:
    enabled: true
    var.paths: ["/var/log/nginx/error.log"]

7、filebeat配置

cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true 
  reload.period: 10s

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
  - index: "nginx-www-%{[beat.version]}-%{+yyyy.MM}"
    when.contains:
      source: "/var/log/nginx/www.log"
  - index: "nginx-blog-%{[beat.version]}-%{+yyyy.MM}"
    when.contains:
      source: "/var/log/nginx/blog.log"
  - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
    when.contains:
      source: "/var/log/nginx/error.log"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

8、重啓filebeat

systemctl restart filebeat

9、訪問nginx生成測試日誌

filebeat使用模塊收集mysql慢日誌
1、配置mysql錯誤日誌和慢日誌路徑

編輯my.cnf
[mysqld]
slow_query_log=ON
slow_query_log_file=/var/log/mariadb/slow.log
long_query_time=1

2、重啓mysql並製造慢日誌

systemctl restart mysql 
慢日誌製造語句
select sleep(2) user,host from mysql.user ;

3、確認慢日誌和錯誤日誌確實有生成

mysql -uroot -poldboy123 -e "show variables like '%slow_query_log%'"

4、激活filebeat的mysql模塊

filebeat module enable mysql

5、配置mysql模塊

module: mysql
error:
  enabled: true
  var.paths: ["/var/log/mariadb/mariadb.log"]

slowlog:
  enabled: true 
  var.paths: ["/var/log/mariadb/slow.log"]

6、配置filebeat根據日誌類型做判斷

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "mysql_slowlog-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        fileset.module: "mysql"
        fileset.name: "slowlog"
    - index: "mysql_error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        fileset.module: "mysql"
        fileset.name: "error"

setup.template.name: "mysql"
setup.template.pattern: "mysql_*"
setup.template.enabled: false
setup.template.overwrite: true

7、重啓filebeat

systemctl restart filebeat

filebeat收集docker日誌
1、docker安裝命令

wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.ustc.edu.cn/docker-ce/linux/centos/docker-ce.repo
sed -i 's#download.docker.com#mirrors.tuna.tsinghua.edu.cn/docker-ce#g' /etc/yum.repos.d/docker-ce.repo
yum install docker-ce -y
systemctl start docker

2、啓動2個nginx容器

systemctl stop nginx
pkill java

docker run -d -p 80:80 nginx
docker run -d -p 8080:80 nginx

3、查看容器日誌

docker logs -f ce22c2583da5

4、修改filebeat配置文件

cat >>/etc/filebeat/filebeat.yml<<EOF 
filebeat.inputs:
- type: docker
  containers.ids: 
    - '*'

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "docker-%{[beat.version]}-%{+yyyy.MM}"

setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

5、重啓filebeat

systemctl restart filebeat

6、訪問nginx製造日誌

curl 127.0.0.1/11111111111111111111
curl 127.0.0.1:8080/22222222222222222222

filebeat收集docker日誌V2
1、理想中的索引

docker-mysql-xxxx
docker-nginx-xxxx

2、理想中的日誌格式

{
	"log": "10.0.0.1 - - [18/Nov/2019:02:16:44 +0000] \"GET /web01 HTTP/1.1\" 404 555 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36\" \"-\"\n",
	"stream": "stdout",
	"time": "2019-11-18T02:16:44.010910131Z",
	"service": "nginx"
}

3、安裝docker-compose

yum install docker-compose -y

4、編寫docker-compose文件

cat >docker-compose.yml<<EOF
version: '3'
services:
  nginx:
    image: nginx:latest
    labels:
      service: nginx
    logging:
      options:
        labels: "service"
    ports:
      - "80:80"
  db:
    image: nginx:latest
    labels:
      service: db 
    logging:
      options:
        labels: "service"
    ports:
      - "8080:80"
EOF

5、刪除舊容器

docker stop $(docker ps -q)
docker rm $(docker ps -qa)

6、啓動docker-compose

docker-compose up -d

7、修改filebeat配置文件

cat >/etc/filebeat/filebeat.yml<<EOF 
filebeat.inputs:
- type: log 
  enabled: true
  paths:
    - /var/lib/docker/containers/*/*-json.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "docker-nginx-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "nginx"
    - index: "docker-db-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "db"

setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite
EOF

8、重啓filebeat

systemctl restart filebeat

9、生成測試命令

curl 127.0.0.1/nginxxxxxxxx
curl 127.0.0.1:8080/dbbbbbbbbbbbbb

filebeat收集docker日誌V3

1、分析正常日誌和錯誤日誌字段的區別

錯誤日誌字段： stream：stderr
正常日誌字段： stream：stdout

2、修改filebeat配置文件

cat >/etc/filebeat/filebeat.yml<EOF
filebeat.inputs:
- type: log 
  enabled: true
  paths:
    - /var/lib/docker/containers/*/*-json.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "docker-nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        stream: "stdout"
        attrs.service: "nginx"
    - index: "docker-nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        stream: "stderr"
        attrs.service: "nginx"
    - index: "docker-db-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        stream: "stdout"
        attrs.service: "db"
    - index: "docker-db-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        stream: "stderr"
        attrs.service: "db"

setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

3、重啓filebeat

systemctl restart filebeat

4、生成測試命令

curl 127.0.0.1/nginxxxxxxxx
curl 127.0.0.1:8080/dbbbbbbbbbbbbb

filebeat收集docker最終版
1、創建容器日誌目錄

mkdir /opt/{nginx,mysql}

2、將容器的日誌目錄掛載到宿主機

docker ps 
docker cp 容器ID:/etc/nginx/nginx.conf .
修改nginx配置文件裏的日誌記錄類型爲json格式 
docker cp /etc/nginx/nginx.conf 容器ID:/etc/nginx/nginx.conf
docker commit 容器ID nginx:v2
docker-compose stop
docker rm -f  $(docker ps -a -q)
docker run -d -p 80:80 -v /opt/nginx:/var/log/nginx nginx:v2
docker run -d -p 8080:80 -v /opt/mysql:/var/log/nginx nginx:v2

2、修改filebeat配置文件

cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /opt/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["nginx_access"]

- type: log
  enabled: true
  paths:
    - /opt/nginx/error.log
  tags: ["nginx_error"]

- type: log
  enabled: true
  paths:
    - /opt/mysql/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["mysql_access"]

- type: log
  enabled: true
  paths:
    - /opt/mysql/error.log
  tags: ["mysql_error"]

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "nginx_access"
    - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "nginx_error"
    - index: "mysql-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "mysql_access"
    - index: "mysql-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "mysql_error"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

3、重啓filebeat

systemctl restart filebeat

4、生成測試命令

curl 127.0.0.1/nginxxxxxxxx
curl 127.0.0.1:8080/dbbbbbbbbbbbbb