filebeat 收集mysql、nginx、tomcat、docker日志

1、filebeat自定义索引名称
2、filebeat收集tomcatjson格式日志
3、filebeat收集mysql慢日志查询
4、filebeat模块方式收集nginxmain格式日志
5、filebeat收集java多行匹配模式

filebeat自定义索引名称
1、理想中的索引名称

nginx-6.6.0-2019.11.15

2、filebeat配置

cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF 

filebeat按照服务类型拆分索引
1、第一种写法

cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        source: "/var/log/nginx/access.log"
    - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        source: "/var/log/nginx/error.log"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF 

2、第二种写法

cat >/etc/filebeat/filebeat.yml<<EOF 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "access"
    - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "error"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

filebeat收集tomcat的json日志
1.安装tomcat

 yum install tomcat tomcat-webapps

2、配置tomcat日志格式为json

[root@web01 ~]# /opt/tomcat/bin/shutdown.sh
[root@web01 ~]# sed -n '162p' /opt/tomcat/conf/server.xml 
	       pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u","AccessTime":"%t","method":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}"/>

3、启动tomcat

/opt/tomcat/bin/startup.sh

4、配置filebeat

cat >/etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /opt/tomcat/logs/localhost_access_log.*.txt
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["tomcat"]

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "tomcat_access-%{[beat.version]}-%{+yyyy.MM}"

setup.template.name: "tomcat"
setup.template.pattern: "tomcat_*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

5、重启filebeat

systemctl restart filebeat

6、访问tomcat查看是否有数据生成

filebeat收集java多行匹配模式
1、filebeat配置文件

[root@db01 ~]# cat /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/elasticsearch/elasticsearch.log
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "es-%{[beat.version]}-%{+yyyy.MM}"
setup.template.name: "es"
setup.template.pattern: "es-*"
setup.template.enabled: false
setup.template.overwrite: true

filebeat使用模块收集nginx日志
1、删除以前的ES索引和kibana索引
2、确认nginx日志是否为普通格式

systemctl stop nginx 
rm -rf /var/log/nginx/* 
自己修改日志格式为main的普通格式
systemctl start nginx

3、安装nginx模块所需的插件

cd /usr/share/elasticsearch/
./bin/elasticsearch-plugin install file:///root/ingest-geoip-6.6.0.zip 
./bin/elasticsearch-plugin install file:///root/ingest-user-agent-6.6.0.zip
systemctl restart elasticsearch

4、检查filebeat配置文件是否包含模块相关参数

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true 
  reload.period: 10s

5、激活filebeat模块并查看

filebeat modules list 
filebeat enable nginx 

6、配置filebeat的nginx模块

[root@web01 ~]# cat /etc/filebeat/modules.d/nginx.yml 
- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/*.log"]

  error:
    enabled: true
    var.paths: ["/var/log/nginx/error.log"]

7、filebeat配置

cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true 
  reload.period: 10s

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
  - index: "nginx-www-%{[beat.version]}-%{+yyyy.MM}"
    when.contains:
      source: "/var/log/nginx/www.log"
  - index: "nginx-blog-%{[beat.version]}-%{+yyyy.MM}"
    when.contains:
      source: "/var/log/nginx/blog.log"
  - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
    when.contains:
      source: "/var/log/nginx/error.log"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

8、重启filebeat

systemctl restart filebeat

9、访问nginx生成测试日志

filebeat使用模块收集mysql慢日志
1、配置mysql错误日志和慢日志路径

编辑my.cnf
[mysqld]
slow_query_log=ON
slow_query_log_file=/var/log/mariadb/slow.log
long_query_time=1

2、重启mysql并制造慢日志

systemctl restart mysql 
慢日志制造语句
select sleep(2) user,host from mysql.user ;

3、确认慢日志和错误日志确实有生成

mysql -uroot -poldboy123 -e "show variables like '%slow_query_log%'"

4、激活filebeat的mysql模块

filebeat module enable mysql

5、配置mysql模块

module: mysql
error:
  enabled: true
  var.paths: ["/var/log/mariadb/mariadb.log"]

slowlog:
  enabled: true 
  var.paths: ["/var/log/mariadb/slow.log"]

6、配置filebeat根据日志类型做判断

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "mysql_slowlog-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        fileset.module: "mysql"
        fileset.name: "slowlog"
    - index: "mysql_error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        fileset.module: "mysql"
        fileset.name: "error"

setup.template.name: "mysql"
setup.template.pattern: "mysql_*"
setup.template.enabled: false
setup.template.overwrite: true

7、重启filebeat

systemctl restart filebeat

filebeat收集docker日志
1、docker安装命令

wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.ustc.edu.cn/docker-ce/linux/centos/docker-ce.repo
sed -i 's#download.docker.com#mirrors.tuna.tsinghua.edu.cn/docker-ce#g' /etc/yum.repos.d/docker-ce.repo
yum install docker-ce -y
systemctl start docker

2、启动2个nginx容器

systemctl stop nginx
pkill java

docker run -d -p 80:80 nginx
docker run -d -p 8080:80 nginx

3、查看容器日志

docker logs -f ce22c2583da5

4、修改filebeat配置文件

cat >>/etc/filebeat/filebeat.yml<<EOF 
filebeat.inputs:
- type: docker
  containers.ids: 
    - '*'

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "docker-%{[beat.version]}-%{+yyyy.MM}"

setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

5、重启filebeat

systemctl restart filebeat

6、访问nginx制造日志

curl 127.0.0.1/11111111111111111111
curl 127.0.0.1:8080/22222222222222222222

filebeat收集docker日志V2
1、理想中的索引

docker-mysql-xxxx
docker-nginx-xxxx

2、理想中的日志格式

{
	"log": "10.0.0.1 - - [18/Nov/2019:02:16:44 +0000] \"GET /web01 HTTP/1.1\" 404 555 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36\" \"-\"\n",
	"stream": "stdout",
	"time": "2019-11-18T02:16:44.010910131Z",
	"service": "nginx"
}

3、安装docker-compose

yum install docker-compose -y

4、编写docker-compose文件

cat >docker-compose.yml<<EOF
version: '3'
services:
  nginx:
    image: nginx:latest
    labels:
      service: nginx
    logging:
      options:
        labels: "service"
    ports:
      - "80:80"
  db:
    image: nginx:latest
    labels:
      service: db 
    logging:
      options:
        labels: "service"
    ports:
      - "8080:80"
EOF

5、删除旧容器

docker stop $(docker ps -q)
docker rm $(docker ps -qa)

6、启动docker-compose

docker-compose up -d

7、修改filebeat配置文件

cat >/etc/filebeat/filebeat.yml<<EOF 
filebeat.inputs:
- type: log 
  enabled: true
  paths:
    - /var/lib/docker/containers/*/*-json.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "docker-nginx-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "nginx"
    - index: "docker-db-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "db"

setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite
EOF

8、重启filebeat

systemctl restart filebeat

9、生成测试命令

curl 127.0.0.1/nginxxxxxxxx
curl 127.0.0.1:8080/dbbbbbbbbbbbbb

filebeat收集docker日志V3

1、分析正常日志和错误日志字段的区别

错误日志字段： stream：stderr
正常日志字段： stream：stdout

2、修改filebeat配置文件

cat >/etc/filebeat/filebeat.yml<EOF
filebeat.inputs:
- type: log 
  enabled: true
  paths:
    - /var/lib/docker/containers/*/*-json.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "docker-nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        stream: "stdout"
        attrs.service: "nginx"
    - index: "docker-nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        stream: "stderr"
        attrs.service: "nginx"
    - index: "docker-db-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        stream: "stdout"
        attrs.service: "db"
    - index: "docker-db-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        stream: "stderr"
        attrs.service: "db"

setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

3、重启filebeat

systemctl restart filebeat

4、生成测试命令

curl 127.0.0.1/nginxxxxxxxx
curl 127.0.0.1:8080/dbbbbbbbbbbbbb

filebeat收集docker最终版
1、创建容器日志目录

mkdir /opt/{nginx,mysql}

2、将容器的日志目录挂载到宿主机

docker ps 
docker cp 容器ID:/etc/nginx/nginx.conf .
修改nginx配置文件里的日志记录类型为json格式 
docker cp /etc/nginx/nginx.conf 容器ID:/etc/nginx/nginx.conf
docker commit 容器ID nginx:v2
docker-compose stop
docker rm -f  $(docker ps -a -q)
docker run -d -p 80:80 -v /opt/nginx:/var/log/nginx nginx:v2
docker run -d -p 8080:80 -v /opt/mysql:/var/log/nginx nginx:v2

2、修改filebeat配置文件

cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /opt/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["nginx_access"]

- type: log
  enabled: true
  paths:
    - /opt/nginx/error.log
  tags: ["nginx_error"]

- type: log
  enabled: true
  paths:
    - /opt/mysql/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["mysql_access"]

- type: log
  enabled: true
  paths:
    - /opt/mysql/error.log
  tags: ["mysql_error"]

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "nginx_access"
    - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "nginx_error"
    - index: "mysql-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "mysql_access"
    - index: "mysql-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "mysql_error"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

3、重启filebeat

systemctl restart filebeat

4、生成测试命令

curl 127.0.0.1/nginxxxxxxxx
curl 127.0.0.1:8080/dbbbbbbbbbbbbb