filebeat作为代理安装在服务器上,监视指定的日志文件或位置,收集日志事件,并将他们转发到logstash,elasticsearch,kafka等
input 我们要采集的日志文件路径, 收割机 harvester 监听文件的变化 -->
splooer程序 --> 转发 es | logstash | kafka | redis
filebeat.inputs:
- type: stdin #标准输入
enabled: true #启用
output.console: #标准输出
pretty: true
enable: true
将文件最新发生变化的内容,存入ES
[root@web01 ~]# cat /etc/filebeat/file.yml
filebeat.inputs:
- type: log
paths: /var/log/nginx/access.log
enabled: true
output.elasticsearch:
hosts:["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]
收集系统日志
特别分散–> syslog --> file.txt
1.减少无用的数据
2.调整索引名称
3.测试调整模板,设定分片
[root@web01 filebeat]# cat filebeat_system.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/oldxu.log
include_lines: ['^ERR','^WARN','sshd'] #只看指定的日志
output.elasticsearch:
hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]
index: "system-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.ilm.enabled: false
setup.template.name: "system"
setup.template.pattern: "system-*"
setup.template.settings: #定义索引分片数和副本
index.number_of_shards: 3
index.number_of_replicas: 1
1.修改system模板 ---> 添加 shards 分片数数量,replicas的数量
2.删除模板关联的索引
3.删除filebeat自行指定的分片数和副本数
4.重启filebeat
5.产生新的日志
收集Nginx日志
配置filebeat
[root@web01 filebeat]# cat filebeat_nginx.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true #默认Flase，还会将json解析的日志存储至messages字段
json.overwrite_keys: true #覆盖默认的key，使用自定义json格式的key
output.elasticsearch:
hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]
index: "nginx-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.ilm.enabled: false
setup.template.name: nginx #索引关联的模板名称
setup.template.pattern: nginx-*
收集nginx访问日志和错误日志
[root@web01 filebeat]# cat filebeat_access.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true #默认Flase，还会将json解析的日志存储至messages字段
json.overwrite_keys: true #覆盖默认的key，使用自定义json格式的key
tags: ["access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["error"]
output.elasticsearch:
hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]
indices:
- index: "nginx-access-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
tags: "access"
- index: "nginx-error-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
tags: "error"
setup.ilm.enabled: false
setup.template.name: nginx #索引关联的模板名称
setup.template.pattern: nginx-*
收集nginx多个虚拟主机的日志
1.虚拟主机
[root@web01 conf.d]# cat elk.oldxu.com.conf
server {
listen 80;
server_name elk.oldxu.com;
root /code/elk;
access_log /var/log/nginx/elk.oldxu.com.log json;
location / {
index index.html;
}
}
[root@web01 conf.d]# cat bk.oldxu.com.conf
server {
listen 80;
server_name bk.oldxu.com;
root /code/bk;
access_log /var/log/nginx/bk.oldxu.com.log json;
location / {
index index.html;
}
}
[root@web01 conf.d]# cat bs.oldxu.com.conf
server {
listen 80;
server_name bs.oldxu.com;
root /code/bs;
access_log /var/log/nginx/bs.oldxu.com.log json;
location / {
index index.html;
}
}
2.测试,模拟产生日志
[root@web01 conf.d]# curl -H Host:elk.oldxu.com http://10.0.0.7
elk.oldux.com
[root@web01 conf.d]# curl -H Host:bs.oldxu.com http://10.0.0.7
bs.oldux.com
[root@web01 conf.d]# curl -H Host:bk.oldxu.com http://10.0.0.7
bk.oldux.com
3.配置filebeat
[root@web01 filebeat]# cat filebeat-vhosts.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/elk.oldxu.com.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["nginx-elk-host"]
- type: log
enabled: true
paths:
- /var/log/nginx/bs.oldxu.com.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["nginx-bs-host"]
- type: log
enabled: true
paths:
- /var/log/nginx/bk.oldxu.com.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["nginx-bk-host"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["nginx-error"]
output.elasticsearch:
hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]
indices:
- index: "nginx-elk-access-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
tags: "nginx-elk-host"
- index: "nginx-bs-access-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
tags: "nginx-bs-host"
- index: "nginx-bk-access-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
tags: "nginx-bk-host"
- index: "nginx-error-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
tags: "nginx-error"
setup.ilm.enabled: false
setup.template.name: nginx #索引关联的模板名称
setup.template.pattern: nginx-*
Tomcat日志
访问日志 —> json格式
1.修改tomcat日志格式
[root@web02 soft]# yum install java -y
[root@web02 soft]# vim tomcat/conf/server.xml
<Host name="tomcat.oldxu.com" appBase="webapps"
unpackWARs="true" autoDeploy="true">
<Valve
className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
prefix="tomcat.oldxu.com.log"
suffix=".txt"
pattern="
{"clientip":"%h","ClientUser&q
uot;:"%l","authenticated":"%u&
quot;,"AccessTime":"%t","metho
d":"%r","status":"%s"
;,"SendBytes":"%b","Query?
string":"%q","partner":"%
{Referer}i","AgentVersion":"%{User�Agent}i"}" />
</Host>
配置filebeat
[root@web01 filebeat]# cat filebeat-tomcat-mutilline.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /soft/tomcat/logs/tomcat.oldxu.com.log.*.txt
json.keys_under_root: true #默认Flase，还会将json解析的日志存储至messages字段
json.overwrite_keys: true #覆盖默认的key，使用自定义json格式的key
tags: ["tomcat-access"]
- type: log
enabled: true
paths:
- /soft/tomcat/logs/catalina.out
multiline.pattern: '^\d{2}' #匹配以2个数字开头的
multiline.negate: true
multiline.match: after
multiline.max_lines: 10000 #默认最大合并行为500，可根据实际情况调整。
tags: ["tomcat-error"]
output.elasticsearch:
hosts: ["10.0.0.161:9200","10.0.0.162:9200"]
indices:
- index: "tomcat-access-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
tags: "tomcat-access"
- index: "tomcat-error-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
tags: "tomcat-error"
setup.ilm.enabled: false
setup.template.name: tomcat #索引关联的模板名称
setup.template.pattern: tomcat-*
参考链接 ：
day105 ELK-filbeate收集日志 ： https://www.jianshu.com/p/fffee6c527de