收集tomcat access訪問日誌

採用filebeat收集，logstash過濾，es索引存儲的架構。
配置含義參考本文集其他文章。

一、Filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/tomcat/apache-tomcat-8.5.39/logs/localhost_access_log.*.txt
  tags: ["tomcat_access"]

output.logstash:
  hosts: ["xx.xx.xx.xx:5044"]  #logstash的端口，默認5044，可以修改

二、logstash

1 input.conf

input {
  #tomcat access log
  beats {
    port => "5044"
  }
}

2 tomcat_out.conf

#過濾器
filter {
    if "tomcat_access" in [tags] {
        grok {
            match => ["message", "%{IPORHOST:client_ip} (%{USER:ident}|-) (%{USER:auth}|-) \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:http_version})?|-)\" %{NUMBER:response} (%{NUMBER:bytes}|-)" ]
            remove_field => "message"
        }
        date {  #時間格式轉換插件，將過濾字段@timestamp的時間重置爲日誌時間
            match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
            target => "@timestamp"
            remove_field => "timestamp"
        }
    }
}

output {
        #此配置爲輸出到控制檯，調試時可以加，正式環境就可以去掉
    stdout {codec => rubydebug } 

    elasticsearch {
        hosts  => ["192.168.18.2:9200"]
        index => "web-tomcat-%{+YYYY.MM.dd}"
    }
}

如果是和system模塊日誌一起收，可以這麼寫

filter {
    if "tomcat_access" in [tags] {
        grok {
            match => ["message", "%{IPORHOST:client_ip} (%{USER:ident}|-) (%{USER:auth}|-) \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:http_version})?|-)\" %{NUMBER:response} (%{NUMBER:bytes}|-)" ]
            remove_field => "message"
        }
        date {
            match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
            target => "@timestamp"
            remove_field => "timestamp"
        }
    }
}


output {

    if "tomcat_access" in [tags] {
        elasticsearch {
            hosts => ["192.168.18.2:9200"]
            index => "web-tomcat-%{+YYYY.MM.dd}"
        }
    }

    if [event][module] == "system" {
        elasticsearch {
            hosts => ["192.168.18.2:9200"]
            index => "os-linux-%{+YYYY.MM.dd}"
        }
    }

}

三、tomcat access源日誌

16.128.231.178 - - [11/Aug/2020:10:56:36 +0800] "GET /testcloud/getHouseholdRegister?idcard=512913560570 HTTP/1.1" 200 64
13.144.212.111 - - [11/Aug/2020:10:56:36 +0800] "GET /testcloud/gateway/Register?idcard=45135311007X HTTP/1.1" 200 342