微服務框架（二十九）Logstash Nginx 日誌上報

此係列文章將會描述Java框架Spring Boot、服務治理框架Dubbo、應用容器引擎Docker，及使用Spring Boot集成Dubbo、Mybatis等開源框架，其中穿插着Spring Boot中日誌切面等技術的實現，然後通過gitlab-CI以持續集成爲Docker鏡像。

本文爲Logstash Nginx 日誌上報

本系列文章中所使用的框架版本爲Spring Boot 2.0.3-RELEASE，Spring 5.0.7-RELEASE，Dubbo 2.6.2。

Nginx 日誌上報

Nginx 日誌上報採用filebeat，需在服務器安裝相關版本的filebeat並配置日誌路徑即可

filebeat安裝文檔

Nginx 日誌格式

Nginx access log日誌格式

log_format  access  '$proxy_add_x_forwarded_for - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" [ $request_body ] - "$request_time" - "$host"';

Logstash 日誌處理

相關插件詳見微服務框架（二十八）Logstash 使用文檔

Nginx 日誌索引爲nginx-*，該索引將自動適配es中自定義的索引模板

input{
  beats {
    host => "172.16.7.7"
    port => "5044"
    add_field => ["log_channel", "nginx"]
  }
}

filter {
  if [log_channel] == "nginx" {
    grok {
      match => {
         "message" => "%{DATA:forwarded_for} - %{DATA:remote_user} \[%{DATA:request_time}\] \"%{WORD:method} %{URIPATHPARAM:uri} %{DATA:http_version}\" %{INT:http_status} %{INT:body_byte} \"%{PROG:http_referer}\" \"%{DATA:user_agent}\" \[ %{DATA:request_body}\ ] - \"%{BASE10NUM:spend_time}\" - \"%{DATA:request_host}\""
      }
    }

    mutate {
      convert => {
        "http_status" => "integer"
        "spend_time" => "float"
        "body_byte" => "integer"
      }
    }

    mutate {
      split => ["forwarded_for",","]
      add_field => ["request_ip", "%{[forwarded_for][0]}"]
      remove_field => ["tags", "forwarded_for"]
    }

    geoip {
      source => "request_ip"
      remove_field => ["tags", "[geoip][latitude]", "[geoip][longitude]", "[geoip][continent_code]", "[geoip][country_code3]", "[geoip][country_code2]"]
    }

    mutate {
        add_field => {
            "[@metadata][index_prefix]" => "nginx"
        }
    }

  }
}

output {
  elasticsearch {
      hosts => ["localhost:9200"]
      index => "%{[@metadata][index_prefix]}-%{+YYYY.MM.dd}"
  }
}