使用Logstash multiline 收集PHP、tomcat等應用服務多行堆棧日誌

很多時候應用程序出錯是都是拋出一堆 堆棧信息（即在日誌文件輸出多行），此時logstash可以使用multiline的插件收集日誌時需要把錯誤堆棧信息收集爲一個記錄。multiline字面意思是多行,顧名思義就是對多行日誌進行處理。

multiline配置與用法

input {
  stdin {
    codec => multiline {
      pattern => "pattern, a regexp"  
      negate => "true" or "false"  
      what => "previous" or "next"  
    }
  }
}

## pattern支持正則表達式，通過正則表達式匹配日誌信息，grok pattern定義的參數也是支持，Eg：%{TIMESTAMP_ISO8601} ，具體信息可以查看grok-patterns

## negate  只支持布爾值，true 或者false，默認爲false。如果設置爲true，表示信息不跟上面正則表達式（pattern）匹配的內容都與匹配的整合，具體整合在前還是在後，看what參數。如果設置爲false，即與pattern匹配的內容

## what  前一行 或者後一行，指出上面對應的規則與前一行內容收集爲一行，還是與後一行整合在一起

 

官方原文

The pattern should match what you believe to be an indicator that the field is part of a multi-line event.

The what must be previous or next and indicates the relation to the multi-line event.

The negate can be true or false (defaults to false). If true, a message not matching the pattern will constitute a match of the multiline filter and the what will be applied. (vice-versa is also true)

 

在這裏用PHP-FPM的慢查詢日誌做個測試

PHP-FPM的慢查詢日誌如下：

[11-Mar-2015 16:54:17]  [pool www] pid 12873
script_filename = /data//index.php
[0x00007f497fa5b620] curl_exec() /data//Account.php:221
[0x00007f497fa5a4e0] call() /data/gintama_app/jidong/game_code/app/controllers/Game.php:31
[0x00007fff29eea180] load() unknown:0
[0x00007f497fa59e18] call_user_func_array() /data/library/BaseCtrl.php:20
[0x00007fff29eea470] handoutAction() unknown:0
[0x00007f497fa59400] run() /data//index.php:30
  
[11-Mar-2015 16:56:46]  [pool www] pid 12881
script_filename = /data/index.php
[0x00007f497fa5b620] curl_exec() /data//Account.php:221
[0x00007f497fa5a4e0] call() /data/Game.php:31
[0x00007fff29eea180] load() unknown:0
[0x00007f497fa59e18] call_user_func_array() /data/library/BaseCtrl.php:20
[0x00007fff29eea470] handoutAction() unknown:0
[0x00007f497fa59400] run() /data/index.php:30

添加Logstash的配置文件logstash_php-fpm.conf

input {
    file {
        path => "/tmp/php-slow.log"  ###收集的日誌文件路徑
        codec => multiline {  ###使用multiline
            pattern => "^(\[\d{2}-%{MONTH}-\d{4})"  ###使用正則表達式，%{MONTH}是在gork定義好的，這邊偷個懶，直接調用
            negate => true  ###設置爲true，即取正則表達式不匹配的行，然後將內容與上一行或者下一行整合
            what => "previous"  ###設置爲previous，表示與上一行內容整合在一起。
        }
    }
}

output{
   stdout { codec => rubydebug }
   elasticsearch{
        hosts => ["110.22.145.155:9200"]
        index => "logstash-php_%{+YYYY.MM.dd}"
   }
}

 

##測試配置文件是否符合語法

{logstash_home}/bin/logstash –t logstash_php-fpm.conf

# /opt/logstash/bin/logstash -t logstash-php_slow.conf 
Configuration OK

 

運行logstash，查看輸出內容

# /opt/logstash/bin/logstash -f ./logstash-php_slow.conf 
Settings: Default pipeline workers: 8
Pipeline main started

{
    "@timestamp" => "2017-07-17T05:40:40.310Z",
       "message" => "[11-Mar-2015 16:54:17]  [pool www] pid 12873\nscript_filename = /data//index.php\n[0x00007f497fa5b620] curl_exec() /data//Account.php:221\n[0x00007f497fa5a4e0] call() /data/gintama_app/jidong/game_code/app/controllers/Game.php:31\n[0x00007fff29eea180] load() unknown:0\n[0x00007f497fa59e18] call_user_func_array() /data/library/BaseCtrl.php:20\n[0x00007fff29eea470] handoutAction() unknown:0\n[0x00007f497fa59400] run() /data//index.php:30\n  ",
      "@version" => "1",
          "tags" => [
        [0] "multiline"
    ],
          "path" => "/tmp/php-slow.log",
          "host" => "test2-web"
}
{
    "@timestamp" => "2017-07-17T05:40:47.321Z",
       "message" => "[11-Mar-2015 16:56:46]  [pool www] pid 12881\nscript_filename = /data/index.php\n[0x00007f497fa5b620] curl_exec() /data//Account.php:221\n[0x00007f497fa5a4e0] call() /data/Game.php:31\n[0x00007fff29eea180] load() unknown:0\n[0x00007f497fa59e18] call_user_func_array() /data/library/BaseCtrl.php:20\n[0x00007fff29eea470] handoutAction() unknown:0\n[0x00007f497fa59400] run() /data/index.php:30",
      "@version" => "1",
          "tags" => [
        [0] "multiline"
    ],
          "path" => "/tmp/php-slow.log",
          "host" => "test2-web"
}

 

Tomcat堆棧信息收集也類似，找出規則，然後進行匹配即可，在此不重複測試