logstash的功能有一點是把 各種軟件生成的各種格式的日誌 轉換成一個方便檢索篩選的格式，本文演示了一個最簡單的例子。
一 轉換的效果
實例： rabbitmq-server 日誌：
=INFO REPORT==== 16-Jan-2017::09:27:09 ===
Mirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0>
轉換後的格式爲：
{
"year" => "2017",
"mounthday" => "16",
"logdata" => "Mirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0>",
"message" => "=INFO REPORT==== 16-Jan-2017::09:27:09 ===\nMirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0>",
"type" => "rabbit",
"tags" => [
[0] "multiline"
],
"path" => "/var/log/rabbitmq/[email protected]",
"@timestamp" => 2017-01-16T01:27:09.718Z,
"loglevel" => "INFO",
"@version" => "1",
"host" => "server-31",
"time" => "09:27:09",
"mounth" => "Jan"
}
轉換後的內容傳入elasticsearch中，用戶就可以按照時間、日誌等級、主機等對彙總的日誌進行篩選檢索
二 轉換的過程
還是以剛纔那條日誌爲例
=INFO REPORT==== 16-Jan-2017::09:27:09 ===
Mirrored queue 'reply_963a14cce15f48e786240aad41817847' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2262.0>
=INFO REPORT==== 16-Jan-2017::09:27:09 ===
Mirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0>
=INFO REPORT==== 16-Jan-2017::09:27:09 ===
Mirrored queue 'q-agent-notifier-network-update' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2270.0>
日誌是多行，前後各有一行空行，日誌行以=開頭，
1、多行合併
首先是合併多行，
安裝多行插件：
/usr/share/logstash/bin/logstash-plugin install logstash-filter-multiline
在配置文件中配置多行合併
codec => multiline {
pattern => "^="
what => "previous"
negate => true
}
最終日誌轉換爲 =INFO REPORT==== 16-Jan-2017::09:27:09 ===\nMirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0>
2、分析日誌的格式和規律
結合所有的rabbitmq的日誌總結規律爲
=“日誌級別” REPORT==== "日期"::“時間” ===\n“日誌內容”
注意不要忘記中間的空格
3、正則匹配
logstash內置了很多常規正則，參見
https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns
本文都是採用內置的正則
=INFO REPORT==== 16-Jan-2017::09:27:09 ===\nMirrored queue 'heat-engine-listener.e9e416bb-6733-4981-bf00-bd64c104ccad' in vhost '/': Adding mirror on node 'rabbit@server-31': <0.2266.0>
我最終的匹配的表達式爲
^=%{LOGLEVEL:loglevel} REPORT=+ %{MONTHDAY:mounthday}-%{MONTH:mounth}-%{YEAR:year}::%{TIME:time} ===\n%{GREEDYDATA:logdata}$
%{LOGLEVEL:loglevel}表示這是一個變量，裏面的內容要匹配logstash內置的LOGLEVE正則，並且裏面的內容和loglevel這個key形成一對kv值："loglevel":"INFO"
其他一直類推
logstash提供了一個測試表達式的網址http://grokdebug.herokuapp.com/