瞭解一下Logstash常用配置

Logstash管道可以配置一個或多個輸入插件、過濾器插件和輸出插件。其中，輸入插件和輸出插件是必選的，過濾器插件是可選的。下圖是Logstash常見的使用場景。

上一節的例子中我們使用標準的輸入和輸出插件做了簡單的示例。接下來我們演示一些複雜的場景。如下圖所示是Logstash的標準管道結構，我們通過一些高級配置來完成Apache日誌的過濾。

# The # character at the beginning of a line indicates a comment. 

Use# comments to describe your configuration. 

input { 

}

# The filter part of this file is commented out to indicate that it is# optional. 

# filter { 

# 

# } 

output { 

}

1. 準備一段apache日誌文件，格式如下：

83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1" 200 171717 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

83.149.9.216 - - [04/Jan/2015:05:13:44 +0000] "GET /presentations/logstash-monitorama-2013/plugin/highlight/highlight.js HTTP/1.1" 200 26185 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

83.149.9.216 - - [04/Jan/2015:05:13:44 +0000] "GET /presentations/logstash-monitorama-2013/plugin/zoom-js/zoom.js HTTP/1.1" 200 7697 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

2. 編寫Logstash管道配置文件，放在Logstash/bin目錄下

input {
    file {
        path => "/opt/cx/logstash/apache-log.log"
        start_position => beginning
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    elasticsearch {}
    stdout {}
}

3. 校驗配置文件是否正確

[root@Server01 bin]# ./logstash -f apache-log-pipeline.conf --configtest
Configuration OK

4.啓動Logstash 

[root@Server05 bin]#  ./logstash -f apache-log-pipeline.conf 

Settings: Default pipeline workers: 4

Pipeline main started

5.完整的Logstash配置文件如下

input {
    file {
        path => "/opt/cx/logstash/apache-log.log"
        start_position => beginning
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    elasticsearch {
		hosts=>["10.0.10.5:9200"]
	}
    stdout {}
}