Kibana+Logstash+Elasticsearch+Redis安裝部署

   最近做日誌分析，發現logstash較符合自己的需求，

•  Logstash：做系統log收集，轉載的工具。同時集成各類日誌插件,對日誌查詢和分析的效率有很大的幫助.一般使用shipper作爲log收集、indexer作爲log轉載.  

• Logstash shipper收集log 並將log轉發給redis 存儲  

• Logstash indexer從redis中讀取數據並轉發給elasticsearch  

• redis：是一個db，logstash shipper將log轉發到redis數據庫中存儲。Logstash indexer從redis中讀取數據並轉發給elasticsearch。  

• Elasticsearch：elasticsearch是基於lucene的開源搜索引擎,用來做索引。

• Kibana： 開源web展現,界面很漂亮,是一個功能強大的elasticsearch數據顯示客戶端，logstash已經內置了kibana，你也可以單獨部署kibana，最新版的kibana3是純html+js客戶端.

軟件下載目錄

http://www.elasticsearch.org/downloads/

我的環境如下

os：centos6.3_x86-64

redis-2.8.7.tar.gz    

kibana-3.0.0

java version "1.7.0_51"

elasticsearch-0.90.12

一，安裝java

yum -y install java

二,安裝redis

cd ~/src
wget http://download.redis.io/releases/redis-2.8.7.tar.gz
tar -zxf redis-2.8.7.tar.gz
cd redis-2.8.7.tar.gz
make
sudo make install

安裝完畢後

/etc/init.d/redis_6379 start

測試是否正常

[root@file1 ~]# redis-cli ping
PONG
[root@file1 ~]#

[root@file1 ~]# netstat -tanpu|grep redis
tcp        0      0 0.0.0.0:6379                0.0.0.0:*                   LISTEN      1391/redis-server *

三，安裝Elasticsearch

cd /search
sudo mkdir elasticsearch
cd elasticsearch
sudo wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.12.zip
sudo unzip elasticsearch-0.90.12.zip

備註：當開始使用的是1.x.x java報錯，後來用的0.9.。

https://groups.google.com/forum/#!topic/logstash-users/fvFT7pgQTEM

Are you using elasticsearch_http for your output?  If not, 1.3.3 is based on 0.90.x elasticsearch, and won’t play nice with 1.0.x elasticsearch with just the “elasticsearch” output.

啓動ES服務器

切換到elasticsearch目錄運行

bin/elasticsearch -f

默認端口是9200

curl -X GET http://localhost:9200
[root@file1 ~]# curl -X GET http://localhost:9200
{
  "ok" : true,
  "status" : 200,
  "name" : "Master Pandemonium",
  "version" : {
    "number" : "0.90.12",
    "build_hash" : "26feed79983063ae83bfa11bd4ce214b1f45c884",
    "build_timestamp" : "2014-02-25T15:38:23Z",
    "build_snapshot" : false,
    "lucene_version" : "4.6"
  },
  "tagline" : "You Know, for Search"
}

四.安裝logstash

cd /search
sudo mkdir logstash
cd logstash
sudo wget http://download.elasticsearch.org/logstash/logstash/logstash-1.2.1-flatjar.jar

新建配置文件index.conf

# This is the logstash server index configuration.
# This file will be put in the same folder with logtash.jar file in the
# /etc/logtash/
# This takes information straight from redis and loads it into elasticsearch.
input {
  redis {
    host => "127.0.0.1"
    type => "syslog"
    threads => 4
    # these settings should match the output of the agent
    data_type => "list"
    key => "logstash"
    # We use json_event here since the sender is a logstash agent
    format => "json_event"
    }
}
output {
  elasticsearch {
    host => "127.0.0.1"
  }
}

新建shiper.conf

input {
  stdin {
    type => "test"
  }
}
output {
  stdout { codec => rubydebug }
  redis { host => "127.0.0.1" data_type => "list" key => "logstash" }
}

運行配置

java -jar logstash.jar agent -f shipper.conf
java -jar logstash.jar agent -f index.conf

五，配置kibana

logstash的最新版已經內置kibana，你也可以單獨部署kibana。kibana3是純粹JavaScript+html的客戶端，所以可以部署到任意http服務器上。

https://download.elasticsearch.org/kibana/kibana/kibana-3.0.0.zip

解壓到web目錄

http://127.0.0.1/kibana/index.html

資料來源參考：

1.http://www.cnblogs.com/buzzlight/p/logstash_elasticsearch_kibana_log.html

2.http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/

3.http://my.oschina.net/guol/blog/179848

4.http://tinytub.github.io/logstash-install.html

5.install in Ubuntu server

http://tips4admin.com/blog/2013/10/how-to-centralize-your-log-with-logstash-elasticsearch-redis-kibana-in-ubuntu-server/  

6.logstash官方文檔 the logstash book