redis做解藕（nginx日誌上傳至elasticsearch）

原創

2018-08-26 23:52

實驗環境

server1    172.25.11.1    elasticsearch、nginx  、logstash  
server2    172.25.11.2    redis、logstash
server3    172.25.11.3    kibana

server3上安裝kibana

yum install -y kibana-4.5.1-1.x86_64.rpm
vim /opt/kibana/config/kibana.yml

/etc/init.d/kibana start
netstat -antupl

瀏覽器訪問：172.25.11.3:5601

點擊settings

點擊Discover–>點擊右上角鐘錶–>選擇today

redis做解藕的流程

logstash input{nginx} out{redis} -> input{redis} output {elasticsearch}–>elasticsearch–>kibana

server2:

安裝redis

tar zxf redis-3.0.6.tar.gz
cd redis-3.0.6
yum install -y gcc
make
make install
cd utils/

開啓服務：（一路回車）

[root@server2 utils]# ./install_server.sh 
Welcome to the redis service installer
This script will help you easily set up a running redis server

Please select the redis port for this instance: [6379] 
Selecting default: 6379
Please select the redis config file name [/etc/redis/6379.conf] 
Selected default - /etc/redis/6379.conf
Please select the redis log file name [/var/log/redis_6379.log] 
Selected default - /var/log/redis_6379.log
Please select the data directory for this instance [/var/lib/redis/6379] 
Selected default - /var/lib/redis/6379
Please select the redis executable path [/usr/local/bin/redis-server] 
Selected config:
Port           : 6379
Config file    : /etc/redis/6379.conf
Log file       : /var/log/redis_6379.log
Data dir       : /var/lib/redis/6379
Executable     : /usr/local/bin/redis-server
Cli Executable : /usr/local/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied /tmp/6379.conf => /etc/init.d/redis_6379
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!

查看端口

netstat -antupl|grep 6379

server1：

rpm -ivh logstash-2.3.3-1.noarch.rpm
vim /etc/logstash/conf.d/nginx.conf
input {
        file {
                path => "/var/log/nginx/access.log"
                start_position => "beginning"
        }
}

filter {
        grok {
                match => { "message" => "%{COMBINEDAPACHELOG} %{QS:x_forwarded_for}" }
        }
}
output {
        redis {
                host => ["172.25.11.2"]
                port => 6379
                data_type => "list"
                key => "logstash:redis"
        }
}

/etc/init.d/nginx start
chmod +x /var/log/nginx/access.log  # 文件在打入後臺執行時是logstash身份執行，所以必須有讀的權限
移出所有/etc/logstash/conf.d/下面除nginx.conf之外的所有.conf文件，否則會影響運行結果

server2

rpm -ivh logstash-2.3.3-1.noarch.rpm
vim /etc/logstash/conf.d/es.conf
input {
        redis {
                host => "172.25.11.2"
                port => 6379
                data_type => "list"
                key => "logstash:redis"
        }
}
output {
        elasticsearch {
                hosts => ["172.25.11.1"]
                index => "nginx-%{+YYYY.MM.dd}"
        }
}

瀏覽器中進入kibana網頁
點擊Visualize–>Markdown widget–>添加聯繫人

添加聯繫人，運行，保存

將聯繫人添加如Dashboard
點擊右上角的+號–>選擇所要添加的東西

添加總訪問量
點擊Visualize–>Metric
創建一個新的，選擇nginx服務

點擊保存，寫入名稱保存

添加top10排行榜
點擊Visualize–>添加條形圖（Vertical bar chart）
選擇服務nginx

點擊添加x軸 X-Axis

選擇x軸參數

點擊運行橫軸策劃線ip

點擊保存

點擊Dashboard將剛纔創建的加入（按照上面同樣的操作）

點擊保存
在server1,2上執行/etc/init.d/logstash start將服務打入後臺運行
在一臺主機上去訪問此主機（執行壓測命令）

瀏覽器上設置刷新時間間隔（點擊右上角–>點擊5s，表示5s刷新一次）

實現實時數據採集和展示