一、Elasticsearch 集羣相關介紹
Elasticsearch用於構建高可用和可擴展的系統。擴展的方式可以是購買更好的服務器(縱向擴展)或者購買更多的服務器（橫向擴展）,Elasticsearch能從更強大的硬件中獲得更好的性能，但是縱向擴展也有一定的侷限性。真正的擴展應該是橫向的，它通過增加節點來傳播負載和增加可靠性。對於大多數數據庫而言，橫向擴展意味着你的程序將做非常大的改動來利用這些新添加的設備。對比來說，Elasticsearch天生是分佈式的：它知道如何管理節點來提供高擴展和高可用，這意味着你的程序不需要關心這些。
對於Elasticsearch集羣搭建，可以把索引進行分片存儲，一個索引可以分成若干個片，分別存儲到集羣裏面，而對於集羣裏面的負載均衡，副本分配，索引動態均衡（根據節點的增加或者減少）都是elasticsearch自己內部完成的，一有情況就會重新進行分配。
以下先是介紹幾個關於elasticsearch 的幾個名詞：
cluster：代表一個集羣，集羣中有多個節點，其中有一個爲主節點，這個主節點是可以通過選舉產生的，主從節點是對於集羣內部來說的。es的一個概念就是去中心化，字面上理解就是無中心節點，這是對於集羣外部來說的，因爲從外部來看es集羣，在邏輯上是個整體，你與任何一個節點的通信和與整個es集羣通信是等價的。
shards：代表索引分片，es可以把一個完整的索引分成多個分片，這樣的好處是可以把一個大的索引拆分成多個，分佈到不同的節點上，構成分佈式搜索。分片的數量只能在索引創建前指定，並且索引創建後不能更改。
replicas：代表索引副本，es可以設置多個索引的副本，副本的作用一是提高系統的容錯性，當某個節點某個分片損壞或丟失時可以從副本中恢復。二是提高es的查詢效率，es會自動對搜索請求進行負載均衡。
recovery：代表數據恢復或叫數據重新分佈，es在有節點加入或退出時會根據機器的負載對索引分片進行重新分配，掛掉的節點重新啓動時也會進行數據恢復。
river：代表es的一個數據源，也是其它存儲方式（如：數據庫）同步數據到es的一個方法。它是以插件方式存在的一個es服務，通過讀取river中的數據並把它索引到es中，官方的river有couchDB的，RabbitMQ的，Twitter的，Wikipedia的等。
gateway：代表es索引的持久化存儲方式，es默認是先把索引存放到內存中，當內存滿了時再持久化到硬盤。當這個es集羣關閉再重新啓動時就會從gateway中讀取索引數據。es支持多種類型的gateway，有本地文件系統（默認），分佈式文件系統，Hadoop的HDFS和amazon的s3雲存儲服務。
discovery.zen：代表es的自動發現節點機制，es是一個基於p2p的系統，它先通過廣播尋找存在的節點，再通過多播協議來進行節點之間的通信，同時也支持點對點的交互。
Transport：代表es內部節點或集羣與客戶端的交互方式，默認內部是使用tcp協議進行交互，同時它支持http協議（json格式）、thrift、servlet、memcached、zeroMQ等的傳輸協議（通過插件方式集成）
集羣和節點：節點(node)是你運行的Elasticsearch實例。一個集羣(cluster)是一組具有相同cluster.name的節點集合，他們協同工作，共享數據並提供故障轉移和擴展功能，當有新的節點加入或者刪除節點，集羣就會感知到並平衡數據。集羣中一個節點會被選舉爲主節點(master),它用來管理集羣中的一些變更，例如新建或刪除索引、增加或移除節點等;當然一個節點也可以組成一個集羣。
節點通信：我們能夠與集羣中的任何節點通信，包括主節點。任何一個節點互相知道文檔存在於哪個節點上，它們可以轉發請求到我們需要數據所在的節點上。我們通信的節點負責收集各節點返回的數據，最後一起返回給客戶端。這一切都由Elasticsearch透明的管理。
分片與副本分片：分片用於Elasticsearch在你的集羣中分配數據。想象把分片當作數據的容器。文檔存儲在分片中，然後分片分配給你集羣中的節點上。 當你的集羣擴容或縮小，Elasticsearch將會自動在你的節點間遷移分片，以使集羣保持平衡。一個分片(shard)是一個最小級別的“工作單元(worker unit)”,它只是保存索引中所有數據的一小片.我們的文檔存儲和被索引在分片中，但是我們的程序不知道如何直接與它們通信。取而代之的是，他們直接與索引通信。Elasticsearch中的分片分爲主分片和副本分片,複製分片只是主分片的一個副本，它用於提供數據的冗餘副本，在硬件故障之後提供數據保護，同時服務於像搜索和檢索等只讀請求，主分片的數量和複製分片的數量都可以通過配置文件配置。但是主切片的數量只能在創建索引時定義且不能修改.相同的分片不會放在同一個節點上。
分片算法: shard = hash(routing) % number_of_primary_shards
routing值是一個任意字符串，它默認是_id，但也可以自定義，這個routing字符串通過哈希函數生成一個數字，然後除以主切片的數量得到一個餘數(remainder)，餘數的範圍永遠是0到number_of_primary_shards - 1，這個數字就是特定文檔所在的分片。這也解釋了爲什麼主切片的數量只能在創建索引時定義且不能修改：如果主切片的數量在未來改變了，所有先前的路由值就失效了，文檔也就永遠找不到了。所有的文檔API（get、index、delete、bulk、update、mget）都接收一個routing參數，它用來自定義文檔到分片的映射。自定義路由值可以確保所有相關文檔.比如用戶的文章,按照用戶賬號路由,就可以實現屬於同一用戶的文檔被保存在同一分片上。
分片和副本交互：新建、索引和刪除請求都是寫(write)操作，它們必須在主分片上成功完成才能複製到相關的複製分片上,下面我們羅列在主分片和複製分片上成功新建、索引或刪除一個文檔必要的順序步驟：
1、客戶端給Node 1發送新建、索引或刪除請求。
2、節點使用文檔的_id確定文檔屬於分片0。它轉發請求到Node 3，分片0位於這個節點上。
3、Node 3在主分片上執行請求，如果成功，它轉發請求到相應的位於Node 1和Node 2的複製節點上。當所有的複製節點報告成功，Node 3報告成功到請求的節點，請求的節點再報告給客戶端。
客戶端接收到成功響應的時候，文檔的修改已經被應用於主分片和所有的複製分片。你的修改生效了。
副本分片複製時的相關的參數說明: （replication）複製默認的值是sync。這將導致主分片得到複製分片的成功響應後才返回，如果你設置replication爲async，請求在主分片上被執行後就會返回給客戶端。它依舊會轉發請求給複製節點，但你將不知道複製節點成功與否。默認的sync複製允許Elasticsearch強制反饋傳輸。async複製可能會因爲在不等待其它分片就緒的情況下發送過多的請求而使Elasticsearch過載。
consistency: 默認主分片在嘗試寫入時需要規定一定數量(quorum)或過半的分片（可以是主節點或複製節點）可用。這是防止數據被寫入到錯的網絡分區。規定的數量計算公式如下：
int( (primary + number_of_replicas) / 2 ) + 1
consistency允許的值爲one（只有一個主分片），all（所有主分片和複製分片）或者默認的quorum或過半分片。注意number_of_replicas是在索引中的設置，用來定義複製分片的數量，而不是現在活動的複製節點的數量。如果你定義了索引有3個複製節點，那規定數量是：int( (primary + 3 replicas) / 2 ) + 1 = 3但如果你只有2個節點，那你的活動分片不夠規定數量，也就不能索引或刪除任何文檔。
注意: 新索引默認有1個複製分片，這意味着爲了滿足quorum的要求需要兩個活動的分片。當然，這個默認設置將阻止我們在單一節點集羣中進行操作。爲了避開這個問題，規定數量只有在number_of_replicas大於一時才生效。
timeout：當分片副本不足時Elasticsearch會等待更多的分片出現。默認等待一分鐘。如果需要，你可以設置timeout參數讓它終止的更早：100表示100毫秒，30s表示30秒。
參考鏈接：http://my.oschina.net/davehe/blog/549189
二、Elasticsearch 集羣架構圖（草圖）
服務器配置：Centos6.6 x86_64 CPU：1核心 MEM：2G （做實驗，配置比較低一些）
注：這裏配置elasticsearch集羣用了3臺服務器，可以根據自己的實際情況進行調整。
三、開始安裝配置nginx和logstash
注：這裏使用yum安裝，如果需要較高版本的，可以使用編譯安裝。
在10.0.18.144上操作，10.0.18.145配置方式和144是一樣的。
1、安裝nginx
配置yum源並安裝nginx
#vim /etc/yum.repos.d/nginx.repo [nginx] name=nginx repo baseurl=http://nginx.org/packages/centos/$releasever/$basearch/ gpgcheck=0 enabled=1 安裝 #yum install nginx -y 查看版本 #rpm -qa nginx nginx-1.10.1-1.el6.ngx.x86_64
修改nginx配置文件，修改爲如下：
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log notice; #默認是warn
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $http_x_forwarded_for $request_length $msec $connection_requests $request_time';
##添加了$request_length $msec $connection_requests $request_time
sendfile on;
keepalive_timeout 65;
server {
listen 80;
server_name localhost;
access_log /var/log/nginx/access.log main;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
}
修改nginx默認頁面
#vi /usr/share/nginx/html/index.html
<body>
<h1>Welcome to nginx!</h1>
改爲
<body>
<h1>Welcome to nginx! 144</h1>啓動nginx，並訪問測試：
#service nginx start #chkconfig --add nginx #chkconfig nginx on 查看啓動情況 #netstat -tunlp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1023/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1101/master tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1353/nginx tcp 0 0 :::22 :::* LISTEN 1023/sshd tcp 0 0 ::1:25 :::* LISTEN 1101/master
在瀏覽器訪問測試，如下：
2、安裝配置java環境
直接使用rpm包安裝，比較方便 #rpm -ivh jdk-8u92-linux-x64.rpm Preparing... ########################################### [100%] 1:jdk1.8.0_92 ########################################### [100%] Unpacking JAR files... tools.jar... plugin.jar... javaws.jar... deploy.jar... rt.jar... jsse.jar... charsets.jar... localedata.jar... #java -version java version "1.8.0_92" Java(TM) SE Runtime Environment (build 1.8.0_92-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)
3、安裝配置logstash
配置logstash的yum源，如下：
#vim /etc/yum.repos.d/logstash.repo [logstash-2.3] name=Logstash repository for 2.3.x packages baseurl=https://packages.elastic.co/logstash/2.3/centos gpgcheck=1 gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 安裝logstash #yum install logstash -y 查看版本 #rpm -qa logstash logstash-2.3.4-1.noarch
配置logstash的配置文件
#cd /etc/logstash/conf.d
#vim logstash.conf
input {
file {
path => ["/var/log/nginx/access.log"]
type => "nginx_log"
start_position => "beginning"
}
}
output {
stdout {
codec => rubydebug
}
}
檢測語法是否有錯
#/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --configtest
Configuration OK #語法OK啓動並查看收集nginx日誌情況：
#列出一部分
#/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf
Settings: Default pipeline workers: 1
Pipeline main started
{
"message" => "10.0.90.8 - - [26/Aug/2016:15:30:18 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C; .NET4.0E)\" \"-\" 415 1472196618.085 1 0.000",
"@version" => "1",
"@timestamp" => "2016-08-26T07:30:32.699Z",
"path" => "/var/log/nginx/access.log",
"host" => "0.0.0.0",
"type" => "nginx_log"
}
{
"message" => "10.0.90.8 - - [26/Aug/2016:15:30:18 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C; .NET4.0E)\" \"-\" 415 1472196618.374 2 0.000",
"@version" => "1",
"@timestamp" => "2016-08-26T07:30:32.848Z",
"path" => "/var/log/nginx/access.log",
"host" => "0.0.0.0",
"type" => "nginx_log"
}
………………
PS:在網上看到其他版本logstash的pipeline workers是默認爲4，但我安裝的2.3.4版本這個默認值爲1
這是因爲這個默認值和服務器本身的cpu核數有關，我這裏的服務器cpu都是1核，故默認值爲1。
可以通過 /opt/logstash/bin/logstash -h 命令查看一些參數修改logstash的配置文件，將日誌數據輸出到redis
#cat /etc/logstash/conf.d/logstash.conf
input {
file {
path => ["/var/log/nginx/access.log"]
type => "nginx_log"
start_position => "beginning"
}
}
output {
redis {
host => "10.0.18.146"
key => 'logstash-redis'
data_type => 'list'
}
}檢查語法並啓動服務
#/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --configtest Configuration OK #service logstash start logstash started. 查看啓動進程 #ps -ef | grep logstash logstash 2029 1 72 15:37 pts/0 00:00:18 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/var/lib/logstash -Xmx1g -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/var/lib/logstash -XX:HeapDumpPath=/opt/logstash/heapdump.hprof -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log root 2076 1145 0 15:37 pts/0 00:00:00 grep logstash
四、安裝配置redis
下載並安裝redis
#yum install wget gcc gcc-c++ -y #安裝過的，就不需要再安裝了
#wget http://download.redis.io/releases/redis-3.0.7.tar.gz
#tar xf redis-3.0.7.tar.gz
#cd redis-3.0.7
#make
make沒問題之後，創建目錄
#mkdir -p /usr/local/redis/{conf,bin}
#cp ./*.conf /usr/local/redis/conf/
#cp runtest* /usr/local/redis/
#cd utils/
#cp mkrelease.sh /usr/local/redis/bin/
#cd ../src
#cp redis-benchmark redis-check-aof redis-check-dump redis-cli redis-sentinel redis-server redis-trib.rb /usr/local/redis/bin/
創建redis數據存儲目錄
#mkdir -pv /data/redis/db
#mkdir -pv /data/log/redis修改redis配置文件
#cd /usr/local/redis/conf #vi redis.conf dir ./ 修改爲dir /data/redis/db/ 保存退出 啓動redis #nohup /usr/local/redis/bin/redis-server /usr/local/redis/conf/redis.conf & 查看redis進程 #ps -ef | grep redis root 4425 1149 0 16:21 pts/0 00:00:00 /usr/local/redis/bin/redis-server *:6379 root 4435 1149 0 16:22 pts/0 00:00:00 grep redis #netstat -tunlp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1402/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1103/master tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 4425/redis-server * tcp 0 0 :::22 :::* LISTEN 1402/sshd tcp 0 0 ::1:25 :::* LISTEN 1103/master tcp 0 0 :::6379 :::* LISTEN 4425/redis-server *
五、安裝配置logstash server
1、安裝jdk
#rpm -ivh jdk-8u92-linux-x64.rpm Preparing... ########################################### [100%] 1:jdk1.8.0_92 ########################################### [100%] Unpacking JAR files... tools.jar... plugin.jar... javaws.jar... deploy.jar... rt.jar... jsse.jar... charsets.jar... localedata.jar...
2、安裝logstash
配置yum源 #vim /etc/yum.repos.d/logstash.repo [logstash-2.3] name=Logstash repository for 2.3.x packages baseurl=https://packages.elastic.co/logstash/2.3/centos gpgcheck=1 gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 安裝logstash #yum install logstash -y
配置logstash server
配置文件如下：
#cd /etc/logstash/conf.d
#vim logstash_server.conf
input {
redis {
port => "6379"
host => "10.0.18.146"
data_type => "list"
key => "logstash-redis"
type => "redis-input"
}
}
output {
stdout {
codec => rubydebug
}
}
檢查語法
#/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash_server.conf --configtest
Configuration OK語法沒問題之後，測試查看收集nginx日誌的情況，如下：
#/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash_server.conf
Settings: Default pipeline workers: 1
Pipeline main started
{
"message" => "10.0.90.8 - - [26/Aug/2016:15:42:01 +0800] \"GET /favicon.ico HTTP/1.1\" 404 571 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36\" \"-\" 263 1472197321.350 1 0.000",
"@version" => "1",
"@timestamp" => "2016-08-26T08:45:25.214Z",
"path" => "/var/log/nginx/access.log",
"host" => "0.0.0.0",
"type" => "nginx_log"
}
{
"message" => "10.0.90.8 - - [26/Aug/2016:16:40:53 +0800] \"GET / HTTP/1.1\" 200 616 \"-\" \"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36\" \"-\" 374 1472200853.324 1 0.000",
"@version" => "1",
"@timestamp" => "2016-08-26T08:45:25.331Z",
"path" => "/var/log/nginx/access.log",
"host" => "0.0.0.0",
"type" => "nginx_log"
}
{
"message" => "10.0.90.8 - - [26/Aug/2016:16:40:53 +0800] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://10.0.18.144/\" \"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36\" \"-\" 314 1472200853.486 2 0.000",
"@version" => "1",
"@timestamp" => "2016-08-26T08:45:25.332Z",
"path" => "/var/log/nginx/access.log",
"host" => "0.0.0.0",
"type" => "nginx_log"
}
{
"message" => "10.0.90.8 - - [26/Aug/2016:16:42:05 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36\" \"-\" 481 1472200925.259 1 0.000",
"@version" => "1",
"@timestamp" => "2016-08-26T08:45:25.332Z",
"path" => "/var/log/nginx/access.log",
"host" => "0.0.0.0",
"type" => "nginx_log"
}
{
"message" => "10.0.90.9 - - [26/Aug/2016:16:47:35 +0800] \"GET / HTTP/1.1\" 200 616 \"-\" \"Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko\" \"-\" 298 1472201255.813 1 0.000",
"@version" => "1",
"@timestamp" => "2016-08-26T08:47:36.623Z",
"path" => "/var/log/nginx/access.log",
"host" => "0.0.0.0",
"type" => "nginx_log"
}
{
"message" => "10.0.90.9 - - [26/Aug/2016:16:47:42 +0800] \"GET /favicon.ico HTTP/1.1\" 404 169 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko\" \"-\" 220 1472201262.653 1 0.000",
"@version" => "1",
"@timestamp" => "2016-08-26T08:47:43.649Z",
"path" => "/var/log/nginx/access.log",
"host" => "0.0.0.0",
"type" => "nginx_log"
}
{
"message" => "10.0.90.8 - - [26/Aug/2016:16:48:09 +0800] \"GET / HTTP/1.1\" 200 616 \"-\" \"Mozilla/5.0 (Windows; U; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; BIDUBrowser 8.4)\" \"-\" 237 1472201289.662 1 0.000",
"@version" => "1",
"@timestamp" => "2016-08-26T08:48:09.684Z",
"path" => "/var/log/nginx/access.log",
"host" => "0.0.0.0",
"type" => "nginx_log"
}
…………………………注：執行此命令之後不會立即有信息顯示，需要等一會，也可以在瀏覽器刷新144和145的nginx頁面或者同一網段的其他機器訪問144、145，就會由如上信息出現。
3、修改logstash配置文件，將蒐集到的數據輸出到ES集羣中
#vim /etc/logstash/conf.d/logstash_server.conf
input {
redis {
port => "6379"
host => "10.0.18.146"
data_type => "list"
key => "logstash-redis"
type => "redis-input"
}
}
output {
elasticsearch {
hosts => "10.0.18.149" #其中一臺ES 服務器
index => "nginx-log-%{+YYYY.MM.dd}" #定義的索引名稱，後面會用到
}
}
啓動logstash
#service logstash start
logstash started.
查看logstash server 進程
#ps -ef | grep logstash
logstash 1740 1 24 17:24 pts/0 00:00:25 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/var/lib/logstash -Xmx1g -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/var/lib/logstash -XX:HeapDumpPath=/opt/logstash/heapdump.hprof -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log
root 1783 1147 0 17:25 pts/0 00:00:00 grep logstash六、安裝配置Elasticsearch
在10.0.18.148、10.0.18.149、10.0.18.150三臺ES上安裝jdk和Elasticsearch！jdk的安裝都是一樣的，這裏不做贅述。
1、添加elasticsearch用戶，因爲Elasticsearch服務器啓動的時候，需要在普通用戶權限下來啓動。
#adduser elasticsearch #passwd elasticsearch #爲用戶設置密碼 #su - elasticsearch 下載Elasticsearch包 $wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.3.4/elasticsearch-2.3.4.tar.gz $tar xf elasticsearch-2.3.4.tar.gz $cd elasticsearch-2.3.4
將elasticsearch的配置文件末尾添加如下：
#vim conf/elasticsearch.yml cluster.name: serverlog #集羣名稱，可以自定義 node.name: node-1 #節點名稱，也可以自定義 path.data: /home/elasticsearch/elasticsearch-2.3.4/data #data存儲路徑 path.logs: /home/elasticsearch/elasticsearch-2.3.4/logs #log存儲路徑 network.host: 10.0.18.148 #節點ip http.port: 9200 #節點端口 discovery.zen.ping.unicast.hosts: ["10.0.18.149","10.0.18.150"] #集羣ip列表 discovery.zen.minimum_master_nodes: 3 #集羣幾點數
啓動服務
$cd elasticsearch-2.3.4 $./bin/elasticsearch -d 查看進程 $ps -ef | grep elasticsearch root 1550 1147 0 17:44 pts/0 00:00:00 su - elasticsearch 500 1592 1 4 17:56 pts/0 00:00:13 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/home/elasticsearch/elasticsearch-2.3.4 -cp /home/elasticsearch/elasticsearch-2.3.4/lib/elasticsearch-2.3.4.jar:/home/elasticsearch/elasticsearch-2.3.4/lib/* org.elasticsearch.bootstrap.Elasticsearch start -d 500 1649 1551 0 18:00 pts/0 00:00:00 grep elasticsearch 查看端口 $netstat -tunlp (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN - tcp 0 0 ::ffff:10.0.18.148:9300 :::* LISTEN 1592/java tcp 0 0 :::22 :::* LISTEN - tcp 0 0 ::1:25 :::* LISTEN - tcp 0 0 ::ffff:10.0.18.148:9200 :::* LISTEN 1592/java
啓動連個端口：9200集羣之間事務通信，9300集羣之間選舉通信。
啓動之後，查看三臺Elasticsearch的日誌，會看到“選舉”產生的master節點
第一臺：10.0.18.148
$tail -f logs/serverlog.log
…………………………
[2016-08-26 17:56:05,771][INFO ][env ] [node-1] heap size [1015.6mb], compressed ordinary object pointers [true]
[2016-08-26 17:56:05,774][WARN ][env ] [node-1] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536]
[2016-08-26 17:56:09,416][INFO ][node ] [node-1] initialized
[2016-08-26 17:56:09,416][INFO ][node ] [node-1] starting ...
[2016-08-26 17:56:09,594][INFO ][transport ] [node-1] publish_address {10.0.18.148:9300}, bound_addresses {10.0.18.148:9300}
[2016-08-26 17:56:09,611][INFO ][discovery ] [node-1] serverlog/py6UOr4rRCCuK3KjA-Aj-Q
[2016-08-26 17:56:39,622][WARN ][discovery ] [node-1] waited for 30s and no initial state was set by the discovery
[2016-08-26 17:56:39,633][INFO ][http ] [node-1] publish_address {10.0.18.148:9200}, bound_addresses {10.0.18.148:9200}
[2016-08-26 17:56:39,633][INFO ][node ] [node-1] started
[2016-08-26 17:59:33,303][INFO ][cluster.service ] [node-1] detected_master {node-2}{k0vpt0khTOG0Kmen8EepAg}{10.0.18.149}{10.0.18.149:9300}, added {{node-3}{lRKjIPpFSd-_NVn7-0-JeA}{10.0.18.150}{10.0.18.150:9300},{node-2}{k0vpt0khTOG0Kmen8EepAg}{10.0.18.149}{10.0.18.149:9300},}, reason: zen-disco-receive(from master [{node-2}{k0vpt0khTOG0Kmen8EepAg}{10.0.18.149}{10.0.18.149:9300}])可以看到自動“選舉”node-2，即10.0.18.149爲master節點
第二臺：10.0.18.149
$tail -f logs/serverlog.log
……………………
[2016-08-26 17:58:20,854][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in
[2016-08-26 17:58:21,480][INFO ][node ] [node-2] version[2.3.4], pid[1552], build[e455fd0/2016-06-30T11:24:31Z]
[2016-08-26 17:58:21,491][INFO ][node ] [node-2] initializing ...
[2016-08-26 17:58:22,537][INFO ][plugins ] [node-2] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2016-08-26 17:58:22,574][INFO ][env ] [node-2] using [1] data paths, mounts [[/ (/dev/mapper/vg_template-lv_root)]], net usable_space [14.9gb], net total_space [17.1gb], spins? [possibly], types [ext4]
[2016-08-26 17:58:22,575][INFO ][env ] [node-2] heap size [1015.6mb], compressed ordinary object pointers [true]
[2016-08-26 17:58:22,578][WARN ][env ] [node-2] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536]
[2016-08-26 17:58:26,437][INFO ][node ] [node-2] initialized
[2016-08-26 17:58:26,440][INFO ][node ] [node-2] starting ...
[2016-08-26 17:58:26,783][INFO ][transport ] [node-2] publish_address {10.0.18.149:9300}, bound_addresses {10.0.18.149:9300}
[2016-08-26 17:58:26,815][INFO ][discovery ] [node-2] serverlog/k0vpt0khTOG0Kmen8EepAg
[2016-08-26 17:58:56,838][WARN ][discovery ] [node-2] waited for 30s and no initial state was set by the discovery
[2016-08-26 17:58:56,853][INFO ][http ] [node-2] publish_address {10.0.18.149:9200}, bound_addresses {10.0.18.149:9200}
[2016-08-26 17:58:56,854][INFO ][node ] [node-2] started
[2016-08-26 17:59:33,130][INFO ][cluster.service ] [node-2] new_master {node-2}{k0vpt0khTOG0Kmen8EepAg}{10.0.18.149}{10.0.18.149:9300}, added {{node-1}{py6UOr4rRCCuK3KjA-Aj-Q}{10.0.18.148}{10.0.18.148:9300},{node-3}{lRKjIPpFSd-_NVn7-0-JeA}{10.0.18.150}{10.0.18.150:9300},}, reason: zen-disco-join(elected_as_master, [2] joins received)
[2016-08-26 17:59:33,686][INFO ][gateway ] [node-2] recovered [0] indices into cluster_state也可以看到自動“選舉”node-2，即10.0.18.149爲master節點
第三臺：10.0.18.150
$tail -f logs/serverlog.log
…………………………
[2016-08-26 17:59:25,644][INFO ][node ] [node-3] initializing ...
[2016-08-26 17:59:26,652][INFO ][plugins ] [node-3] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2016-08-26 17:59:26,689][INFO ][env ] [node-3] using [1] data paths, mounts [[/ (/dev/mapper/vg_template-lv_root)]], net usable_space [14.9gb], net total_space [17.1gb], spins? [possibly], types [ext4]
[2016-08-26 17:59:26,689][INFO ][env ] [node-3] heap size [1015.6mb], compressed ordinary object pointers [true]
[2016-08-26 17:59:26,693][WARN ][env ] [node-3] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536]
[2016-08-26 17:59:30,398][INFO ][node ] [node-3] initialized
[2016-08-26 17:59:30,398][INFO ][node ] [node-3] starting ...
[2016-08-26 17:59:30,549][INFO ][transport ] [node-3] publish_address {10.0.18.150:9300}, bound_addresses {10.0.18.150:9300}
[2016-08-26 17:59:30,564][INFO ][discovery ] [node-3] serverlog/lRKjIPpFSd-_NVn7-0-JeA
[2016-08-26 17:59:33,924][INFO ][cluster.service ] [node-3] detected_master {node-2}{k0vpt0khTOG0Kmen8EepAg}{10.0.18.149}{10.0.18.149:9300}, added {{node-1}{py6UOr4rRCCuK3KjA-Aj-Q}{10.0.18.148}{10.0.18.148:9300},{node-2}{k0vpt0khTOG0Kmen8EepAg}{10.0.18.149}{10.0.18.149:9300},}, reason: zen-disco-receive(from master [{node-2}{k0vpt0khTOG0Kmen8EepAg}{10.0.18.149}{10.0.18.149:9300}])
[2016-08-26 17:59:33,999][INFO ][http ] [node-3] publish_address {10.0.18.150:9200}, bound_addresses {10.0.18.150:9200}
[2016-08-26 17:59:34,000][INFO ][node ] [node-3] started也是可以看到自動“選舉”node-2，即10.0.18.149爲master節點！
2、其他信息查看
查看健康信息：
#curl -XGET 'http://10.0.18.148:9200/_cluster/health?pretty'
{
"cluster_name" : "serverlog",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}3、查看節點數
#curl -XGET 'http://10.0.18.148:9200/_cat/nodes?v' host ip heap.percent ram.percent load node.role master name 10.0.18.148 10.0.18.148 7 51 0.00 d m node-1 10.0.18.150 10.0.18.150 5 50 0.00 d m node-3 10.0.18.149 10.0.18.149 7 51 0.00 d * node-2
注意：*表示當前master節點
4、查看節點分片的信息
#curl -XGET 'http://10.0.18.148:9200/_cat/indices?v' health status index pri rep docs.count docs.deleted store.size pri.store.size
還沒有看到分片的信息，後面會介紹原因。
5、在三臺Elasticsearch節點上安裝插件，如下：
#su - elasticsearch $cd elasticsearch-2.3.4 $./bin/plugin install license #license插件 -> Installing license... Trying https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/license/2.3.4/license-2.3.4.zip ... Downloading .......DONE Verifying https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/license/2.3.4/license-2.3.4.zip checksums if available ... Downloading .DONE Installed license into /home/elasticsearch/elasticsearch-2.3.4/plugins/license $ ./bin/plugin install marvel-agent #marvel-agent插件 -> Installing marvel-agent... Trying https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/marvel-agent/2.3.4/marvel-agent-2.3.4.zip ... Downloading ..........DONE Verifying https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/marvel-agent/2.3.4/marvel-agent-2.3.4.zip checksums if available ... Downloading .DONE @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission setFactory * javax.net.ssl.SSLPermission setHostnameVerifier See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y #輸入y，表示同意安裝此插件 Installed marvel-agent into /home/elasticsearch/elasticsearch-2.3.4/plugins/marvel-agent $ ./bin/plugin install mobz/elasticsearch-head #安裝head插件 -> Installing mobz/elasticsearch-head... Trying https://github.com/mobz/elasticsearch-head/archive/master.zip ... Downloading ...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE Verifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksums if available ... NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify) Installed head into /home/elasticsearch/elasticsearch-2.3.4/plugins/head 安裝bigdesk插件 $cd plugins/ $mkdir bigdesk $cd bigdesk $git clone https://github.com/lukas-vlcek/bigdesk _site Initialized empty Git repository in /home/elasticsearch/elasticsearch-2.3.4/plugins/bigdesk/_site/.git/ remote: Counting objects: 5016, done. remote: Total 5016 (delta 0), reused 0 (delta 0), pack-reused 5016 Receiving objects: 100% (5016/5016), 17.80 MiB | 1.39 MiB/s, done. Resolving deltas: 100% (1860/1860), done. 修改_site/js/store/BigdeskStore.js文件，大致在142行，如下： return (major == 1 && minor >= 0 && maintenance >= 0 && (build != 'Beta1' || build != 'Beta2')); 修改爲： return (major >= 1 && minor >= 0 && maintenance >= 0 && (build != 'Beta1' || build != 'Beta2')); 添加插件的properties文件： $cat >plugin-descriptor.properties<<EOF description=bigdesk - Live charts and statistics for Elasticsearch cluster. version=2.5.1 site=true name=bigdesk EOF 安裝kopf插件 $./bin/plugin install lmenezes/elasticsearch-kopf -> Installing lmenezes/elasticsearch-kopf... Trying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip ... Downloading ................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE Verifying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip checksums if available ... NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify) Installed kopf into /home/elasticsearch/elasticsearch-2.3.4/plugins/kopf
查看安裝的插件，如下：
$cd elasticsearch-2.3.4 $ ./bin/plugin list Installed plugins in /home/elasticsearch/elasticsearch-2.3.4/plugins: - head - license - bigdesk - marvel-agent - kopf
七、安裝配置kibana
說明：在10.0.18.150服務器上安裝kibana！
1、配置yum源
#vi /etc/yum.repos.d/kibana.repo [kibana-4.5] name=Kibana repository for 4.5.x packages baseurl=http://packages.elastic.co/kibana/4.5/centos gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 安裝kibana #yum install kibana -y 查看kibana #rpm -qa kibana kibana-4.5.4-1.x86_64 注：使用yum安裝的kibana是默認安裝到/opt目錄下的
2、安裝插件
#cd /opt/kibana/bin/kibana #./kibana plugin --install elasticsearch/marvel/latest Installing marvel Attempting to transfer from https://download.elastic.co/elasticsearch/marvel/marvel-latest.tar.gz Transferring 2421607 bytes.................... Transfer complete Extracting plugin archive Extraction complete Optimizing and caching browser bundles... Plugin installation complete
3、修改kibana配置文件
# vim /opt/kibana/config/kibana.yml #修改爲下面3個參數 server.port: 5601 server.host: "0.0.0.0" elasticsearch.url: "http://10.0.18.150:9200"
4、啓動kibana
#service kibana start kibana started 查看進程 #ps -ef | grep kibana kibana 2050 1 12 20:40 pts/0 00:00:03 /opt/kibana/bin/../node/bin/node /opt/kibana/bin/../src/cli root 2075 1149 0 20:40 pts/0 00:00:00 grep kibana 設置開機自啓動 #chkconfig --add kibana #chkconfig kibana on 查看啓動端口 #netstat -tunlp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1025/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1103/master tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 2050/node #已經啓動成功 tcp 0 0 ::ffff:10.0.18.150:9300 :::* LISTEN 1547/java tcp 0 0 :::22 :::* LISTEN 1025/sshd tcp 0 0 ::1:25 :::* LISTEN 1103/master tcp 0 0 ::ffff:10.0.18.150:9200 :::* LISTEN 1547/java
在瀏覽器訪問kibana端口並創建index，如下：
紅方框中的索引名稱是我在logstash server 服務器的配置文件中配置的index名稱，但是無法創建，提示的信息Unable to fetch mapping…… 說明是Elasticsearch沒有讀取到這個index名稱，逐步排查，看日誌，最後在10.0.18.144、10.0.18.145上查看logstash的日誌，報錯如下：
#tail logstash.log
{:timestamp=>"2016-08-26T20:33:28.404000+0800", :message=>"failed to open /var/log/nginx/access.log: Permission denied - /var/log/nginx/access.log", :level=>:warn}
{:timestamp=>"2016-08-26T20:38:29.110000+0800", :message=>"failed to open /var/log/nginx/access.log: Permission denied - /var/log/nginx/access.log", :level=>:warn}
{:timestamp=>"2016-08-26T20:43:30.834000+0800", :message=>"failed to open /var/log/nginx/access.log: Permission denied - /var/log/nginx/access.log", :level=>:warn}
{:timestamp=>"2016-08-26T20:48:31.559000+0800", :message=>"failed to open /var/log/nginx/access.log: Permission denied - /var/log/nginx/access.log", :level=>:warn}
{:timestamp=>"2016-08-26T20:53:32.298000+0800", :message=>"failed to open /var/log/nginx/access.log: Permission denied - /var/log/nginx/access.log", :level=>:warn}
{:timestamp=>"2016-08-26T20:58:33.028000+0800", :message=>"failed to open /var/log/nginx/access.log: Permission denied - /var/log/nginx/access.log", :level=>:warn}在兩臺nginx服務器操作 #chmod 755 /var/log/nginx/access.log
重新刷新kibana頁面，並創建index名爲nginx-log-*的索引，這次就可以了，如下:
點擊綠色按鈕“Create”，就可以創建成功了！然後查看kibana界面的“Discovery”，就會看到蒐集的nginx日誌了，如下：
可以看到已經蒐集到日誌數據了！
5、訪問head，查看集羣是否一致，如下圖：
6、訪問bigdesk，查看信息，如下圖：
上圖中也標記了node-2爲master節點（有星星標記），上圖顯示的數據是不斷刷新的！
7、訪問kopf,查看信息，如下圖：
上面提到了查看節點分片的信息，結果是沒有數據（因爲剛配置好，還沒有創建索引，所以分片信息還沒有），現在再測試一次，就可以看到數據了，如下圖：
#curl -XGET '10.0.18.148:9200/_cat/indices?v' health status index pri rep docs.count docs.deleted store.size pri.store.size green open .kibana 1 1 3 0 45.2kb 23.9kb green open nginx-log-2016.08.26 5 1 222 0 549.7kb 272.4kb
8、在kibana界面可以查看到nginx-log-*這個index蒐集到的nginx日誌數據，也可以看到Elasticsearch集羣的index--marvel-es-1-*關於集羣的一些信息，如下圖：
八、ELK遇到的一些問題
1、關於kibana端口
配置過kibana的都知道kibana的默認端口是5601，我想修改爲80，結果啓動kibana報錯，如下：
#cat /var/log/kibana/kibana.stderr
FATAL { [Error: listen EACCES 0.0.0.0:80]
cause:
{ [Error: listen EACCES 0.0.0.0:80]
code: 'EACCES',
errno: 'EACCES',
syscall: 'listen',
address: '0.0.0.0',
port: 80 },
isOperational: true,
code: 'EACCES',
errno: 'EACCES',
syscall: 'listen',
address: '0.0.0.0',
port: 80 }
FATAL { [Error: listen EACCES 10.0.18.150:80]
cause:
{ [Error: listen EACCES 10.0.18.150:80]
code: 'EACCES',
errno: 'EACCES',
syscall: 'listen',
address: '10.0.18.150',
port: 80 },
isOperational: true,
code: 'EACCES',
errno: 'EACCES',
syscall: 'listen',
address: '10.0.18.150',
port: 80 }
#tail /var/log/kibana/kibana.stdout
{"type":"log","@timestamp":"2016-08-29T02:54:21+00:00","tags":["fatal"],"pid":3217,"level":"fatal","message":"listen EACCES 10.0.18.150:80","error":{"message":"listen EACCES 10.0.18.150:80","name":"Error","stack":"Error: listen EACCES 10.0.18.150:80\n at Object.exports._errnoException (util.js:873:11)\n at exports._exceptionWithHostPort (util.js:896:20)\n at Server._listen2 (net.js:1237:19)\n at listen (net.js:1286:10)\n at net.js:1395:9\n at nextTickCallbackWith3Args (node.js:453:9)\n at process._tickDomainCallback (node.js:400:17)","code":"EACCES"}}沒有找到解決方法，只能將端口改爲默認的5601了。
2、關於nginx日誌問題
本次實驗是使用yum安裝的nginx，版本是1.10.1,開始是因爲nginx日誌權限，導致無法讀取nginx日誌，後來將日誌文件權限修改爲了755，就可以了。但是，nginx日誌是每天進行logrotate的，新生成的日誌依然是640的權限，所以依然獲取不到日誌數據。所以只能修改nginx的默認logrotate文件了:
#cd /etc/logrotate.d
#cat nginx #默認如下
/var/log/nginx/*.log {
daily
missingok
rotate 52
compress
delaycompress
notifempty
create 640 nginx adm #可以看到默認權限是640，屬主和屬組分別是nginx和adm
sharedscripts
postrotate
[ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
endscript
}
修改後如下：
/var/log/nginx/*.log {
daily
missingok
rotate 52
compress
delaycompress
notifempty
create 755 nginx nginx #修改爲755，屬主和屬組都是nginx
sharedscripts
postrotate
[ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
endscript
}
然後重啓nginx,以後在logrotage的日誌權限就是755了。3、關於Marvel的問題
說明：Marvel是Elasticsearch集羣的monitor ，英文解釋如下：
Marvel is the best way to monitor your Elasticsearch cluster and provide actionable insights to help you get the most out of your cluster. It is free to use in both development and production.
Marvel是監控你的Elasticsearch集羣，並提供可操作的見解，以幫助您充分利用集羣的最佳方式，它是免費的在開發和生產中使用。
問題：Elasticsearch集羣都搭建好之後，在瀏覽器訪問Marvel，查看監控信息的時候頁面報錯，無法顯示監控的信息大致意識是no-data之類的，後來通過排查三臺Elasticsearch的log，有一些錯誤，具體沒有搞清楚是什麼錯，於是重啓了三臺Elasticsearch的elasticsearch服務，再訪問Marvel的監控頁面，就OK了，如下圖：
可以看到serverlog是我配置的集羣名稱，點進去繼續查看，如下圖：
4、節點分片信息相關的問題
在本次實驗的過程中，第一次查看分片信息是沒有的，因爲沒有創建索引，後面等創建過索引之後，就可以看到創建的索引信息了，但是還有集羣的信息沒有顯示出來，問題應該和第2個一樣，Elasticsearch有問題，重啓之後，就查看到了如下：
查看節點分片信息： #curl -XGET '10.0.18.148:9200/_cat/indices?v' health status index pri rep docs.count docs.deleted store.size pri.store.size green open nginx-log-2016.08.29 5 1 2374 0 1.7mb 902.9kb green open nginx-log-2016.08.27 5 1 2323 0 1mb 528.6kb green open .marvel-es-data-1 1 1 5 3 17.6kb 8.8kb green open .kibana 1 1 3 0 45.2kb 21.3kb green open .marvel-es-1-2016.08.29 1 1 16666 108 12.1mb 6.1mb green open nginx-log-2016.08.26 5 1 1430 0 800.4kb 397.8kb
5、關於創建多個index索引名稱，存儲不同類型日誌的情況
也許我們不止nginx這一種日誌需要蒐集分析，還有httpd、tomcat、mysql等日誌，但是如果都蒐集在nginx-log-*這個索引下面，會很亂，不易於排查問題，如果每一種類型的日誌都創建一個索引，這樣分類創建索引，會比較直觀，實現是在logstash server 服務器上創建多個conf文件，然後逐個啓動，如下：
#cd /etc/logstash/conf.d/
#cat logstash_server.conf
input {
redis {
port => "6379"
host => "10.0.18.146"
data_type => "list"
key => "logstash-redis"
type => "redis-input"
}
}
output {
elasticsearch {
hosts => "10.0.18.149"
index => "nginx-log-%{+YYYY.MM.dd}"
}
}
#cat logstash_server1.conf
input {
redis {
port => "6379"
host => "10.0.18.146"
data_type => "list"
key => "logstash-redisa"
type => "redis-input"
}
}
output {
elasticsearch {
hosts => "10.0.18.149"
index => "httpd-log-%{+YYYY.MM.dd}"
}
}
如果還有其他日誌，仿照上面的conf文件即可，不同的是index名稱和key
然後逐個啓動
#nohup /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash_server.conf &
#nohup /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash_server1.conf &
再對應的日誌服務器（稱爲客戶端）本身配置conf文件，如下：
#cat /etc/logstash/conf.d/logstash-web.conf
input {
file {
path => ["/var/log/httpd/access_log"]
type => "httpd_log" #type
start_position => "beginning"
}
}
output {
redis {
host => "10.0.18.146"
key => 'logstash-redisa' #key
data_type => 'list'
}
}
然後啓動logstash服務，再到kibana界面創建新的索引httpd-log-*,就可以在這個索引下面查看到蒐集到的httpd日誌了！6、elasticsearch啓動之後，提示最大文件數太小的問題
ELK集羣搭建好之後，開啓elasticsearch，提示下面的warn：
[WARN ][env ] [node-1] max file descriptors [65535] for elasticsearch process likely too low, consider increasing to at least [65536] 於是修改文件/etc/security/limits.conf ,添加如下： * soft nofile 65536 * hard nofile 65536
參考鏈接：http://ckl893.blog.51cto.com/8827818/1772287
不足之處，請多多指出！