Zabbix 3.4.6版本開始支持歷史數據存儲到Elasticsearch, 早就想測試這個功能，最近有個需求需保存zabbix的歷史數據上達十億條，因此決定測試這功能的實用性，事實證明確實效果挺好。從今以後zabbix也支持大量的歷史數據。
以下是測試12億條數據存儲環境：
測試環境
服務器系統：Ubuntu 16.04
Elasticsearch服務器IP：192.168.1.231
安裝Elasticsearch
設置sysctl.conf
#vi /etc/sysctl.conf
vm.max_map_count=655360
#sysctl -p
設置limits.conf
#vi /etc/security/limits.conf
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
elasticsearch soft nofile 65536
elasticsearch hard nofile 131072
elasticsearch soft nproc 65536
elasticsearch hard nproc 65536
/etc/elasticsearch/jvm.option
禁用swap
#vi /etc/fstab
#/dev/mapper/cryptswap1 none swap sw 0 0 註釋
安裝java
ELK依賴java，因此需要安裝
#add-apt-repository ppa:webupd8team/java
#apt-get update
#apt-get install oracle-java8-installer
安裝Elasticsearch服務
#wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.2.deb
#dpkg -i elasticsearch-6.4.2.deb
配置jvm文件
以下配置僅供參考
#vim /etc/elasticsearch/jvm.options
-Xms4g #配置服務器內存的50%
-Xmx4g #配置服務器內存的50%
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+AlwaysPreTouch
-server
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-XX:-OmitStackTraceInFastThrow
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/lib/elasticsearch
配置elasticsearch.yml
以下配置僅供參考
#vim /etc/elasticsearch/elasticsearch.yml
cluster.name: elk-group
node.name: elk-1
node.master: true
node.data: true
node.attr.rack: r1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
transport.tcp.port: 9300
http.cors.enabled: true
discovery.zen.ping.unicast.hosts: ["192.168.1.231"]
discovery.zen.minimum_master_nodes: 1
cluster.routing.allocation.same_shard.host: true
discovery.zen.fd.ping_timeout: 60s
discovery.zen.fd.ping_retries: 5
如需Salve Elasticsearch配置如下：
#vim /etc/elasticsearch/elasticsearch.yml
cluster.name: elk-group
node.name: elk-2
node.master: false
node.data: true
node.attr.rack: r1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
transport.tcp.port: 9300
http.cors.enabled: true
discovery.zen.ping.unicast.hosts: ["192.168.1.231"]
discovery.zen.minimum_master_nodes: 1
cluster.routing.allocation.same_shard.host: true
discovery.zen.fd.ping_timeout: 60s
discovery.zen.fd.ping_retries: 5
啓動Elasticsearch服務
service elasticsearch start
添加Elasticsearch mapping
Elasticsearch 支持Zabbix的監控項類型：uint，dbl，str，log，text，對應如下
|
Zabbix監控項數據類型 |
對應Zabbix表 |
對應Elasticsearch類型 |
|
Numeric(unsigned) |
history_uint |
uint |
|
Numeric(float) |
history |
dbl |
|
Character |
history_str |
str |
|
Log |
history_log |
log |
|
Text |
history_text |
text |
添加Elasticsearch mapping
#curl -H "Content-Type:application/json" -XPUT http://192.168.1.231:9200/uint -d ' { "settings" : { "index" : { "number_of_replicas" : 1, "number_of_shards" : 5 } }, "mappings" : { "values" : { "properties" : { "itemid" : { "type" : "long" }, "clock" : { "format" : "epoch_second", "type" : "date" }, "value" : { "type" : "long" } } } } } '
#curl -H "Content-Type:application/json" -XPUT http://192.168.1.231:9200/dbl -d ' { "settings" : { "index" : { "number_of_replicas" : 1, "number_of_shards" : 5 } }, "mappings" : { "values" : { "properties" : { "itemid" : { "type" : "long" }, "clock" : { "format" : "epoch_second", "type" : "date" }, "value" : { "type" : "double" } } } } } '
#curl -H "Content-Type:application/json" -XPUT http://192.168.1.231:9200/log -d ' { "settings" : { "index" : { "number_of_replicas" : 1, "number_of_shards" : 5 } }, "mappings" : { "values" : { "properties" : { "itemid" : { "type" : "long" }, "clock" : { "format" : "epoch_second", "type" : "date" }, "value" : { "fields" : { "analyzed" : { "index" : true, "type" : "text", "analyzer" : "standard" } }, "index" : false, "type" : "text" } } } } } '
#curl -H "Content-Type:application/json" -XPUT http://192.168.1.231:9200/text -d ' { "settings" : { "index" : { "number_of_replicas" : 1, "number_of_shards" : 5 } }, "mappings" : { "values" : { "properties" : { "itemid" : { "type" : "long" }, "clock" : { "format" : "epoch_second", "type" : "date" }, "value" : { "fields" : { "analyzed" : { "index" : true, "type" : "text", "analyzer" : "standard" } }, "index" : false, "type" : "text" } } } } } '
#curl -H "Content-Type:application/json" -XPUT http://192.168.1.231:9200/str -d ' { "settings" : { "index" : { "number_of_replicas" : 1, "number_of_shards" : 5 } }, "mappings" : { "values" : { "properties" : { "itemid" : { "type" : "long" }, "clock" : { "format" : "epoch_second", "type" : "date" }, "value" : { "fields" : { "analyzed" : { "index" : true, "type" : "text", "analyzer" : "standard" } }, "index" : false, "type" : "text" } } } } } '
配置zabbix服務器
Zabbix安裝過程忽略
配置zabbix_server.conf文件
在/etc/zabbix/zabbix_server.conf文件下添加elasticsearch配置，指定數據類型使用elasticsearch。
#vim /etc/zabbix/zabbix_server.conf
HistoryStorageURL=http://192.168.1.231:9200
HistoryStorageTypes=uint,dbl,str,log,text
配置zabbix.conf.php文件
在/etc/zabbix/web/zabbix.conf.php文件下添加elasticsearch配置
#vim /etc/zabbix/zabbix_server.conf
<?php
// Zabbix GUI configuration file.
global $DB,$HISTORY;
$DB['TYPE'] = 'MYSQL';
$DB['SERVER'] = '192.168.1.230';
$DB['PORT'] = '0';
$DB['DATABASE'] = 'zabbix';
$DB['USER'] = 'zabbix';
$DB['PASSWORD'] = 'Zabbix';
// Schema name. Used for IBM DB2 and PostgreSQL.
$DB['SCHEMA'] = '';
$ZBX_SERVER = 'localhost';
$ZBX_SERVER_PORT = '10051';
$ZBX_SERVER_NAME = '';
$IMAGE_FORMAT_DEFAULT = IMAGE_FORMAT_PNG;
// Elasticsearch url (can be string if same url is used for all types).
$HISTORY['url'] = 'http://192.168.1.231:9200';
// Value types stored in Elasticsearch.
$HISTORY['types'] = ['uint', 'text', 'log', 'str', 'dbl'];
多臺elasticsearch集羣可按以下格式配置
$HISTORY['url'] = [ 'uint' => 'http://192.168.1.230:9200 ', 'text' => 'http://192.168.1.234:9200 '
'log' => 'http://192.168.1.235:9200 ' ];
$HISTORY['types'] = ['uint', 'text','log'];
啓動zabbix服務
service zabbix-server start
數據測試
安裝kibana
在Elasticsearch服務器安裝，如獨立服務器請另外安裝java
#wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.2-amd64.deb
#dpkg -i kibana-6.4.2-amd64.deb
配置kibana
#vim /etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.1.231:9200"
啓動kibana服務
service kibana start
數據驗證
web登錄