最近在研究日誌處理，採用 EKF，E 是 Elasticsearch，K 是 Kibana，F 是 Filebeat。Filebeat 用於對每臺機器的日誌進行採集，然後發送到 Elasticsearch，使用 Kibana 進行展示分析。
搭建一整套環境，整套 EKF 的版本一直是最佳的，所以搭建採用了版本爲 6.3.2，下面是詳細的搭建過程：
Elasticsearch 搭建
1、下載 Elasticsearch（下載其他版本換掉後面版本號即可）
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.tar.gz
tar -xzvf elasticsearch-6.3.2.tar.gz
2、因爲 Elasticsearch 不能用 root 賬戶啓動，所以我們要創建一個新賬戶
useradd elasticsearch
passwd elasticsearch
chown elasticsearch:elasticsearch -R elasticsearch-6.3.2
3、修改配置文件，Elasticsearch 主要配置文件有兩個 elasticsearch.yml 和 jvm.options，路徑是 elasticsearch-6.3.2/config/
vim elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#集羣名稱
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /data/ELK/data
# data 索引數據存儲路徑
#
# Path to log files:
#
path.logs: /data/ELK/logs
# 日誌存儲路徑
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
# 配置hosts 地址
#
# Set a custom port for HTTP:
#
http.port: 9200
# 配置端口
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes:
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
bootstrap.system_call_filter: false
xpack.security.enabled: false
xpack.security.transport.ssl.enabled: false
# 若暫時不用 x-pack 可增加以上參數配置暫時關閉 因爲es6.3之後默認配置了x-pack 插件
vim jvm.options 根據服務器的配置修改 jvm 參數 -Xms -Xmx 的大小
4、啓動
su elasticsearch
cd elasticsearch-6.3.2/bin
sh elasticsearch &
5、可能遇到的報錯
1⃣️ max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]
切換到 root 用戶
vim /etc/security/limits.conf
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
2⃣️ memory locking requested for elasticsearch process but memory is not locked
vim /etc/security/limits.conf
elastic - memlock unlimited
3⃣️ max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
vim /etc/sysctl.conf
vm.max_map_count=655360
Filebeat 部署
1、下載 filebeat
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.2-linux-x86_64.tar.gz
tar -xzvf filebeat-6.3.2-linux-x86_64
2、修改配置文件
filebeat 本身自帶了許多模板可以直接使用，6.3.2版本已經提供了 nginx、apache、redis、mysql等常用服務模板。
cd filebeat-6.3.2-linux-x86_64
# 查看已經啓動的模板
./filebeat modules list
# 啓動 nginx 模板
./filebeat modules enable nginx
# 關閉 nginx 模板
./filebeat modules disable nginx
修改配置文件
vim filebeat.yml
# 以下幾個地方需要修改
filebeat.inputs:
- type: log
enabled: true
paths:
- /usr/local/mysql.log
setup.kibana:
host: "0.0.0.0:8000" # 根據自己的ip和端口進行調整
output.elasticsearch:
hosts: ["0.0.0.0:9200"]
啓動：
# 設置初始環境
./filebeat setup -e
# 運行
./filebeat -c filebeat.yml -e &
Kibana 部署
1、下載 Kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-linux-x86_64.tar.gz
tar -xzvf kibana-6.3.2-linux-x86_64.tar.gz
2、修改配置文件
# 修改配置文件
vim kibana-6.3.2-linux-x86_64/config/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://localhost:9200"
3、啓動
# 啓動
cd kibana-6.3.2-linux-x86_64/bin
sh kibana &
# 查看進程
ps -ef | grep node