ELK+Zookeeper+Kafka+Filebeat
一、實驗目的：
1. 強大的搜索功能，elasticsearch可以以分佈式搜索的方式快速檢索，而且支持DSL的語法來進行搜索，簡單的說，就是通過類似配置的語言，快速篩選數據。
2. 完美的展示功能，可以展示非常詳細的圖表信息，而且可以定製展示內容，將數據可視化發揮的淋漓盡致。
3. 分佈式功能，能夠解決大型集羣運維工作很多問題，包括監控、預警、日誌收集解析等。
二、實驗邏輯思路：
1，使用Filebeat這個工具安裝到客戶端。
2，接收到的日誌文件推送給Kafka上存儲。
3，Zookeeper來協調管理Kafka日誌隊列最終推送給Logstash集羣處理。
3，Logstash集羣處理過的日誌文集推送給Easticsearch集羣來處理日誌文件。
4，Easticsearch集羣處理過後把信息推送給Kibana。
5，Kibana通過圖片、圖形、網頁形式呈現給用戶。
三、安裝部署
1.爲了方便實驗的順利進行，Linux防火牆和selinux我建議還是關閉比較好的選擇，不必要爲此折騰時間。
[root@server1 ~]# systemctl stop firewalld [root@server1 ~]#systemctl disable firewalld [root@server1 ~]#sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
2，實驗要求的軟件版本。
1，Easticsearch-6.3.2 2，Logstash-6.3.2 3，Kibana-6.3.2 4，JDK 1.8.0_181 5，zookeeper-3.4.12 6，filebeat-6.3.2-linux-x86_64 7，Kafka_2.10-0.10.0.1
3，在第一臺主機上的配置server1。
[root@server1 ~]# rpm -qa |grep jdk [root@server1 ~]# rpm -e --nodeps java-1.7.0-openjdk-1.7.0.171-2.6.13.2.el7.x86_64 [root@server1 ~]# mkdir -pv /usr/local/ [root@server1 ~]# tar xf jdk-8u181-linux-x64.tar.gz -C /usr/local/ [root@server1 ~]# vim /etc/profile JAVA_HOME=/usr/local/jdk1.8.0_181 export PATH=$PATH:$JAVA_HOME/bin [root@server1 ~]# source /etc/profile [root@server1 ~]# java -version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) [root@server1 ~]# tar -xf elasticsearch-6.4.2.tar.gz -C /usr/local/ [root@server1 ~]# cd /usr/local [root@server1 local]# mv elasticsearch-6.4.2 elasticsearch [root@server1 local]# groupadd elasticsearch [root@server1 local]# useradd -g elasticsearch elasticsearch -m [root@server1 local]# chown -R elasticsearch.elasticsearch /usr/local/ [root@server1 local]# ll /usr/local/elasticsearch/ [root@server1 local]# mkdir -p /data/elasticsearch [root@server1 local]# chown -R elasticsearch.elasticsearch /data/elasticsearch [root@server1 local]# cd /usr/local/elasticsearch/config/ [root@server1 config]# cp elasticsearch.yml elasticsearch.yml.bak
4，server1 上修改elasticsearch.yml配置文件。
[root@server1 ~]# vim /usr/local/elasticsearch/config/elasticsearch.yml cluster.name: ELK-Cluster node.name: server1 node.master: true node.data: true path.data: /data/elasticsearch path.logs: /usr/local/elasticsearch/logs network.host: 0.0.0.0 http.port: 9200 discovery.zen.minimum_master_nodes: 1 discovery.zen.ping_timeout: 3s http.cors.enabled: true http.cors.allow-origin: "*" discovery.zen.ping.unicast.hosts: ["10.93.58.66:9300","10.93.58.41:9300"]
5，在server1修改系統配置參數。
[root@server1 config]# echo "vm.max_map_count=262144" >> /etc/sysctl.conf [root@server1 config]# echo "fs.file-max=655360" >> /etc/sysctl.conf [root@server1 config]# sysctl -p vm.max_map_count = 262144 fs.file-max = 655360 [root@server1 config]# vim /etc/security/limits.conf 最後加上 * soft nproc 65536 * hard nproc 65536 * soft nofile 65536 * hard nofile 65536 [root@server1 config]# vim /etc/security/limits.d/20-nproc.conf * soft nproc 20480
6，切換用戶啓動elasticsearch。
earch [root@server1 ~]# cd /usr/local/elasticsearch/ [root@server1 elasticsearch]# ./bin/elasticsearch -d [root@server1 config]# netstat -lantup |grep java
7，server1安裝elasticsearch-head。
[root@server1 ~]# yum install epel-release.noarch -y [root@server1 ~]# yum install -y nodejs npm [root@server1 ~]# yum install -y git [root@server1 ~]# cd /usr/local/ [root@server1 ~]# git clone git://github.com/mobz/elasticsearch-head.git [root@server1 ~]# npm config set registry=http://registry.npm.taobao.org/ [root@server1 ~]#cd elasticsearch-head/ [root@server1 elasticsearch-head]# npm install [root@server1 elasticsearch-head]# cd _site [root@server1 _site]# vim app.js "http://10.93.58.66:9200"; [root@server1 _site]# su - elasticsearch [elasticsearch@server1 ~]$ cd /usr/local/elasticsearch-head/ [elasticsearch@server1 elasticsearch-head]$ npm run start > [email protected] start /usr/local/elasticsearch-head > grunt server Running "connect:server" (connect) task Waiting forever... Started connect web server on http://localhost:9100
8，server2安裝elasticsearch.yml配置。
cluster.name: ELK-Cluster node.name: server2 node.master: true node.data: true path.data: /data/elasticsearch path.logs: /usr/local/elasticsearch/logs network.host: 0.0.0.0 http.port: 9200 discovery.zen.minimum_master_nodes: 1 discovery.zen.ping_timeout: 3s http.cors.enabled: true http.cors.allow-origin: "*" discovery.zen.ping.unicast.hosts: ["10.93.58.66:9300","10.93.58.41:9300"]
9，server3安裝elasticsearch.yml配置
cluster.name: ELK-Cluster node.name: server3 node.master: false node.data: true path.data: /data/elasticsearch path.logs: /usr/local/elasticsearch/logs network.host: 0.0.0.0 http.port: 9200 discovery.zen.minimum_master_nodes: 1 discovery.zen.ping_timeout: 3s http.cors.enabled: true http.cors.allow-origin: "*" discovery.zen.ping.unicast.hosts: ["10.93.58.66:9300","10.93.58.41:9300"]
10，kafkzk安裝kafkzk-3.4.13配置。
關閉防火牆和selinux [root@kafkzk~]# systemctl stop firewalld [root@kafkzk~]#systemctl disable firewalld [root@kafkzk~]#sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
11，解決一些依賴的問題。
[root@kafkzk ~]#yum install cppunit -y [root@kafkzk ~]#yum install python-setuptools -y
12，新建目錄上傳Java安裝包安裝Java並配置環境變量。
[root@kafkzk ~]#rz [root@kafkzk ~]# mkdir -pv /usr/local/ [root@kafkzk ~]# tar xf jdk-8u181-linux-x64.tar.gz -C /usr/local/ [root@kafkzk ~]# vim /etc/profile 添加如下內容： JAVA_HOME=/usr/local/jdk1.8.0_181 export PATH=$PATH:$JAVA_HOME/bin [root@server1 ~]# source /etc/profile [root@server1 ~]# java -version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
13，安裝kafkzk加入環境變量並啓動服務。
[root@kafkzk ~]#tar xf zookeeper-3.4.13.tar.gz -C /usr/local/ [root@kafkzk ~]#mv /usr/local/zookeeper-3.4.13 /usr/local/zookeeper [root@kafkzk ~]#cd /usr/local/zookeeper/conf [root@kafkzk ~]# cp zoo_sample.cfg zoo_sample.cfg.bak [root@kafkzk ~]#mv zoo_sample.cfg zoo.cfg [root@kafkzk ~]# grep -v "^*" /usr/local/zookeeper/conf/zoo_sample.cfg.bak |grep -v "^#" >/usr/local/zookeeper/conf/zoo.cfg [root@kafkzk ~]#vim zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data/zookeeper clientPort=2181 server.1=10.93.58.66:2888:3888 server.2=10.93.58.41:2888:3888 server.3=10.93.58.209:2888:3888
14，修改環境變量，添加如下內容。
[root@kafkzk ~]#vim /etc/profile export ZOOKEEPER_HOME=/usr/local/zookeeper export PATH=$PATH:$ZOOKEEPER_HOME/bin [root@kafkzk ~]#source /etc/profile [root@kafkzk ~]# zkServer.sh start ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [root@kafkzk ~]# jps 1521 Jps /usr/local/logstash
15，未完成待寫。