kafka收集elk日誌
Kafka是最初由Linkedin公司開發，是一個分佈式、支持分區的（partition）、多副本的（replica），基於zookeeper協調的分佈式消息系統，它的最大的特性就是可以實時的處理大量數據以滿足各種需求場景;
之前搭建過Redis做爲中間件來緩存日誌，由於redis做消息隊列並不是他的強項，日誌量過多就會有他自己的瓶頸，不能很好的緩解es集羣的壓力，正規的企業還是會引進專業的kafka消息隊列作爲中間件，來滿足PB級別的日誌量。
kafka做消息系統的優勢：
-
解耦：在項目啓動之初來預測將來項目會碰到什麼需求，是極其困難的。消息系統在處理過程中間插入了一個隱含的、基於數據的接口層，兩邊的處理過程都要實現這一接口。通俗地講，解耦就是解決了2個不同的人對待同一件事的做法，是這2個人最終都滿意的過程就是解耦
-
冗餘：有些情況下，處理數據的過程會失敗。除非數據被持久化，否則將造成丟失。消息隊列把數據進行持久化直到它們已經被完全處理，通過這一方式規避了數據丟失風險。
-
擴展性：因爲消息隊列解耦了你的處理過程，所以增大消息入隊和處理的頻率是很容易的，只要另外增加處理過程即可
-
靈活性&削峯：在訪問量劇增的情況下，應用仍然需要繼續發揮作用，但是這樣的突發流量並不常見；如果爲以能處理這類峯值訪問爲標準來投入資源隨時待命無疑是巨大的浪費。使用消息隊列能夠使關鍵組件頂住突發的訪問壓力，而不會因爲突發的超負荷的請求而完全崩潰
-
異步通信：很多時候，用戶不想也不需要立即處理消息。消息隊列提供了異步處理機制，允許用戶把一個消息放入隊列，但並不立即處理它。想向隊列中放入多少消息就放多少，然後在需要的時候再去處理它們
-
順序保證：在大多使用場景下，數據處理的順序都很重要。大部分消息隊列本來就是排序的，並且能保證數據會按照特定的順序來處理。Kafka能保證一個Partition內的消息的有序性。
kafka官網下載地址：http://kafka.apache.org/downloads
zookeeper官網下載地址：https://zookeeper.apache.org/releases.html
注：kafka不能單獨使用，必須依賴於zookeeper，zookeeper就像是它 的管理工具一樣
elk+kafka架構圖
- filebeat輕量級日誌收集工具，區別於logstash就是filebeat佔用的資源更少
- filebeat收集完後appserver產生的日誌，傳送到kafka集羣
- kafka作爲中間件先去接收這些日誌，主要發揮削峯，緩解es壓力
- 傳送到logstash開始過濾等處理後，最後輸出給es集羣
- kibana在web界面登錄展示
搭建zookeeper集羣部署
注：官網最新版zookeeper3.5.5版本以後，我們需要下載的包是apache-zookeeper-3.5.8-bin.tar.gz
帶bin纔是二進制編譯完後的包，我們可以直接使用
而之前的普通的tar.gz的包裏面是隻是源碼的包無法直接使用
由於是實時收集日誌，一定要保證時間一致
羣集內每一臺節點都要做
yum -y install ntpdate
ntpdate ntp1.aliyun.com
es,kibana,logstrash,zookeeper等都需要java環境
yum -y install java-1.8.0-openjdk*
解壓縮包
tar zxf apache-zookeeper-3.5.8-bin.tar.gz -C /opt
cd /opt/
mv apache-zookeeper-3.5.8-bin/ ./zookeeper
cp zookeeper/conf/zoo_sample.cfg ./zookeeper/conf/zoo.cfg ##複製樣例文件修改
添加主配置信息
vim /opt/zookeeper/conf/zoo.cfg
dataDir=/data/zookeeper #數據文件存放目錄
clientPort=2181
tickTime=2000
initLimit=20
syncLimit=10
server.1=192.168.10.2:2888:3888
server.2=192.168.10.1:2888:3888
server.3=192.168.10.7:2888:3888
每一臺的節點的配置文件添加的都是一樣的，只是myid 不一樣而已，
myid來源於server.x；
mkdir -p /data/zookeeper
echo '1' > /data/zookeeper/myid
把zookeeper有關的目錄同步到其他的節點
rsync -avh --delete /opt/zookeeper [email protected]:/opt/
rsync -avh --delete /opt/zookeeper [email protected]:/opt/
rsync -avh --delete /data [email protected]:/
rsync -avh --delete /data [email protected]:/
修改節點的myid
sed -i 's#1#2#g' /data/zookeeper/myid
sed -i 's#1#3#g' /data/zookeeper/myid
啓動zookeeper
/opt/zookeeper/bin/zkServer.sh start #開啓
/opt/zookeeper/bin/zkServer.sh stop #關閉
/opt/zookeeper/bin/zkServer.sh restart #重啓
/opt/zookeeper/bin/zkServer.sh status #查看狀態
都啓動以後查看狀態
會顯示有1個leader，2個follower
[root@eskibana ~]# /opt/zookeeper/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
[root@filebeat data]# /opt/zookeeper/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader
[root@localhost ~]# /opt/zookeeper/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Client port 181. Client address: localhost.
Mode: follower
測試zookeeper集羣
在其中任何一臺節點上創建數據都可以
[root@localhost ~]# /opt/zookeeper/bin/zkCli.sh -server 192.168.10.2:2181
/usr/bin/java
Connecting to 192.168.10.2:2181
2020-06-27 17:48:53,074 [myid:] - INFO [main:Environment@109] - Client environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT
2020-06-27 17:48:53,079 [myid:] - INFO [main:Environment@109] - Client environment:host.name=localhost
2020-06-27 17:48:53,079 [myid:] - INFO [main:Environment@109] - Client environment:java.version=1.8.0_102
2020-06-27 17:48:53,088 [myid:] - INFO [main:Environment@109] - Client environment:java.vendor=Oracle Corporation
2020-06-27 17:48:53,088 [myid:] - INFO [main:Environment@109] - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-4.b14.el7.x86_64/jre
2020-06-27 17:48:53,088 [myid:] - INFO [main:Environment@109] - Client environment:java.class.path=/opt/zookeeper/bin/../zookeeper-server/target/classes:/opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../zookeeper-server/target/lib/*.jar:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/zookeeper-jute-3.5.8.jar:/opt/zookeeper/bin/../lib/zookeeper-3.5.8.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.7.25.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.7.25.jar:/opt/zookeeper/bin/../lib/netty-transport-native-unix-common-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-transport-native-epoll-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-transport-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-resolver-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-handler-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-common-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-codec-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-buffer-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.17.jar:/opt/zookeeper/bin/../lib/json-simple-1.1.1.jar:/opt/zookeeper/bin/../lib/jline-2.11.jar:/opt/zookeeper/bin/../lib/jetty-util-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/jetty-servlet-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/jetty-server-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/jetty-security-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/jetty-io-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/jetty-http-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/zookeeper/bin/../lib/jackson-databind-2.10.3.jar:/opt/zookeeper/bin/../lib/jackson-core-2.10.3.jar:/opt/zookeeper/bin/../lib/jackson-annotations-2.10.3.jar:/opt/zookeeper/bin/../lib/commons-cli-1.2.jar:/opt/zookeeper/bin/../lib/audience-annotations-0.5.0.jar:/opt/zookeeper/bin/../zookeeper-*.jar:/opt/zookeeper/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt/zookeeper/bin/../conf:
2020-06-27 17:48:53,088 [myid:] - INFO [main:Environment@109] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2020-06-27 17:48:53,088 [myid:] - INFO [main:Environment@109] - Client environment:java.io.tmpdir=/tmp
2020-06-27 17:48:53,088 [myid:] - INFO [main:Environment@109] - Client environment:java.compiler=<NA>
2020-06-27 17:48:53,088 [myid:] - INFO [main:Environment@109] - Client environment:os.name=Linux
2020-06-27 17:48:53,088 [myid:] - INFO [main:Environment@109] - Client environment:os.arch=amd64
2020-06-27 17:48:53,088 [myid:] - INFO [main:Environment@109] - Client environment:os.version=3.10.0-514.el7.x86_64
2020-06-27 17:48:53,089 [myid:] - INFO [main:Environment@109] - Client environment:user.name=root
2020-06-27 17:48:53,089 [myid:] - INFO [main:Environment@109] - Client environment:user.home=/root
2020-06-27 17:48:53,089 [myid:] - INFO [main:Environment@109] - Client environment:user.dir=/root
2020-06-27 17:48:53,089 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.free=24MB
2020-06-27 17:48:53,090 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.max=228MB
2020-06-27 17:48:53,090 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.total=29MB
2020-06-27 17:48:53,097 [myid:] - INFO [main:ZooKeeper@868] - Initiating client connection, connectString=192.168.10.2:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@7e774085
2020-06-27 17:48:53,108 [myid:] - INFO [main:X509Util@79] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2020-06-27 17:48:53,120 [myid:] - INFO [main:ClientCnxnSocket@237] - jute.maxbuffer value is 4194304 Bytes
2020-06-27 17:48:53,134 [myid:] - INFO [main:ClientCnxn@1653] - zookeeper.request.timeout value is 0. feature enabled=
Welcome to ZooKeeper!
JLine support is enabled
2020-06-27 17:48:53,223 [myid:192.168.10.2:2181] - INFO [main-SendThread(192.168.10.2:2181):ClientCnxn$SendThread@1112] - Opening socket connection to server 192.168.10.2/192.168.10.2:2181. Will not attempt to authenticate using SASL (unknown error)
2020-06-27 17:48:53,274 [myid:192.168.10.2:2181] - INFO [main-SendThread(192.168.10.2:2181):ClientCnxn$SendThread@959] - Socket connection established, initiating session, client: /192.168.10.2:38026, server: 192.168.10.2/192.168.10.2:2181
2020-06-27 17:48:53,289 [myid:192.168.10.2:2181] - INFO [main-SendThread(192.168.10.2:2181):ClientCnxn$SendThread@1394] - Session establishment complete on server 192.168.10.2/192.168.10.2:2181, sessionid = 0x1000024bc5c0002, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: 192.168.10.2:2181(CONNECTED) 0]
[zk: 192.168.10.2:2181(CONNECTED) 0] create /test "haha"
Created /test
[zk: 192.168.10.2:2181(CONNECTED) 1]
[root@filebeat ~]# /opt/zookeeper/bin/zkCli.sh -server 192.168.10.7:2181
Connecting to 192.168.10.7:2181
2020-06-27 17:50:09,374 [myid:] - INFO [main:Environment@109] - Client environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT
2020-06-27 17:50:09,377 [myid:] - INFO [main:Environment@109] - Client environment:host.name=filebeat
2020-06-27 17:50:09,377 [myid:] - INFO [main:Environment@109] - Client environment:java.version=1.8.0_102
2020-06-27 17:50:09,379 [myid:] - INFO [main:Environment@109] - Client environment:java.vendor=Oracle Corporation
2020-06-27 17:50:09,379 [myid:] - INFO [main:Environment@109] - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-4.b14.el7.x86_64/jre
2020-06-27 17:50:09,379 [myid:] - INFO [main:Environment@109] - Client environment:java.class.path=/opt/zookeeper/bin/../zookeeper-server/target/classes:/opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../zookeeper-server/target/lib/*.jar:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/zookeeper-jute-3.5.8.jar:/opt/zookeeper/bin/../lib/zookeeper-3.5.8.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.7.25.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.7.25.jar:/opt/zookeeper/bin/../lib/netty-transport-native-unix-common-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-transport-native-epoll-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-transport-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-resolver-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-handler-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-common-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-codec-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-buffer-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.17.jar:/opt/zookeeper/bin/../lib/json-simple-1.1.1.jar:/opt/zookeeper/bin/../lib/jline-2.11.jar:/opt/zookeeper/bin/../lib/jetty-util-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/jetty-servlet-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/jetty-server-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/jetty-security-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/jetty-io-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/jetty-http-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/zookeeper/bin/../lib/jackson-databind-2.10.3.jar:/opt/zookeeper/bin/../lib/jackson-core-2.10.3.jar:/opt/zookeeper/bin/../lib/jackson-annotations-2.10.3.jar:/opt/zookeeper/bin/../lib/commons-cli-1.2.jar:/opt/zookeeper/bin/../lib/audience-annotations-0.5.0.jar:/opt/zookeeper/bin/../zookeeper-*.jar:/opt/zookeeper/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt/zookeeper/bin/../conf:
2020-06-27 17:50:09,379 [myid:] - INFO [main:Environment@109] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2020-06-27 17:50:09,379 [myid:] - INFO [main:Environment@109] - Client environment:java.io.tmpdir=/tmp
2020-06-27 17:50:09,379 [myid:] - INFO [main:Environment@109] - Client environment:java.compiler=<NA>
2020-06-27 17:50:09,379 [myid:] - INFO [main:Environment@109] - Client environment:os.name=Linux
2020-06-27 17:50:09,379 [myid:] - INFO [main:Environment@109] - Client environment:os.arch=amd64
2020-06-27 17:50:09,379 [myid:] - INFO [main:Environment@109] - Client environment:os.version=3.10.0-514.el7.x86_64
2020-06-27 17:50:09,379 [myid:] - INFO [main:Environment@109] - Client environment:user.name=root
2020-06-27 17:50:09,379 [myid:] - INFO [main:Environment@109] - Client environment:user.home=/root
2020-06-27 17:50:09,379 [myid:] - INFO [main:Environment@109] - Client environment:user.dir=/root
2020-06-27 17:50:09,379 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.free=24MB
2020-06-27 17:50:09,382 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.max=228MB
2020-06-27 17:50:09,382 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.total=29MB
2020-06-27 17:50:09,387 [myid:] - INFO [main:ZooKeeper@868] - Initiating client connection, connectString=192.168.10.7:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@7e774085
2020-06-27 17:50:09,393 [myid:] - INFO [main:X509Util@79] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2020-06-27 17:50:09,400 [myid:] - INFO [main:ClientCnxnSocket@237] - jute.maxbuffer value is 4194304 Bytes
2020-06-27 17:50:09,410 [myid:] - INFO [main:ClientCnxn@1653] - zookeeper.request.timeout value is 0. feature enabled=
Welcome to ZooKeeper!
JLine support is enabled
2020-06-27 17:50:09,503 [myid:192.168.10.7:2181] - INFO [main-SendThread(192.168.10.7:2181):ClientCnxn$SendThread@1112] - Opening socket connection to server filebeat/192.168.10.7:2181. Will not attempt to authenticate using SASL (unknown error)
2020-06-27 17:50:09,513 [myid:192.168.10.7:2181] - INFO [main-SendThread(192.168.10.7:2181):ClientCnxn$SendThread@959] - Socket connection established, initiating session, client: /192.168.10.7:43706, server: filebeat/192.168.10.7:2181
2020-06-27 17:50:09,529 [myid:192.168.10.7:2181] - INFO [main-SendThread(192.168.10.7:2181):ClientCnxn$SendThread@1394] - Session establishment complete on server filebeat/192.168.10.7:2181, sessionid = 0x3000156e8410002, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: 192.168.10.7:2181(CONNECTED) 0] get /test
haha
[zk: 192.168.10.7:2181(CONNECTED) 1]
[root@eskibana ~]# /opt/zookeeper/bin/zkCli.sh -server 192.168.10.1:2181
Connecting to 192.168.10.1:2181
2020-06-27 17:51:04,114 [myid:] - INFO [main:Environment@109] - Client environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT
2020-06-27 17:51:04,118 [myid:] - INFO [main:Environment@109] - Client environment:host.name=eskibana
2020-06-27 17:51:04,119 [myid:] - INFO [main:Environment@109] - Client environment:java.version=1.8.0_102
2020-06-27 17:51:04,120 [myid:] - INFO [main:Environment@109] - Client environment:java.vendor=Oracle Corporation
2020-06-27 17:51:04,120 [myid:] - INFO [main:Environment@109] - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-4.b14.el7.x86_64/jre
2020-06-27 17:51:04,120 [myid:] - INFO [main:Environment@109] - Client environment:java.class.path=/opt/zookeeper/bin/../zookeeper-server/target/classes:/opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../zookeeper-server/target/lib/*.jar:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/zookeeper-jute-3.5.8.jar:/opt/zookeeper/bin/../lib/zookeeper-3.5.8.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.7.25.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.7.25.jar:/opt/zookeeper/bin/../lib/netty-transport-native-unix-common-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-transport-native-epoll-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-transport-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-resolver-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-handler-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-common-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-codec-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/netty-buffer-4.1.48.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.17.jar:/opt/zookeeper/bin/../lib/json-simple-1.1.1.jar:/opt/zookeeper/bin/../lib/jline-2.11.jar:/opt/zookeeper/bin/../lib/jetty-util-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/jetty-servlet-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/jetty-server-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/jetty-security-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/jetty-io-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/jetty-http-9.4.24.v20191120.jar:/opt/zookeeper/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/zookeeper/bin/../lib/jackson-databind-2.10.3.jar:/opt/zookeeper/bin/../lib/jackson-core-2.10.3.jar:/opt/zookeeper/bin/../lib/jackson-annotations-2.10.3.jar:/opt/zookeeper/bin/../lib/commons-cli-1.2.jar:/opt/zookeeper/bin/../lib/audience-annotations-0.5.0.jar:/opt/zookeeper/bin/../zookeeper-*.jar:/opt/zookeeper/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt/zookeeper/bin/../conf:
2020-06-27 17:51:04,120 [myid:] - INFO [main:Environment@109] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2020-06-27 17:51:04,120 [myid:] - INFO [main:Environment@109] - Client environment:java.io.tmpdir=/tmp
2020-06-27 17:51:04,120 [myid:] - INFO [main:Environment@109] - Client environment:java.compiler=<NA>
2020-06-27 17:51:04,120 [myid:] - INFO [main:Environment@109] - Client environment:os.name=Linux
2020-06-27 17:51:04,120 [myid:] - INFO [main:Environment@109] - Client environment:os.arch=amd64
2020-06-27 17:51:04,120 [myid:] - INFO [main:Environment@109] - Client environment:os.version=3.10.0-514.el7.x86_64
2020-06-27 17:51:04,121 [myid:] - INFO [main:Environment@109] - Client environment:user.name=root
2020-06-27 17:51:04,121 [myid:] - INFO [main:Environment@109] - Client environment:user.home=/root
2020-06-27 17:51:04,121 [myid:] - INFO [main:Environment@109] - Client environment:user.dir=/root
2020-06-27 17:51:04,121 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.free=52MB
2020-06-27 17:51:04,122 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.max=228MB
2020-06-27 17:51:04,122 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.total=57MB
2020-06-27 17:51:04,124 [myid:] - INFO [main:ZooKeeper@868] - Initiating client connection, connectString=192.168.10.1:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@ee7d9f1
2020-06-27 17:51:04,128 [myid:] - INFO [main:X509Util@79] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2020-06-27 17:51:04,137 [myid:] - INFO [main:ClientCnxnSocket@237] - jute.maxbuffer value is 4194304 Bytes
2020-06-27 17:51:04,149 [myid:] - INFO [main:ClientCnxn@1653] - zookeeper.request.timeout value is 0. feature enabled=
Welcome to ZooKeeper!
JLine support is enabled
[zk: 192.168.10.1:2181(CONNECTING) 0]
[zk: 192.168.10.1:2181(CONNECTING) 0]
[zk: 192.168.10.1:2181(CONNECTING) 0] get /2020-06-27 17:51:14,239 [myid:192.168.10.1:2181] - INFO [main-SendThread(192.168.10.1:2181):ClientCnxn$SendThread@1112] - Opening socket connection to server eskibana/192.168.10.1:2181. Will not attempt to authenticate using SASL (unknown error)
2020-06-27 17:51:14,245 [myid:192.168.10.1:2181] - INFO [main-SendThread(192.168.10.1:2181):ClientCnxn$SendThread@959] - Socket connection established, initiating session, client: /192.168.10.1:48198, server: eskibana/192.168.10.1:2181
2020-06-27 17:51:14,259 [myid:192.168.10.1:2181] - INFO [main-SendThread(192.168.10.1:2181):ClientCnxn$SendThread@1394] - Session establishment complete on server eskibana/192.168.10.1:2181, sessionid = 0x2000180e3d60001, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: 192.168.10.1:2181(CONNECTED) 1]
[zk: 192.168.10.1:2181(CONNECTED) 1] get /test
haha
[zk: 192.168.10.1:2181(CONNECTED) 2]
現在zookeeper環境搭建H好
部署kafka
tar zxf kafka_2.12-2.5.0.tgz -C /opt
cd /opt/
mv kafka_2.12-2.5.0/ ./kafka
cd kafka/
mkdir /opt/kafka/logs #創建存放日誌目錄
vim /opt/kafka/config/server.properties
broker.id=1
listeners=PLAINTEXT://192.168.10.2:9092
log.dirs=/opt/kafka/logs
log.retention.hours=24
zookeeper.connect=192.168.10.1:2181,192.168.10.7:2181,192.168.10.2:2181
把kafka目錄複製到其他節點
scp -rp /opt/kafka [email protected]:/opt/
scp -rp /opt/kafka [email protected]:/opt/
需要修改2處地方
broker.id=2
listeners=PLAINTEXT://192.168.10.1:9092
broker.id=3
listeners=PLAINTEXT://192.168.10.7:9092
測試啓動
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
[2020-06-27 18:10:27,192] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)
在最後一行會出現id=1,started 字樣說明已經開啓成功！
這時候可以放到後臺
查看日誌觀察啓動情況
/opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties
tail -f /opt/kafka/logs/server.log
[2020-06-27 18:11:38,237] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)
其他節點啓動
/opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties
[2020-06-27 18:20:03,940] INFO [KafkaServer id=2] started (kafka.server.KafkaServer)
/opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties
[2020-06-27 18:20:59,501] INFO [KafkaServer id=3] started (kafka.server.KafkaServer)
創建主題topic
[root@localhost ~]# /opt/kafka/bin/kafka-topics.sh --create --zookeeper 192.168.10.1:2181,192.168.10.2:2181,192.168.10.7:2181 --partitions 3 --replication-factor 3 --topic kafkatest
Created topic kafkatest.
[root@localhost ~]# /opt/kafka/bin/kafka-topics.sh --describe --zookeeper 192.168.10.1:2181,192.168.10.2:2181,192.168.10.7:2181 --topic kafkatest
Topic: kafkatest PartitionCount: 3 ReplicationFactor: 3 Configs:
Topic: kafkatest Partition: 0 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1
Topic: kafkatest Partition: 1 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2
Topic: kafkatest Partition: 2 Leader: 2 Replicas: 2,1,3 Isr: 2,1,3
##登錄測試
[root@localhost ~]# /opt/kafka/bin/kafka-console-producer.sh --broker-list 192.168.10.1:2181,192.168.10.2:2181,192.168.10.7:2181 --topic kafkatest
>
>hello
>hi
>lalalala
注：老版本消費者測試用：/opt/kafka/bin/kafka-console-consumer.sh --zookeeper 192.168.10.1:2181,192.168.10.2:2181,192.168.10.7:2181 --topic message --from-beginning
出現狀況：
[root@filebeat opt]# /opt/kafka/bin/kafka-console-consumer.sh --zookeeper 192.168.10.1:2181,192.168.10.2:2181,192.168.10.7:2181 --topic message --from-beginning
zookeeper is not a recognized option
Option Description
------ -----------
--bootstrap-server <String: server to REQUIRED: The server(s) to connect to.
connect to>
--consumer-property <String: A mechanism to pass user-defined
consumer_prop> properties in the form key=value to
the consumer.
--consumer.config <String: config file> Consumer config properties file. Note
that [consumer-property] takes
precedence over this config.
--enable-systest-events Log lifecycle events of the consumer
in addition to logging consumed
messages. (This is specific for
system tests.)
--formatter <String: class> The name of a class to use for
formatting kafka messages for
display. (default: kafka.tools.
DefaultMessageFormatter)
--from-beginning If the consumer does not already have
an established offset to consume
from, start with the earliest
message present in the log rather
than the latest message.
--group <String: consumer group id> The consumer group id of the consumer.
--help Print usage information.
--isolation-level <String> Set to read_committed in order to
filter out transactional messages
which are not committed. Set to
read_uncommitted to read all
messages. (default: read_uncommitted)
--key-deserializer <String:
deserializer for key>
--max-messages <Integer: num_messages> The maximum number of messages to
consume before exiting. If not set,
consumption is continual.
--offset <String: consume offset> The offset id to consume from (a non-
negative number), or 'earliest'
which means from beginning, or
'latest' which means from end
(default: latest)
--partition <Integer: partition> The partition to consume from.
Consumption starts from the end of
the partition unless '--offset' is
specified.
--property <String: prop> The properties to initialize the
message formatter. Default
properties include:
print.timestamp=true|false
print.key=true|false
print.value=true|false
key.separator=<key.separator>
line.separator=<line.separator>
key.deserializer=<key.deserializer>
value.deserializer=<value.
deserializer>
Users can also pass in customized
properties for their formatter; more
specifically, users can pass in
properties keyed with 'key.
deserializer.' and 'value.
deserializer.' prefixes to configure
their deserializers.
--skip-message-on-error If there is an error when processing a
message, skip it instead of halt.
--timeout-ms <Integer: timeout_ms> If specified, exit if no message is
available for consumption for the
specified interval.
--topic <String: topic> The topic id to consume on.
--value-deserializer <String:
deserializer for values>
--version Display Kafka version.
--whitelist <String: whitelist> Regular expression specifying
whitelist of topics to include for
consumption.
查閱資料發現新版本已經把上面的啓動方式刪除
新版本使用時已經不支持上面這種寫法：
參考
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.10.1:9092 --topic message --from-beginning
[root@filebeat opt]# /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.10.7:9092 --topic message --from-beginning
hello
hi
lalalala
[root@eskibana ~]# /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.10.1:9092 --topic message --from-beginning
hello
hi
lalalala
安裝部署es集羣
下載安裝es
wget https://mirror.tuna.tsinghua.edu.cn/elasticstack/yum/elastic6.x/6.6.0/elasticsearch-6.6.0.rpm
#安裝
rpm -ivh elasticsearch-6.6.0.rpm
修改配置文件
grep -Ev '^#|^$' /etc/elasticsearch/elasticsearch.yml
cluster.name: ES_cluster
node.name: node-1
path.data: /data/es #數據存放目錄
path.logs: /var/log/elasticsearch #日誌存放目錄
bootstrap.memory_lock: true #開啓內存鎖定,會與/etc/elasticsearch/jvm.options有關聯
network.host: 192.168.10.1
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.10.1", "192.168.10.2"]
discovery.zen.minimum_master_nodes: 2
創建目錄，修改權限
mkdir -p /data/elasticsearch
chown -R elasticsearch:elasticsearch /data/elasticsearch/
若開啓了內存鎖定，需要添加修改這個與內存分配有關的文件
vim /etc/elasticsearch/jvm.options
-Xms2g #分配最小內存 ，根據你自己的而定=你物理內存的一半
-Xmx2g #分配最大內存,官方推薦爲物理內存的一半，但最大爲32G，超過了32G,反而可能會有副作用
接着添加這一行，具體爲什麼可以去參考官方文檔
添加完成以後重啓OK
systemctl edit elasticsearch
添加：
[Service]
LimitMEMLOCK=infinity
F2 回車 保存退出
systemctl daemon-reload
systemctl start elasticsearch
node-2配置和node-1一模一樣
es羣集配置完會通過單播的形式在節點之間互相發現，所以節點IP不要寫錯
node-2安裝完成以後，複製主配置文件
scp -p /etc/elasticsearch/elasticsearch.yml [email protected]:/etc/elasticsearch/
scp -rp /data/ [email protected]:/
修改
node.name: node-2
network.host: 192.168.10.2
啓動elasticsearch
安裝部署kibana
# yum 安裝
rpm -ivh kibana-6.6.0-x86_64.rpm
修改配置文件
server.port: 5601
server.host: "192.168.10.1"
server.name: "db01"
elasticsearch.hosts: ["http://192.168.10.1:9200"]
保存退出
啓動kibana
systemctl start kibana
安裝部署filebeat
安裝filebeat和nginx
rpm -ivh filebeat-6.6.0-x86_64.rpm
yum -y install nginx
systemctl start nignx
修改filebeat文件
添加收集 nignx的日誌信息；定義發送的目的地kafka集羣
vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["error"]
setup.template.settings:
index.number_of_shards: 5
setup.kibana:
hosts: ["192.168.10.1"]
output.kafka:
hosts: ["192.168.10.1:9092","192.168.10.2:9092","192.168.10.7:9092"]
topic: "elklog"
安裝logstash
rpm -ivh logstash-6.6.0.rpm
修改logstash文件
從kafka收集日誌信息，然後過濾傳輸到es
input {
kafka {
bootstrap_servers => "192.168.10.1:9092" #zookeeper羣集任何一個就可以
topics => ["elklog"]
group_id =>'logstrash'
codec => "json"
}
}
filter {
mutate {
convert => ["upstream_time","float"]
convert => ["request_time","float"]
}
}
output {
stdout {}
if "access" in [tags] {
elasticsearch {
hosts => ["http://192.168.10.1:9200"]
index => "nginx_access-%{+YYYY.MM}"
manage_template => false
}
}
if "error" in [tags] {
elasticsearch {
hosts => ["http://192.168.10.1:9200"]
index => "nginx_error-%{+YYYY.MM}"
manage_template => false
}
}
}
重啓filebeat，logstash
systemctl start filebeat
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/kafka.conf & #放到後臺執行
壓力測試
307 ab -n 1000 -c 100 http://192.168.10.7
308 ab -n 1000 -c 100 http://192.168.10.7/
309 ab -n 1000 -c 100 http://192.168.10.7/aaa
310 ab -n 1000 -c 100 http://192.168.10.7/bbb
![[外鏈圖片轉存失敗,源站可能有防盜鏈機制,建議將圖片保存下來直接上傳(img-KExxmfya-1593510604844)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\image-20200627222900316.png)]](https://pic1.xuehuaimg.com/proxy/csdn/https://img-blog.csdnimg.cn/20200630175633958.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzgxNTE0MA==,size_16,color_FFFFFF,t_70)
nginx配置文件默認支持json格式日誌
添加內容如下
log_format log_json '{ "@timestamp": "$time_local", '
'"remote_addr": "$remote_addr", '
'"referer": "$http_referer", '
'"request": "$request", '
'"status": $status, '
'"bytes": $body_bytes_sent, '
'"agent": "$http_user_agent", '
'"x_forwarded": "$http_x_forwarded_for", '
'"up_addr": "$upstream_addr",'
'"up_host": "$upstream_http_host",'
'"up_resp_time": "$upstream_response_time",'
'"request_time": "$request_time"'
' }';
最後記得把nignx默認開啓的日誌格式main改log_json
重新啓動nginx
重啓壓測，查看日誌
