節點1:10.116.186.6 jdk+zookeeper+kafka
節點2:10.116.186.7 jdk+zookeeper+kafka
節點3:10.116.186.8 jdk+zookeeper+kafka
OS：CentOS7.6 min x86_64
1. 軟件安裝
節點：節點1，節點2，節點3
說明：軟件包（rpm）爲我們企業內部自己打的包，主要是爲了統一安裝目錄，日誌目錄，數據目錄等。
yum list jdk zookeeper kafka
jdk.x86_64 1.8.0_121-1.eju.el7 ejusofts
kafka.x86_64 2.12-1.eju.el7 ejusofts
zookeeper.x86_64 3.4.10-1.eju.el7 ejusofts
yum -y install jdk zookeeper kafka
2. zookeeper集羣
2.1 配置
節點：節點1，節點2，節點3
grep -v -e ^$ -e ^# zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/lib/zookeeper/data
clientPort=2181
server.1=10.116.186.6:2888:3888
server.2=10.116.186.7:2888:3888
server.3=10.116.186.8:2888:3888
節點： 節點1 10.116.186.6
mkdir -p /var/lib/zookeeper/data
echo 1 > /var/lib/zookeeper/data/myid
節點： 節點2 10.116.186.7
mkdir -p /var/lib/zookeeper/data
echo 2 > /var/lib/zookeeper/data/myid
節點： 節點3 10.116.186.8
mkdir -p /var/lib/zookeeper/data
echo 3 > /var/lib/zookeeper/data/myid
# 2.2 啓動節點，並驗證集羣
# 節點3,2,1 （順序無所謂）根據實際安裝目錄執行命令
/opt/app/zookeeper/bin/zkServer.sh start
# 節點：1
/opt/app/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/bin/../conf/zoo.cfg
Mode: follower
# 節點：2
/opt/app/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/bin/../conf/zoo.cfg
Mode: follower
# 節點：3
/opt/app/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/bin/../conf/zoo.cfg
Mode: leader
2.3 連接測試
/opt/app/zookeeper/bin/zkCli.sh -server 10.116.186.6:2181
Connecting to 10.116.186.6:2181
2019-04-26 17:22:46,841 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2019-04-26 17:22:46,843 [myid:] - INFO [main:Environment@100] - Client environment:host.name=fangkafka-shylf-3
2019-04-26 17:22:46,843 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_121
2019-04-26 17:22:46,845 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-04-26 17:22:46,845 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/opt/app/install/jdk1.8.0_121/jre
2019-04-26 17:22:46,845 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/opt/app/zookeeper/bin/../build/classes:/opt/app/zookeeper/bin/../build/lib/*.jar:/opt/app/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/app/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/opt/app/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/opt/app/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/app/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/app/zookeeper/bin/../zookeeper-3.4.10.jar:/opt/app/zookeeper/bin/../src/java/lib/*.jar:/opt/app/zookeeper/bin/../conf:.:/opt/app/jdk/jre/lib/rt.jar:/opt/app/jdk/lib/dt.jar:/opt/app/jdk/lib/tools.jar
2019-04-26 17:22:46,845 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-04-26 17:22:46,845 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-04-26 17:22:46,845 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2019-04-26 17:22:46,845 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2019-04-26 17:22:46,846 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2019-04-26 17:22:46,846 [myid:] - INFO [main:Environment@100] - Client environment:os.version=3.10.0-957.10.1.el7.x86_64
2019-04-26 17:22:46,846 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root
2019-04-26 17:22:46,846 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root
2019-04-26 17:22:46,846 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/opt/app/install/zookeeper-3.4.10/conf
2019-04-26 17:22:46,847 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=10.116.186.6:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@799f7e29
Welcome to ZooKeeper!
2019-04-26 17:22:46,866 [myid:] - INFO [main-SendThread(10.116.186.6:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server 10.116.186.6/10.116.186.6:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2019-04-26 17:22:46,954 [myid:] - INFO [main-SendThread(10.116.186.6:2181):ClientCnxn$SendThread@876] - Socket connection established to 10.116.186.6/10.116.186.6:2181, initiating session
2019-04-26 17:22:46,983 [myid:] - INFO [main-SendThread(10.116.186.6:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server 10.116.186.6/10.116.186.6:2181, sessionid = 0x16a58ed1b1c0000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: 10.116.186.6:2181(CONNECTED) 0]
3. kafka集羣
3.1 配置
grep -v -e ^$ -e ^# server.properties
broker.id=1 # broker.id=2 #broker.id=3 各節點不同值
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/var/lib/kafka/logs
num.partitions=3
num.recovery.threads.per.data.dir=1
log.retention.hours=168
message.max.byte=5242880
default.replication.factor=2
replica.fetch.max.bytes=5242880
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.116.186.6:2181,10.116.186.7:2181,10.116.186.8:2181
zookeeper.connection.timeout.ms=6000
mkdir -p /var/lib/kafka/logs
3.2 啓動並驗證
# 節點1,2，3
# 根據實際目錄執行
/opt/app/kafka/bin/kafka-server-start.sh -daemon /opt/app/kafka/config/server.properties
jps
22119 Kafka
21739 QuorumPeerMain
22239 Jps
3.3 topic驗證
創建topic test
/opt/app/kafka/bin/kafka-topics.sh --zookeeper 10.116.186.6:2181 --create --replication-factor 2 --partitions 1 --topic test
[OUTPUT]
Created topic "test".
查看topic test
/opt/app/kafka/bin/kafka-topics.sh --zookeeper 10.116.186.6:2181 --list
[OUTPUT]
test
生產消息
/opt/app/kafka/bin/kafka-console-producer.sh --broker-list 10.116.186.6:9092 --topic test
[INPUT]
hello
消費消息[老的測試方式]
/opt/app/kafka/bin/kafka-console-consumer.sh --zookeeper 10.116.186.6:2181 --topic test --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
[OUTPUT]
hello
消費消息[新方式]
/opt/app/kafka/bin/kafka-console-consumer.sh --bootstrap-server 10.116.186.6:9092 --topic test --from-beginning
hello