手把手教你實現kafka集羣搭建

選擇搭建主機
選擇node1,node2,node3搭建集羣

1.解壓

[root@node1 apps]# tar -zxvf kafka_2.11-0.11.0.3.tgz -C /opt

2.修改配置

[root@node1 config]# vim server.properties

2.1將log目錄修改爲/kafaka-logs

2.2修改zookeeper

2.3編寫啓動腳本

nohup bin/kafka-server-start.sh   config/server.properties > kafka.log 2>&1 &

放到根目錄下

授權腳本
將軟件發送到node2和node3上

[root@node1 opt]# scp -r kafka_2.11-0.11.0.3/ node3:`pwd`
[root@node1 opt]# scp -r kafka_2.11-0.11.0.3/ node2:`pwd`

2.4修改node2和node3的service.properties

[root@node2 ~]# cd /opt/kafka_2.11-0.11.0.3/config/
[root@node2 config]# vim server.properties

node2的改爲

node3改爲2

3.啓動

node1,node2,node3

[root@node1 kafka_2.11-0.11.0.3]# ./startkafka.sh
[root@node2 kafka_2.11-0.11.0.3]# ./startkafka.sh
[root@node3 kafka_2.11-0.11.0.3]# ./startkafka.sh

啓動集羣成功

4.測試

[root@node1 bin]# ./kafka-topics.sh --zookeeper node2,node3,node4 --create --topic t0210 --partitions 3 --replication-factor 3
Created topic "t0210".
[root@node1 bin]# ./kafka-topics.sh --zookeeper node2,node3,node4 --list
t0210

讓node1作爲生產者生產數據
node2作爲消費者消費數據

[root@node1 bin]# ./kafka-console-producer.sh --broker-list node1:9092,node2:9092,node3:9092 --topic t0210
>

[root@node2 bin]# ./kafka-console-consumer.sh --bootstrap-server node1:9092,node2:9092,node3:9092 --topic t0210

查看數據

[root@node3 bin]# ./kafka-consumer-groups.sh --bootstrap-server node1:9092,node2:9092,node3:9092 --list
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

console-consumer-91847
[root@node3 bin]# ./kafka-consumer-groups.sh --bootstrap-server node1:9092,node2:9092,node3:9092 --describe --group console-consumer-91847
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).


TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
t0210                          0          0               0               0          consumer-1-c0e7bb36-59aa-40ad-a832-e79648def3d4   /192.168.40.62                 consumer-1
t0210                          1          1               1               0          consumer-1-c0e7bb36-59aa-40ad-a832-e79648def3d4   /192.168.40.62                 consumer-1
t0210                          2          1               1               0          consumer-1-c0e7bb36-59aa-40ad-a832-e79648def3d4   /192.168.40.62                 consumer-1
[root@node3 bin]#