Kafka集羣安裝和使用

Kafka集羣安裝

安裝Kafka之前需安裝Zookeeper：https://hubaoquan.cn/zookeeperanzhuangjijiandanshiyong/

1.集羣規劃

ip:10.7.68.213 主機名：ecs-az3-yc-0020 實例id: broker.id=0
ip:10.7.68.188 主機名：ecs-az3-yc-0021 實例id: broker.id=1
ip:10.7.68.197 主機名：ecs-az3-yc-0022 實例id: broker.id=2

端口：9092
kafka版本爲kafka_2.12-2.2.0.tgz

2.jar包下載

[root@ecs-az3-yc-0020 module]$ wget http://soft.hubaoquan.cn/kafka_2.12-2.2.0.tgz

3.kafka集羣部署

1）解壓安裝包
[root@ecs-az3-yc-0020 module]$ tar -zxvf kafka_2.12-2.2.0.tgz
2）在/opt/module/kafkHa目錄下創建logs文件夾
[root@ecs-az3-yc-0020 kafka_2.12-2.2.0]$ mkdir logs
3）修改配置文件
[root@ecs-az3-yc-0020 kafka_2.12-2.2.0]$ cd config/
[root@ecs-az3-yc-0020 config]$ vi server.properties
修改*標註內容其他根據主機環境，也可以保持默認：

#************broker.id的全局唯一編號，不能重複
broker.id=0

#開啓刪除topic功能
delete.topic.enable=true

#處理網絡請求的線程數量​
num.network.threads=3
​
#用來處理磁盤IO的現成數量
num.io.threads=8
​
#發送套接字的緩衝區大小
socket.send.buffer.bytes=102400
​
#接收套接字的緩衝區大小
socket.receive.buffer.bytes=102400
​
#請求套接字的緩衝區大小
socket.request.max.bytes=104857600
​
#*****kafka運行日誌存放的路徑
log.dirs=/opt/module/kafka_2.12-2.2.0/logs
​
#topic在當前broker上的分區個數
num.partitions=1
​
#用來恢復和清理data下數據的線程數量
num.recovery.threads.per.data.dir=1
​
#segment文件保留的最長時間，超時將被刪除
log.retention.hours=168
​
#*****配置連接Zookeeper集羣地址
zookeeper.connect=10.7.68.197:2181,10.7.68.188:2181,10.7.68.213:2181

4）配置環境變量
[root@ecs-az3-yc-0020 module]$ sudo vi /etc/profile
​添加以下內容：

#KAFKA_HOME
export KAFKA_HOME=/opt/module/kafka_2.12-2.2.0
export PATH=$PATH:$KAFKA_HOME/bin

使環境變量生效：
[root@ecs-az3-yc-0020 module]$ source /etc/profile

5）分發安裝包
[root@ecs-az3-yc-0020 module]$ scp –r root@ecs-az3-yc-0021:/opt/module
[root@ecs-az3-yc-0020 module]$ scp –r root@ecs-az3-yc-0022:/opt/module

6）分別在ecs-az3-yc-0021和ecs-az3-yc-0022上修改配置文件/opt/module/kafka_2.12-2.2.0/config/server.properties中的broker.id=1、broker.id=2
注：broker.id不得重複

7）啓動集羣
依次在ecs-az3-yc-0020、ecs-az3-yc-0021、ecs-az3-yc-0022節點上啓動kafka

[root@ecs-az3-yc-0020 kafka_2.12-2.2.0]$ bin/kafka-server-start.sh config/server.properties &
[root@ecs-az3-yc-0021 kafka_2.12-2.2.0]$ bin/kafka-server-start.sh config/server.properties &
[root@ecs-az3-yc-0022 kafka_2.12-2.2.0]$ bin/kafka-server-start.sh config/server.properties &

8）關閉集羣

[root@ecs-az3-yc-0020 kafka_2.12-2.2.0]$ bin/kafka-server-stop.sh stop
[root@ecs-az3-yc-0021 kafka_2.12-2.2.0]$ bin/kafka-server-stop.sh stop
[root@ecs-az3-yc-0022 kafka_2.12-2.2.0]$ bin/kafka-server-stop.sh stop

4.kafka命令行操作

1）查看當前服務器中的所有topic
[root@ecs-az3-yc-0020 kafka_2.12-2.2.0]$ bin/kafka-topics.sh --zookeeper ecs-az3-yc-0020:2181 --list
2）創建topic

[root@ecs-az3-yc-0020 kafka_2.12-2.2.0]# bin/kafka-topics.sh --zookeeper ecs-az3-yc-0020:2181 --create --replication-factor 3 --partitions 1 --topic hbqtest
Created topic hbqtest.

選項說明：
–topic 定義topic名
–replication-factor 定義副本數
–partitions 定義分區數
4)查看topic新增了hbqtest

[root@ecs-az3-yc-0020 kafka_2.12-2.2.0]# bin/kafka-topics.sh --zookeeper ecs-az3-yc-0020:2181 --li​st
hbqtest
nts-foundation-bids

5）生產者ecs-az3-yc-0020發送消息

[root@ecs-az3-yc-0020 kafka_2.12-2.2.0]# bin/kafka-console-producer.sh --broker-list 10.7.68.213:9092 --topic hbqtest
>hello world
>hi kafka
>^C

5）消費者ecs-az3-yc-0022接收消息

從當前時間接收消息

[root@ecs-az3-yc-0022 kafka_2.12-2.2.0]# bin/kafka-console-consumer.sh --bootstrap-server 10.7.68.213:9092  --topic hbqtest
hello world
hi kafka

從初始位置接收消息

[root@ecs-az3-yc-0022 kafka_2.12-2.2.0]# bin/kafka-console-consumer.sh --bootstrap-server 10.7.68.213:9092 --from-beginning --topic hbqtest

–from-beginning：會把first主題中以往所有的數據都讀取出來。根據業務場景選擇是否增加該配置。

顯示key消費

[root@ecs-az3-yc-0022 kafka_2.12-2.2.0]# bin/kafka-console-consumer.sh --bootstrap-server  10.7.68.213:9092 --from-beginning --property print.key=true  --topic hbqtest
null   hello world
null   hi kafka

6）查看某個Topic的詳情

[root@ecs-az3-yc-0021 kafka_2.12-2.2.0]# bin/kafka-topics.sh --zookeeper 10.7.68.213:2181 --describe --topic hbqtest
Topic:hbqtest PartitionCount:1 ReplicationFactor:3 Configs:
Topic:hbqtest Partition: 0 Leader: 3 Replicas: 3,1,0 Isr: 3,1,0

3）刪除topic

[root@ecs-az3-yc-0021 kafka_2.12-2.2.0]# bin/kafka-topics.sh --zookeeper 10.7.68.197:2181 --delete --topic hbqtest
Topic hbqtest is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.

需要server.properties中設置delete.topic.enable=true否則只是標記刪除或者直接重啓。

4)刪除topic後消費者會因找不到topic而報錯

[20xx-04-24 17:00:26,545] WARN [Consumer clientId=consumer-1, groupId=console-consumer-14567] Received unknown topic or partition error in fetch for partition hbqtest-0 (org.apache.kafka.clients.consumer.internals.Fetcher)
[20xx-04-24 17:00:26,545] WARN [Consumer clientId=consumer-1, groupId=console-consumer-14567] Error while fetching metadata with correlation id 662 : {hbqtest=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)