kafka/zookeeper集羣部署以及常用命令學習

環境準備

三臺虛擬機（192.168.209.188，192.168.209.189，192.168.209.190）
kafka_2.10-0.10.0.1
zookeeper-3.4.12.tar.gz

下載地址

Zookeeper:
http://mirror.bit.edu.cn/apache/zookeeper/current/

Scala:
http://www.scala-lang.org/download/2.11.8.html

Kafka:
http://kafka.apache.org/downloads

部署zookeeper集羣

下載解壓後zookeeper與kafka目錄爲 /opt/software

[root@hadoop002 software]# ll
總用量 1108584
lrwxrwxrwx.  1 root   root          19 2月  28 22:30 kafka -> kafka_2.10-0.10.0.1
drwxr-xr-x.  7 root   root         118 3月   2 15:39 kafka_2.10-0.10.0.1
-rw-r--r--.  1 root   root    32609012 8月   9 2016 kafka_2.10-0.10.0.1.tgz
drwxr-xr-x. 11 root   root        4096 3月   2 15:11 zookeeper
-rw-r--r--.  1 root   root    36667596 4月  25 2018 zookeeper-3.4.12.tar.gz

進入zookeeper/conf配置zookeeper

[root@hadoop002 software]# cd zookeeper/conf
[root@hadoop002 conf]# cp zoo_sample.cfg zoo.cfg
[root@hadoop002 conf]# ll
總用量 16
-rw-rw-r--. 1 root root  535 3月  27 2018 configuration.xsl
-rw-rw-r--. 1 root root 2161 3月  27 2018 log4j.properties
-rw-r--r--. 1 root root 1042 3月   2 15:14 zoo.cfg
-rw-rw-r--. 1 root root  922 3月  27 2018 zoo_sample.cfg
[root@hadoop002 conf]#

[root@hadoop002 conf]# vi zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/opt/software/zookeeper/data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

server.1=192.168.209.188:2888:3888
server.2=192.168.209.189:2888:3888
server.3=192.168.209.190:2888:3888
~                                                                                                                                                                          
"zoo.cfg" 32L, 1042C

三臺虛擬機zookeeper配置一樣，可以scp過去

[root@hadoop002 conf]# cd ../
[root@hadoop002 zookeeper]#  mkdir data
[root@hadoop002 zookeeper]# touch data/myid
[root@hadoop002 zookeeper]# echo 1 > data/myid
[root@hadoop002 zookeeper]#
[root@hadoop002 zookeeper]# echo 2 > data/myid
[root@hadoop002 zookeeper]# echo 3 > data/myid
###切記不可echo 3>data/myid,將>前後空格保留,否則無法將 3 寫入myid文件

接下來啓動zookeeper集羣

[root@hadoop002 bin]# ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/software/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@hadoop002 bin]# 

[root@hadoop002 bin]# ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/software/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@hadoop002 bin]# 

[root@hadoop002 bin]# ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/software/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@hadoop002 bin]#

查看狀態

[root@hadoop002 bin]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/software/zookeeper/bin/../conf/zoo.cfg
Mode: follower
[root@hadoop002 bin]# 

[root@hadoop002 bin]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/software/zookeeper/bin/../conf/zoo.cfg
Mode: leader
[root@hadoop002 bin]# 

[root@hadoop002 bin]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/software/zookeeper/bin/../conf/zoo.cfg
Mode: follower
[root@hadoop002 bin]# 
其中192.168.209.189這臺被選舉爲leader，由於我虛擬機是複製的，主機名沒改，所以看起來都一樣

進入客戶端

[zk: localhost:2181(CONNECTED) 1] ls /
[zookeeper, kafka]
[zk: localhost:2181(CONNECTED) 2] help
ZooKeeper -server host:port cmd args
        stat path [watch]
        set path data [version]
        ls path [watch]
        delquota [-n|-b] path
        ls2 path [watch]
        setAcl path acl
        setquota -n|-b val path
        history 
        redo cmdno
        printwatches on|off
        delete path [version]
        sync path
        listquota path
        rmr path
        get path [watch]
        create [-s] [-e] path data acl
        addauth scheme auth
        quit 
        getAcl path
        close 
        connect host:port
[zk: localhost:2181(CONNECTED) 3] 

下載kafka之前要下載scala
scala安裝部署好後
下載基於Scala 2.11的kafka版本爲0.10.0.1

創建logs目錄和修改server.properties
[root@hadoop002 software]# cd kafka
[root@hadoop002 kafka]# mkdir logs
[root@hadoop002 kafka]# cd config/
[root@hadoop002 config]# vi server.properties
broker.id=1
port=9092
host.name=192.168.209.188
log.dirs=/opt/software/kafka/logs
zookeeper.connect=192.168.209.188:2181,192.168.209.189:2181,192.168.209.190:2181/kafka

環境變量
[root@hadoop001 config]# vi /etc/profile
export KAFKA_HOME=/opt/software/kafka
export PATH=$KAFKA_HOME/bin:$PATH
[root@hadoop001 config]# source /etc/profile

啓動它的啓動腳本

[root@hadoop002 bin]# ./kafka-server-start.sh
USAGE: ./kafka-server-start.sh [-daemon] server.properties [--override property=value]*
[root@hadoop002 bin]#
提示可以用-daemon使用守護進程啓動   後面跟上配置文件
[root@hadoop002 bin]# ./kafka-server-start.sh -daemon ../config/server.properties
[root@hadoop002 bin]# jps
8676 QuorumPeerMain
9252 Jps
9193 Kafka
[root@hadoop002 bin]#
kafka已經啓動可以使用bin/kafka-server-stop.sh停止

在一個終端，啓動Producer，並向我們上面創建的名稱爲my-replicated-topic5的Topic中生產消息，執行如下腳本：

bin/kafka-console-producer.sh \
--broker-list 192.168.209.188:9092,192.168.209.189:9092,192.168.209.190:9092 --topic my-topic

在另一個終端，啓動Consumer，並訂閱我們上面創建的名稱爲my-replicated-topic5的Topic中生產的消息，執行如下腳本：

bin/kafka-console-consumer.sh \
--zookeeper 192.168.209.188:2181,192.168.209.189:2181,192.168.209.190:2181/kafka \
--from-beginning --topic my-topic

可以在Producer終端上輸入字符串消息行，就可以在Consumer終端上看到消費者消費的消息內容

生產

[root@hadoop002 kafka]# bin/kafka-console-producer.sh \
> --broker-list 192.168.209.188:9092,192.168.209.189:9092,192.168.209.190:9092 --topic my-topic
hello kafka I am faker
[2019-03-02 21:08:14,090] WARN Error while fetching metadata with correlation id 0 : {my-topic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

消費

[root@hadoop002 kafka]# bin/kafka-console-consumer.sh \
> --zookeeper 192.168.209.188:2181,192.168.209.189:2181,192.168.209.190:2181/kafka \
> --from-beginning --topic my-topic
hello kafka I am faker

核心概念

創建主題

[root@hadoop002 kafka]# bin/kafka-topics.sh --create \
> --zookeeper 192.168.209.188:2181,192.168.209.189:2181,192.168.209.190:2181/kafka \
> --replication-factor 3 --partitions 3 --topic topic-first
Created topic "topic-first".

Topic: topic-first 主題  1----10
partitions:  3個分區  下標是從0開始的   物理上的分區
replication: 3個副本 指的是一個分區被複制3份
3臺機器：
        192.168.209.188   192.168.209.189   192.168.209.190
0分區: topic-first-0  	topic-first-0 	topic-first-0
1分區: topic-first-1 	 topic-first-1 	topic-first-1
2分區: topic-first-2  	topic-first-2 	topic-first-2 

描述

[root@hadoop002 bin]# ./kafka-topics.sh --describe \
> --zookeeper 192.168.209.188:2181,192.168.209.189:2181,192.168.209.190:2181/kafka \
> --topic topic-first
Topic:topic-first       PartitionCount:3        ReplicationFactor:3     Configs:
        Topic: topic-first      Partition: 0    Leader: 3       Replicas: 3,1,2 Isr: 3,1,2
        Topic: topic-first      Partition: 1    Leader: 1       Replicas: 1,2,3 Isr: 1,2,3
        Topic: topic-first      Partition: 2    Leader: 2       Replicas: 2,3,1 Isr: 2,3,1
[root@hadoop002 bin]#

Partition: 0
Leader: 3 指的是broker.id=3 讀寫
Replicas: 複製該分區數據的節點列表，第一位代表leader 靜態表述
Isr: in-sync Replicas 正在複製的副本節點列表 動態表述
當leader掛了，會從這個列表選舉出leader

broker:Kafka實例的節點

一般生產上:
–replication-factor 3 --partitions 3
–partitions 取決於你的broker數量

consumer group:

一個消費組可以包含一個或多個消費者，分區只能被一個消費組的其中一個消費者去消費
正常在企業開發使用多分區方式去提高計算的能力

常用命令

### 創建
 ./kafka-topics.sh --create \
--zookeeper 192.168.209.188:2181,192.168.209.189:2181,192.168.209.190:2181/kafka \
--replication-factor 3 --partitions 3 --topic test

### 查看
./kafka-topics.sh --list \
--zookeeper 192.168.209.188:2181,192.168.209.189:2181,192.168.209.190:2181/kafka 

### 查看某個topic
 ./kafka-topics.sh --describe \
--zookeeper 192.168.209.188:2181,192.168.209.189:2181,192.168.209.190:2181/kafka \
--topic test

### 刪除
 ./kafka-topics.sh --delete \
--zookeeper 192.168.209.188:2181,192.168.209.189:2181,192.168.209.190:2181/kafka \
--topic test
Topic test is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.

刪除後還能發送和接受數據，因爲這是假刪除，即使將delete.topic.enable=true配置好也是不行的
這時就要將kafka的元數據以及數據刪除，kafka元數據在zookeeper中 數據在磁盤
進入zookeeper客戶端，刪除元數據
rmr /config/topics/topic-first
rmr /brokers/topics/topic-first
rmr /admin/delete_topics/topic-first
刪除數據
rm -rf $KAFKA_HOME/logs/topic-first-*
這樣就徹底刪完了

調整/修改
 ./kafka-topics.sh --alter \
--zookeeper 192.168.209.188:2181,192.168.209.189:2181,192.168.209.190:2181/kafka \
--partitions 4 --topic topic-two
這裏的分區只能增大不能減小