小白搞kafka之安裝篇

一、簡介

kafka是由Scala語言開發的並且依賴zookeeper，所以我們在部署kafka環境的時候需要安裝scala以及zookeeper。一般都是多節點部署（除學習外），所以我通過kvm虛擬了6臺虛擬機，三臺做爲zookeeper集羣，三臺做爲kafka集羣。

二、安裝zookeeper

2.1、環境

名稱	IP地址
zookeeper1	192.168.125.210
zookeeper2	192.168.125.220
zookeeper3	192.168.125.230

2.2、下載並安裝

[root@centos7 local]# wget https://archive.apache.org/dist/zookeeper/zookeeper-3.5.5/apache-zookeeper-3.5.5-bin.tar.gz
[root@centos7 local]# tar -zxvf apache-zookeeper-3.5.5-bin.tar.gz -C /usr/local
[root@centos7 local]# vi ~/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin:/usr/local/apache-zookeeper-3.5.5-bin/bin

export PATH
[root@centos7 bin]# source ~/.bash_profile 
[root@centos7 bin]# 

我們將zookeeper安裝到/usr/local目錄中，由於我們下載的bin文件爲了使用方便，將安裝目錄添加到環境變了PATH中。

2.3、配置

2.3.1、配置文件

在zookeeper提供了一個默認配置，conf/zoo_sample.cfg，我們可以對其進行簡單修改：指定dataDir和增加集羣配置，如下所示：

[root@zookeeper1 conf]# cp zoo_sample.cfg zoo.cfg
[root@zookeeper1 conf]#
[root@zookeeper1 conf]# vi zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
#zookeeper數據存儲目錄
dataDir=/opt/zookeeper/data
# the port at which the clients will connect
# 與客戶端通信的端口
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

#server.服務編號=服務地址、LF通信端口、選舉端口
server.1=192.168.125.210:2888:3888
server.2=192.168.125.220:2888:3888
server.3=192.168.125.230:2888:3888

將zoo.cfg配置文件拷貝到其他機器上即可。 

2.3.2、創建id文件

在三臺機器上，/opt/zookeeper/data目錄（與配置文件dataDir指定的目錄保持一致）下面創建myid，內容分別爲1,2,3（與配置文件server.x中x保持一致），具體如下所示：

#server1 配置
[root@zookeeper1 conf]# mkdir -p /opt/zookeeper/data
[root@zookeeper1 conf]# echo 1 > /opt/zookeeper/data/myid

#server2 配置
[root@zookeeper2 conf]# mkdir -p /opt/zookeeper/data
[root@zookeeper2 conf]# echo 2 > /opt/zookeeper/data/myid

#server3 配置
[root@zookeeper3 conf]# mkdir -p /opt/zookeeper/data
[root@zookeeper3 conf]# echo 3 > /opt/zookeeper/data/myid

2.4、啓動 

分別啓動三個節點，如下所示：

[root@zookeeper1 ~]# 
[root@zookeeper1 ~]# zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@zookeeper1 ~]# 

[root@zookeeper2 ~]# 
[root@zookeeper2 ~]# zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@zookeeper2 ~]# 

[root@zookeeper3 ~]#
[root@zookeeper3 ~]# zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@zookeeper3 ~]#

通過status命令查看狀態，如下所示：

[root@zookeeper3 ~]#  zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
[root@zookeeper3 ~]# 

注意：如果status報錯，如下錯誤，可嘗試將防火牆關閉掉：

[root@zookeeper3 ~]# zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Error contacting service. It is probably not running.
[root@zookeeper3 ~]# 

三、安裝kafka

3.1、安裝Scala

由於kafka是由Scala語言實現的，所以在安裝kafka之前我們需要安裝scala，若系統中已經安裝了可忽略，如下所示： 

[root@centos7 ~]#  wget https://downloads.lightbend.com/scala/2.12.10/scala-2.12.10.rpm
[root@centos7 ~]# 
[root@centos7 ~]# rpm -ivh scala-2.12.10.rpm 
準備中...                          ################################# [100%]
正在升級/安裝...
   1:scala-2.12.10-1                  ################################# [100%]
[root@centos7 ~]# 

3.2、下載並安裝

下在軟件包並進行安裝，最後將目錄添加到PATH環境變量中

[root@kafka1 ~]# wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.3.0/kafka_2.12-2.3.0.tgz
[root@kafka1 ~]#
[root@kafka1 ~]# tar -zxvf kafka_2.12-2.3.0.tgz -C /usr/local/
[root@kafka1 ~]# vi ~/.bash_profile 
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin:/usr/local/kafka_2.12-2.3.0/bin

export PATH
[root@kafka1 ~]# 
[root@kafka1 ~]# source ~/.bash_profile 
[root@kafka1 ~]# 

3.3、配置

kafka的配置，我們需要修改四點：指定kafka broker標識，監聽服務地址，設置存儲目錄，設置zookeeper集羣。修改config/server.properties文件，具體如下：

vi /usr/local/kafka_2.12-2.3.0/config/server.properties

# 每個broker id不能重複
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1

# 顯示設置監聽地址  主要避免錯誤
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://192.168.125.110:9092

# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://192.168.125.110:9092

#設置存儲目錄
# A comma separated list of directories under which to store log files
log.dirs=/opt/kafka/kafka-logs

#設置zookeeper集羣
zookeeper.connect=192.168.125.210:2181,192.168.125.220:2181,192.168.125.230:2181

注意：三個kafka集羣broker id不能重複，必須是數字。

3.4、啓動

我們kafka提供的腳本啓動kafka，如下命令：

[root@kafka1 ~]# kafka-server-start.sh -daemon /usr/local/kafka_2.12-2.3.0/config/server.properties 
[root@kafka1 ~]# jps
7488 Jps
7455 Kafka
[root@kafka1 ~]# 

我們通過jps命令可以查看到Kafka已經啓動。