小白搞kafka之安装篇

一、简介

kafka是由Scala语言开发的并且依赖zookeeper，所以我们在部署kafka环境的时候需要安装scala以及zookeeper。一般都是多节点部署（除学习外），所以我通过kvm虚拟了6台虚拟机，三台做为zookeeper集群，三台做为kafka集群。

二、安装zookeeper

2.1、环境

名称	IP地址
zookeeper1	192.168.125.210
zookeeper2	192.168.125.220
zookeeper3	192.168.125.230

2.2、下载并安装

[root@centos7 local]# wget https://archive.apache.org/dist/zookeeper/zookeeper-3.5.5/apache-zookeeper-3.5.5-bin.tar.gz
[root@centos7 local]# tar -zxvf apache-zookeeper-3.5.5-bin.tar.gz -C /usr/local
[root@centos7 local]# vi ~/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin:/usr/local/apache-zookeeper-3.5.5-bin/bin

export PATH
[root@centos7 bin]# source ~/.bash_profile 
[root@centos7 bin]# 

我们将zookeeper安装到/usr/local目录中，由于我们下载的bin文件为了使用方便，将安装目录添加到环境变了PATH中。

2.3、配置

2.3.1、配置文件

在zookeeper提供了一个默认配置，conf/zoo_sample.cfg，我们可以对其进行简单修改：指定dataDir和增加集群配置，如下所示：

[root@zookeeper1 conf]# cp zoo_sample.cfg zoo.cfg
[root@zookeeper1 conf]#
[root@zookeeper1 conf]# vi zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
#zookeeper数据存储目录
dataDir=/opt/zookeeper/data
# the port at which the clients will connect
# 与客户端通信的端口
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

#server.服务编号=服务地址、LF通信端口、选举端口
server.1=192.168.125.210:2888:3888
server.2=192.168.125.220:2888:3888
server.3=192.168.125.230:2888:3888

将zoo.cfg配置文件拷贝到其他机器上即可。 

2.3.2、创建id文件

在三台机器上，/opt/zookeeper/data目录（与配置文件dataDir指定的目录保持一致）下面创建myid，内容分别为1,2,3（与配置文件server.x中x保持一致），具体如下所示：

#server1 配置
[root@zookeeper1 conf]# mkdir -p /opt/zookeeper/data
[root@zookeeper1 conf]# echo 1 > /opt/zookeeper/data/myid

#server2 配置
[root@zookeeper2 conf]# mkdir -p /opt/zookeeper/data
[root@zookeeper2 conf]# echo 2 > /opt/zookeeper/data/myid

#server3 配置
[root@zookeeper3 conf]# mkdir -p /opt/zookeeper/data
[root@zookeeper3 conf]# echo 3 > /opt/zookeeper/data/myid

2.4、启动 

分别启动三个节点，如下所示：

[root@zookeeper1 ~]# 
[root@zookeeper1 ~]# zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@zookeeper1 ~]# 

[root@zookeeper2 ~]# 
[root@zookeeper2 ~]# zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@zookeeper2 ~]# 

[root@zookeeper3 ~]#
[root@zookeeper3 ~]# zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@zookeeper3 ~]#

通过status命令查看状态，如下所示：

[root@zookeeper3 ~]#  zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
[root@zookeeper3 ~]# 

注意：如果status报错，如下错误，可尝试将防火墙关闭掉：

[root@zookeeper3 ~]# zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Error contacting service. It is probably not running.
[root@zookeeper3 ~]# 

三、安装kafka

3.1、安装Scala

由于kafka是由Scala语言实现的，所以在安装kafka之前我们需要安装scala，若系统中已经安装了可忽略，如下所示： 

[root@centos7 ~]#  wget https://downloads.lightbend.com/scala/2.12.10/scala-2.12.10.rpm
[root@centos7 ~]# 
[root@centos7 ~]# rpm -ivh scala-2.12.10.rpm 
准备中...                          ################################# [100%]
正在升级/安装...
   1:scala-2.12.10-1                  ################################# [100%]
[root@centos7 ~]# 

3.2、下载并安装

下在软件包并进行安装，最后将目录添加到PATH环境变量中

[root@kafka1 ~]# wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.3.0/kafka_2.12-2.3.0.tgz
[root@kafka1 ~]#
[root@kafka1 ~]# tar -zxvf kafka_2.12-2.3.0.tgz -C /usr/local/
[root@kafka1 ~]# vi ~/.bash_profile 
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin:/usr/local/kafka_2.12-2.3.0/bin

export PATH
[root@kafka1 ~]# 
[root@kafka1 ~]# source ~/.bash_profile 
[root@kafka1 ~]# 

3.3、配置

kafka的配置，我们需要修改四点：指定kafka broker标识，监听服务地址，设置存储目录，设置zookeeper集群。修改config/server.properties文件，具体如下：

vi /usr/local/kafka_2.12-2.3.0/config/server.properties

# 每个broker id不能重复
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1

# 显示设置监听地址  主要避免错误
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://192.168.125.110:9092

# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://192.168.125.110:9092

#设置存储目录
# A comma separated list of directories under which to store log files
log.dirs=/opt/kafka/kafka-logs

#设置zookeeper集群
zookeeper.connect=192.168.125.210:2181,192.168.125.220:2181,192.168.125.230:2181

注意：三个kafka集群broker id不能重复，必须是数字。

3.4、启动

我们kafka提供的脚本启动kafka，如下命令：

[root@kafka1 ~]# kafka-server-start.sh -daemon /usr/local/kafka_2.12-2.3.0/config/server.properties 
[root@kafka1 ~]# jps
7488 Jps
7455 Kafka
[root@kafka1 ~]# 

我们通过jps命令可以查看到Kafka已经启动。