kafka+zookeeper高可用集羣搭建shell腳本使用教程
前言
Kafka是最初由Linkedin公司開發,是一個分佈式、支持分區的(partition)、多副本的(replica),基於zookeeper協調的分佈式消息系統,它的最大的特性就是可以實時的處理大量數據以滿足各種需求場景:比如基於hadoop的批處理系統、低延遲的實時系統、storm/Spark流式處理引擎,web/nginx日誌、訪問日誌,消息服務等等,用scala語言編寫,Linkedin於2010年貢獻給了Apache基金會併成爲頂級開源 項目。
一、準備工作
1、環境準備
kafka-zookeeper-1虛擬機 :192.168.25.81
kafka-zookeeper-2虛擬機 :192.168.25.82
kafka-zookeeper-3虛擬機 :192.168.25.83
2、安裝3個CentOS7
3、配置服務器以上的ip地址
// 修改network文件,不同機器的文件名不一樣,但都是ifcfg-開頭
vi /etc/sysconfig/network-scripts/ifcfg-xxxxx
// 修改一下參數
BOOTPROTO=static
IPADDR=xxxxxxx // 如kafka-zookeeper-1虛擬機 IPADDR="192.168.25.81"
GATEWAY=xxxxxx // 網關 GATEWAY="192.168.25.2"
NETMASK="255.255.255.0"
DNS1="114.114.114.114"
ZONE=public
ONBOOT=yes
保存退出
重啓網絡
systemctl restart network
二、運行腳本
1、3個節點zookeeper服務器啓動
一、zookeeper_1.sh
#!/bin/bash
#linux基本軟件和jdk1.8+zookeeper_1高可用集羣安裝腳本
echo "開始安裝vim"
rpm -qa|grep vim
yum -y install vim*
echo "vim安裝完畢"
echo "安裝telnet和xinetd和設置開機啓動"
rpm -qa telnet-server
rpm -qa xinetd
yum list |grep telnet
yum -y install telnet-server.x86_64
yum -y install telnet.x86_64
yum list |grep xinetd
yum -y install xinetd.x86_64
systemctl enable xinetd.service
systemctl enable telnet.socket
echo "telnet和xinetd和設置開機啓動安裝完畢"
echo "開啓telnet和xinetd的service"
systemctl start telnet.socket
systemctl start xinetd
echo "開啓telnet和xinetd的service完畢"
echo "開啓防火牆端口"
firewall-cmd --zone=public --add-port=23/tcp --permanent
firewall-cmd --reload
echo "開啓防火牆端口完畢"
echo "安裝net工具包"
x=`rpm -qa | grep net-tools`
if [ `rpm -qa | grep net-tools |wc -l` -ne 0 ];then
echo "net-tools已存在"
else
yum install -y net-tools
fi
echo "安裝net工具包完畢"
echo "安裝wget"
x=`rpm -qa | grep wget`
if [ `rpm -qa | grep wget |wc -l` -ne 0 ];then
echo "wget已存在"
else
yum -y install wget
fi
x=`rpm -qa | grep setup`
if [ `rpm -qa | grep setup |wc -l` -ne 0 ];then
echo "setup已存在"
else
yum -y install setup
fi
x=`rpm -qa | grep perl`
if [ `rpm -qa | grep perl |wc -l` -ne 0 ];then
echo "perl已存在"
else
yum -y install perl
fi
echo "安裝wget成功"
echo "安裝ntp時間同步"
rpm -qa|grep ntp
yum install -y ntp
echo "安裝ntp時間同步完畢"
echo "設開機啓動ntp時間同步"
systemctl start ntpd.service
systemctl enable ntpd.service
echo "設開機啓動ntp時間同步完畢"
echo "開始安裝jdk1.8"
echo "下載jdk"
x=`rpm -qa | grep jdk`
if [ `rpm -qa | grep jdk |wc -l` -ne 0 ];then
echo "jdk1.8已存在"
else
yum install java-1.8.0-openjdk* -y
fi
echo "
# /usr/share/doc/setup-*/uidgid file
if [ \$UID -gt 199 ] && [ \"\`/usr/bin/id -gn\`\" = \"\`/usr/bin/id -un\`\" ]; then
umask 002
else
umask 022
fi
for i in /etc/profile.d/*.sh /etc/profile.d/sh.local ; do
if [ -r \"\$i\" ]; then
if [ \"\${-#*i}\" != \"\$-\" ]; then
. \"\$i\"
else
. \"\$i\" >/dev/null
fi
fi
done
unset i
unset -f pathmunge
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64
export JRE_HOME=\${JAVA_HOME}/jre
export CLASSPATH=.:\${JAVA_HOME}/lib:\${JRE_HOME}/lib
export PATH=\${JAVA_HOME}/bin:\$PATH
" > /etc/profile
echo "安裝jdk1.8完畢"
echo "開始安裝Zookeeper集羣搭建"
cd /opt
wget https://archive.apache.org/dist/zookeeper/stable/zookeeper-3.4.10.tar.gz
tar zxvf zookeeper-3.4.10.tar.gz
echo "創建/opt/zookeeper-3.4.10/conf/zoo.cfg"
echo "開始編輯/opt/zookeeper-3.4.10/conf/zoo.cfg"
echo "
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/root/zookeeper/data
dataLogDir=/root/zookeeper/logs
clientPort=2181
server.1=$1:2888:3888
server.2=$2:2888:3888
server.3=$3:2888:3888
" > /opt/zookeeper-3.4.10/conf/zoo.cfg
echo "編輯/opt/zookeeper-3.4.10/conf/zoo.cfg完畢"
echo "創建/root/zookeeper/data/myid"
cd /root
mkdir zookeeper
cd zookeeper
mkdir data
echo "1" > /root/zookeeper/data/myid
echo "創建/root/zookeeper/data/myid完畢"
echo "開啓防火牆端口"
firewall-cmd --zone=public --add-port=2181/tcp --permanent
firewall-cmd --zone=public --add-port=2888/tcp --permanent
firewall-cmd --zone=public --add-port=3888/tcp --permanent
firewall-cmd --zone=public --add-port=9092/tcp --permanent
firewall-cmd --reload
echo "開啓防火牆端口成功"
echo "啓動zookeeper"
cd /opt/zookeeper-3.4.10/bin/
./zkServer.sh start
echo "zookeeper啓動成功"
說明:
1、dataDir 和 dataLogDir 需要在啓動前創建完成
2、clientPort 爲 zookeeper的服務端口
3、server.0、server.1、server.2 爲 zk 集羣中三個 node 的信息,定義格式hostname:port1:port2,其中 port1 是 node 間通信使用的端口,port2 是node 選舉使用的端口,需確保三臺主機的這兩個端口都是互通的
4、在第一臺節點1主機上建立的 myid 文件內容是 1,第二臺節點2主機上建立的myid文件內容是 2,第三臺節點3主機上建立的myid文件內容是 3。myid文件內容需要與/opt/zookeeper-3.4.10/conf/zoo.cfg中的配置的server.id的編號對應。
-
把zookeeper_1.sh上傳到節點1虛擬機/home目錄下,用以下命令執行腳本權限
- cd /home
- chmod 777 ./*.sh
-
執行腳本命令
- ./zookeeper_1.sh 節點1虛擬機ip地址 節點2虛擬機ip地址 節點3虛擬機ip地址
- ./zookeeper_1.sh 192.168.25.81 192.168.25.82 192.168.25.83
參數說明 :
ip地址 :zookeeper集羣3個節點的ip地址
打開cd /opt/zookeeper-3.4.10/conf/zoo.cfg 檢查
輸入jpa命令,zookeeper啓動成功
二、zookeeper_2.sh
#!/bin/bash
#linux基本軟件和jdk1.8+zookeeper_2高可用集羣安裝腳本
echo "開始安裝vim"
rpm -qa|grep vim
yum -y install vim*
echo "vim安裝完畢"
echo "安裝telnet和xinetd和設置開機啓動"
rpm -qa telnet-server
rpm -qa xinetd
yum list |grep telnet
yum -y install telnet-server.x86_64
yum -y install telnet.x86_64
yum list |grep xinetd
yum -y install xinetd.x86_64
systemctl enable xinetd.service
systemctl enable telnet.socket
echo "telnet和xinetd和設置開機啓動安裝完畢"
echo "開啓telnet和xinetd的service"
systemctl start telnet.socket
systemctl start xinetd
echo "開啓telnet和xinetd的service完畢"
echo "開啓防火牆端口"
firewall-cmd --zone=public --add-port=23/tcp --permanent
firewall-cmd --reload
echo "開啓防火牆端口完畢"
echo "安裝net工具包"
x=`rpm -qa | grep net-tools`
if [ `rpm -qa | grep net-tools |wc -l` -ne 0 ];then
echo "net-tools已存在"
else
yum install -y net-tools
fi
echo "安裝net工具包完畢"
echo "安裝wget"
x=`rpm -qa | grep wget`
if [ `rpm -qa | grep wget |wc -l` -ne 0 ];then
echo "wget已存在"
else
yum -y install wget
fi
x=`rpm -qa | grep setup`
if [ `rpm -qa | grep setup |wc -l` -ne 0 ];then
echo "setup已存在"
else
yum -y install setup
fi
x=`rpm -qa | grep perl`
if [ `rpm -qa | grep perl |wc -l` -ne 0 ];then
echo "perl已存在"
else
yum -y install perl
fi
echo "安裝wget成功"
echo "安裝ntp時間同步"
rpm -qa|grep ntp
yum install -y ntp
echo "安裝ntp時間同步完畢"
echo "設開機啓動ntp時間同步"
systemctl start ntpd.service
systemctl enable ntpd.service
echo "設開機啓動ntp時間同步完畢"
echo "開始安裝jdk1.8"
echo "下載jdk"
x=`rpm -qa | grep jdk`
if [ `rpm -qa | grep jdk |wc -l` -ne 0 ];then
echo "jdk1.8已存在"
else
yum install java-1.8.0-openjdk* -y
fi
echo "
# /usr/share/doc/setup-*/uidgid file
if [ \$UID -gt 199 ] && [ \"\`/usr/bin/id -gn\`\" = \"\`/usr/bin/id -un\`\" ]; then
umask 002
else
umask 022
fi
for i in /etc/profile.d/*.sh /etc/profile.d/sh.local ; do
if [ -r \"\$i\" ]; then
if [ \"\${-#*i}\" != \"\$-\" ]; then
. \"\$i\"
else
. \"\$i\" >/dev/null
fi
fi
done
unset i
unset -f pathmunge
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64
export JRE_HOME=\${JAVA_HOME}/jre
export CLASSPATH=.:\${JAVA_HOME}/lib:\${JRE_HOME}/lib
export PATH=\${JAVA_HOME}/bin:\$PATH
" > /etc/profile
echo "安裝jdk1.8完畢"
echo "開始安裝Zookeeper集羣搭建"
cd /opt
wget https://archive.apache.org/dist/zookeeper/stable/zookeeper-3.4.10.tar.gz
tar zxvf zookeeper-3.4.10.tar.gz
echo "創建/opt/zookeeper-3.4.10/conf/zoo.cfg"
echo "開始編輯/opt/zookeeper-3.4.10/conf/zoo.cfg"
echo "
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/root/zookeeper/data
dataLogDir=/root/zookeeper/logs
clientPort=2181
server.1=$1:2888:3888
server.2=$2:2888:3888
server.3=$3:2888:3888
" > /opt/zookeeper-3.4.10/conf/zoo.cfg
echo "編輯/opt/zookeeper-3.4.10/conf/zoo.cfg完畢"
echo "創建/root/zookeeper/data/myid"
cd /root
mkdir zookeeper
cd zookeeper
mkdir data
echo "2" > /root/zookeeper/data/myid
echo "創建/root/zookeeper/data/myid完畢"
echo "開啓防火牆端口"
firewall-cmd --zone=public --add-port=2181/tcp --permanent
firewall-cmd --zone=public --add-port=2888/tcp --permanent
firewall-cmd --zone=public --add-port=3888/tcp --permanent
firewall-cmd --zone=public --add-port=9092/tcp --permanent
firewall-cmd --reload
echo "開啓防火牆端口成功"
echo "啓動zookeeper"
cd /opt/zookeeper-3.4.10/bin/
./zkServer.sh start
echo "zookeeper啓動成功"
三、zookeeper_3.sh
#!/bin/bash
#linux基本軟件和jdk1.8+zookeeper_3高可用集羣安裝腳本
echo "開始安裝vim"
rpm -qa|grep vim
yum -y install vim*
echo "vim安裝完畢"
echo "安裝telnet和xinetd和設置開機啓動"
rpm -qa telnet-server
rpm -qa xinetd
yum list |grep telnet
yum -y install telnet-server.x86_64
yum -y install telnet.x86_64
yum list |grep xinetd
yum -y install xinetd.x86_64
systemctl enable xinetd.service
systemctl enable telnet.socket
echo "telnet和xinetd和設置開機啓動安裝完畢"
echo "開啓telnet和xinetd的service"
systemctl start telnet.socket
systemctl start xinetd
echo "開啓telnet和xinetd的service完畢"
echo "開啓防火牆端口"
firewall-cmd --zone=public --add-port=23/tcp --permanent
firewall-cmd --reload
echo "開啓防火牆端口完畢"
echo "安裝net工具包"
x=`rpm -qa | grep net-tools`
if [ `rpm -qa | grep net-tools |wc -l` -ne 0 ];then
echo "net-tools已存在"
else
yum install -y net-tools
fi
echo "安裝net工具包完畢"
echo "安裝wget"
x=`rpm -qa | grep wget`
if [ `rpm -qa | grep wget |wc -l` -ne 0 ];then
echo "wget已存在"
else
yum -y install wget
fi
x=`rpm -qa | grep setup`
if [ `rpm -qa | grep setup |wc -l` -ne 0 ];then
echo "setup已存在"
else
yum -y install setup
fi
x=`rpm -qa | grep perl`
if [ `rpm -qa | grep perl |wc -l` -ne 0 ];then
echo "perl已存在"
else
yum -y install perl
fi
echo "安裝wget成功"
echo "安裝ntp時間同步"
rpm -qa|grep ntp
yum install -y ntp
echo "安裝ntp時間同步完畢"
echo "設開機啓動ntp時間同步"
systemctl start ntpd.service
systemctl enable ntpd.service
echo "設開機啓動ntp時間同步完畢"
echo "開始安裝jdk1.8"
echo "下載jdk"
x=`rpm -qa | grep jdk`
if [ `rpm -qa | grep jdk |wc -l` -ne 0 ];then
echo "jdk1.8已存在"
else
yum install java-1.8.0-openjdk* -y
fi
echo "
# /usr/share/doc/setup-*/uidgid file
if [ \$UID -gt 199 ] && [ \"\`/usr/bin/id -gn\`\" = \"\`/usr/bin/id -un\`\" ]; then
umask 002
else
umask 022
fi
for i in /etc/profile.d/*.sh /etc/profile.d/sh.local ; do
if [ -r \"\$i\" ]; then
if [ \"\${-#*i}\" != \"\$-\" ]; then
. \"\$i\"
else
. \"\$i\" >/dev/null
fi
fi
done
unset i
unset -f pathmunge
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64
export JRE_HOME=\${JAVA_HOME}/jre
export CLASSPATH=.:\${JAVA_HOME}/lib:\${JRE_HOME}/lib
export PATH=\${JAVA_HOME}/bin:\$PATH
" > /etc/profile
echo "安裝jdk1.8完畢"
echo "開始安裝Zookeeper集羣搭建"
cd /opt
wget https://archive.apache.org/dist/zookeeper/stable/zookeeper-3.4.10.tar.gz
tar zxvf zookeeper-3.4.10.tar.gz
echo "創建/opt/zookeeper-3.4.10/conf/zoo.cfg"
echo "開始編輯/opt/zookeeper-3.4.10/conf/zoo.cfg"
echo "
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/root/zookeeper/data
dataLogDir=/root/zookeeper/logs
clientPort=2181
server.1=$1:2888:3888
server.2=$2:2888:3888
server.3=$3:2888:3888
" > /opt/zookeeper-3.4.10/conf/zoo.cfg
echo "編輯/opt/zookeeper-3.4.10/conf/zoo.cfg完畢"
echo "創建/root/zookeeper/data/myid"
cd /root
mkdir zookeeper
cd zookeeper
mkdir data
echo "3" > /root/zookeeper/data/myid
echo "創建/root/zookeeper/data/myid完畢"
echo "開啓防火牆端口"
firewall-cmd --zone=public --add-port=2181/tcp --permanent
firewall-cmd --zone=public --add-port=2888/tcp --permanent
firewall-cmd --zone=public --add-port=3888/tcp --permanent
firewall-cmd --zone=public --add-port=9092/tcp --permanent
firewall-cmd --reload
echo "開啓防火牆端口成功"
echo "啓動zookeeper"
cd /opt/zookeeper-3.4.10/bin/
./zkServer.sh start
echo "zookeeper啓動成功"
-
把zookeeper_2.sh和zookeeper_3.sh上傳到節點2虛擬機、虛擬機節點3的/home目錄下,同樣執行以上命令,把三臺zookeeper服務啓動起來
- ./zookeeper_2.sh 192.168.25.81 192.168.25.82 192.168.25.83
- ./zookeeper_3.sh 192.168.25.81 192.168.25.82 192.168.25.83
測試
cd /opt/zookeeper-3.4.10/bin/
./zkServer.sh status
把192.168.25.83節點的leader服務停掉
./zkServer.sh stop
可以看出192.168.25.82節點的變成leader服務
再開啓192.168.25.83節點的服務,發現變成follower
./zkServer.sh start
如果提示Starting zookeeper … already running as process ……,表示之前有啓動過zookeeper,但是沒有正常關閉。 進入和bin同層級的data目錄下,刪除zookeeper_server.pid文件,再執行啓動命令。
2、3個節點kafka服務器啓動
一、kafka_zookeeper_1.sh
#!/bin/bash
#kafka+zookeeper_1高可用集羣安裝腳本
echo "開始Kafka 集羣搭建"
cd /opt
wget http://apache.01link.hk/kafka/2.0.0/kafka_2.11-2.0.0.tgz
tar zxvf kafka_2.11-2.0.0.tgz
mv kafka_2.11-2.0.0 kafka
mkdir /root/kafka
mkdir /root/kafka/logs/
echo "開始編輯/opt/kafka/config/server.properties"
echo "
broker.id=1
listeners=PLAINTEXT://:9092
port=9092
host.name=$1
advertised.host.name=$2
advertised.port=9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/root/kafka/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=$3:2181,$4:2181,$5:2181
zookeeper.connection.timeout.ms=6000
" > /opt/kafka/config/server.properties
echo "編輯/opt/kafka/config/server.properties完畢"
echo "啓動kafka"
cd /opt/kafka
bin/kafka-server-start.sh config/server.properties &
echo "kafka啓動成功"
注意!!!:
每個節點的broker.id不一樣,不然會有衝突
按照官方文檔的說法,advertised.host.name 和 advertised.port 這兩個參數用於定義集羣向 Producer 和 Consumer 廣播的節點 host 和 port,如果不定義,會默認使用 host.name 和 port 的定義。但在實際應用中,發現如果不定義 advertised.host.name 參數,使用 Java 客戶端從遠端連接集羣時,會發生連接超時,拋出異常:org.apache.kafka.common.errors.TimeoutException: Batch Expired
經過過 debug 發現,連接到集羣是成功的,但連接到集羣后更新回來的集羣 meta 信息卻是錯誤的。metadata 中的 Cluster 信息中節點的 hostname 是一串字符,而不是實際的ip地址。這串其實是遠端主機的 hostname,這說明在沒有配置 advertised.host.name 的情況下,Kafka 並沒有像官方文檔宣稱的那樣改爲廣播我們配置的 host.name,而是廣播了主機配置的 hostname 。遠端的客戶端並沒有配置 hosts,所以自然是連接不上這個 hostname 的。要解決這一問題,把 host.name 和 advertised.host.name 都配置成絕對的 ip 地址就可以了。
-
一、把kafka_zookeeper_1.sh上傳到節點1虛擬機/home目錄下,用以下命令執行腳本權限
- cd /home
- chmod 777 ./*.sh
-
執行腳本命令
- ./kafka_zookeeper_1.sh host.name advertised.host.name 節點1虛擬機ip地址 節點2虛擬機ip地址 節點3虛擬機ip地址
- ./kafka_zookeeper_1.sh 192.168.25.81 192.168.25.81 192.168.25.81 192.168.25.82 192.168.25.83
參數說明 :
host.name :節點1虛擬機ip地址 advertised.host.name :節點1虛擬機ip地址
-
二、把kafka_zookeeper_2.sh上傳到節點2虛擬機/home目錄下,用以下命令執行腳本權限
- cd /home
- chmod 777 ./*.sh
-
執行腳本命令
- ./kafka_zookeeper_2.sh host.name advertised.host.name 節點1虛擬機ip地址 節點2虛擬機ip地址 節點3虛擬機ip地址
- ./kafka_zookeeper_2.sh 192.168.25.82 192.168.25.82 192.168.25.81 192.168.25.82 192.168.25.83
參數說明 :
host.name :節點2虛擬機ip地址 advertised.host.name :節點2虛擬機ip地址
-
三、把kafka_zookeeper_3.sh上傳到節點3虛擬機/home目錄下,用以下命令執行腳本權限
- cd /home
- chmod 777 ./*.sh
-
執行腳本命令
- ./kafka_zookeeper_3.sh host.name advertised.host.name 節點1虛擬機ip地址 節點2虛擬機ip地址 節點3虛擬機ip地址
- ./kafka_zookeeper_3.sh 192.168.25.83 192.168.25.83 192.168.25.81 192.168.25.82 192.168.25.83
參數說明 :
host.name :節點3虛擬機ip地址 advertised.host.name :節點3虛擬機ip地址
- 四、測試
在節點1創建test的topic
cd /opt/kafka
bin/kafka-topics.sh --create --zookeeper 192.168.25.81:2181,192.168.25.82:2181,192.168.25.83:2181 --replication-factor 3 --partitions 3 --topic test
查看topic信息
cd /opt/kafka
bin/kafka-topics.sh --describe --zookeeper 192.168.25.81:2181,192.168.25.82:2181,192.168.25.83:2181 --topic test
創建生產者,並輸入信息
cd /opt/kafka
bin/kafka-console-producer.sh --broker-list 192.168.25.81:9092 -topic test
在節點2查看節點3消費者信息,也可以自己分別測試
bin/kafka-console-consumer.sh --bootstrap-server 192.168.25.83:9092 --topic test --from-beginning
繼續輸入
繼續輸出