kafka+zookeeper高可用集羣搭建shell腳本使用教程

kafka+zookeeper高可用集羣搭建shell腳本使用教程

前言

Kafka是最初由Linkedin公司開發,是一個分佈式、支持分區的(partition)、多副本的(replica),基於zookeeper協調的分佈式消息系統,它的最大的特性就是可以實時的處理大量數據以滿足各種需求場景:比如基於hadoop的批處理系統、低延遲的實時系統、storm/Spark流式處理引擎,web/nginx日誌、訪問日誌,消息服務等等,用scala語言編寫,Linkedin於2010年貢獻給了Apache基金會併成爲頂級開源 項目。

一、準備工作

1、環境準備

kafka-zookeeper-1虛擬機 :192.168.25.81

kafka-zookeeper-2虛擬機 :192.168.25.82

kafka-zookeeper-3虛擬機 :192.168.25.83

2、安裝3個CentOS7

3、配置服務器以上的ip地址

// 修改network文件,不同機器的文件名不一樣,但都是ifcfg-開頭
vi /etc/sysconfig/network-scripts/ifcfg-xxxxx

// 修改一下參數
BOOTPROTO=static
IPADDR=xxxxxxx // 如kafka-zookeeper-1虛擬機 IPADDR="192.168.25.81"
GATEWAY=xxxxxx // 網關         GATEWAY="192.168.25.2"
NETMASK="255.255.255.0"
DNS1="114.114.114.114"
ZONE=public
ONBOOT=yes

保存退出
重啓網絡
systemctl restart network

二、運行腳本

1、3個節點zookeeper服務器啓動

一、zookeeper_1.sh

#!/bin/bash
#linux基本軟件和jdk1.8+zookeeper_1高可用集羣安裝腳本

echo "開始安裝vim"
rpm -qa|grep vim
yum -y install vim*
echo "vim安裝完畢"

echo "安裝telnet和xinetd和設置開機啓動"
rpm -qa telnet-server
rpm -qa xinetd
yum list |grep telnet
yum -y install telnet-server.x86_64
yum -y install telnet.x86_64
yum list |grep xinetd
yum -y install xinetd.x86_64
systemctl enable xinetd.service
systemctl enable telnet.socket
echo "telnet和xinetd和設置開機啓動安裝完畢"

echo "開啓telnet和xinetd的service"
systemctl start telnet.socket
systemctl start xinetd
echo "開啓telnet和xinetd的service完畢"

echo "開啓防火牆端口"
firewall-cmd --zone=public --add-port=23/tcp --permanent
firewall-cmd --reload
echo "開啓防火牆端口完畢"

echo "安裝net工具包"
x=`rpm -qa | grep net-tools`
if [ `rpm -qa | grep net-tools |wc -l` -ne 0 ];then
echo "net-tools已存在"
else
yum install -y net-tools
fi
echo "安裝net工具包完畢"

echo "安裝wget"
x=`rpm -qa | grep wget`
if [ `rpm -qa | grep wget |wc -l` -ne 0 ];then
echo "wget已存在"
else
yum -y install wget
fi
x=`rpm -qa | grep setup`
if [ `rpm -qa | grep setup |wc -l` -ne 0 ];then
echo "setup已存在"
else
yum -y install setup
fi
x=`rpm -qa | grep perl`
if [ `rpm -qa | grep perl |wc -l` -ne 0 ];then
echo "perl已存在"
else
yum -y install perl
fi
echo "安裝wget成功"

echo "安裝ntp時間同步"
rpm -qa|grep ntp
yum install -y ntp
echo "安裝ntp時間同步完畢"
echo "設開機啓動ntp時間同步"
systemctl start ntpd.service
systemctl enable ntpd.service
echo "設開機啓動ntp時間同步完畢"

echo "開始安裝jdk1.8"
echo "下載jdk"
x=`rpm -qa | grep jdk`
if [ `rpm -qa | grep jdk |wc -l` -ne 0 ];then
echo "jdk1.8已存在"
else
yum install java-1.8.0-openjdk* -y
fi
echo "
# /usr/share/doc/setup-*/uidgid file
if [ \$UID -gt 199 ] && [ \"\`/usr/bin/id -gn\`\" = \"\`/usr/bin/id -un\`\" ]; then
    umask 002
else
    umask 022
fi          

for i in /etc/profile.d/*.sh /etc/profile.d/sh.local ; do
    if [ -r \"\$i\" ]; then      
        if [ \"\${-#*i}\" != \"\$-\" ]; then
            . \"\$i\"
        else  
            . \"\$i\" >/dev/null
        fi
    fi
done

unset i
unset -f pathmunge
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64
export JRE_HOME=\${JAVA_HOME}/jre
export CLASSPATH=.:\${JAVA_HOME}/lib:\${JRE_HOME}/lib
export PATH=\${JAVA_HOME}/bin:\$PATH 
" > /etc/profile
echo "安裝jdk1.8完畢"


echo "開始安裝Zookeeper集羣搭建"
cd /opt
wget https://archive.apache.org/dist/zookeeper/stable/zookeeper-3.4.10.tar.gz
tar zxvf zookeeper-3.4.10.tar.gz

echo "創建/opt/zookeeper-3.4.10/conf/zoo.cfg"
echo "開始編輯/opt/zookeeper-3.4.10/conf/zoo.cfg"
echo "
    tickTime=2000
    initLimit=10
    syncLimit=5
    dataDir=/root/zookeeper/data
    dataLogDir=/root/zookeeper/logs
    clientPort=2181
    server.1=$1:2888:3888
    server.2=$2:2888:3888
    server.3=$3:2888:3888
" > /opt/zookeeper-3.4.10/conf/zoo.cfg
echo "編輯/opt/zookeeper-3.4.10/conf/zoo.cfg完畢"

echo "創建/root/zookeeper/data/myid"
cd /root
mkdir zookeeper
cd zookeeper
mkdir data
echo "1" > /root/zookeeper/data/myid
echo "創建/root/zookeeper/data/myid完畢"

echo "開啓防火牆端口"
firewall-cmd --zone=public --add-port=2181/tcp --permanent
firewall-cmd --zone=public --add-port=2888/tcp --permanent
firewall-cmd --zone=public --add-port=3888/tcp --permanent
firewall-cmd --zone=public --add-port=9092/tcp --permanent
firewall-cmd --reload
echo "開啓防火牆端口成功"

echo "啓動zookeeper"
cd /opt/zookeeper-3.4.10/bin/
./zkServer.sh start
echo "zookeeper啓動成功"

說明:

1、dataDir 和 dataLogDir 需要在啓動前創建完成

2、clientPort 爲 zookeeper的服務端口

3、server.0、server.1、server.2 爲 zk 集羣中三個 node 的信息,定義格式hostname:port1:port2,其中 port1 是 node 間通信使用的端口,port2 是node 選舉使用的端口,需確保三臺主機的這兩個端口都是互通的

4、在第一臺節點1主機上建立的 myid 文件內容是 1,第二臺節點2主機上建立的myid文件內容是 2,第三臺節點3主機上建立的myid文件內容是 3。myid文件內容需要與/opt/zookeeper-3.4.10/conf/zoo.cfg中的配置的server.id的編號對應。
  • 把zookeeper_1.sh上傳到節點1虛擬機/home目錄下,用以下命令執行腳本權限

    • cd /home
    • chmod 777 ./*.sh
  • 執行腳本命令

    • ./zookeeper_1.sh 節點1虛擬機ip地址 節點2虛擬機ip地址 節點3虛擬機ip地址
    • ./zookeeper_1.sh 192.168.25.81 192.168.25.82 192.168.25.83

    參數說明 :

      ip地址 :zookeeper集羣3個節點的ip地址
    

打開cd /opt/zookeeper-3.4.10/conf/zoo.cfg 檢查

輸入jpa命令,zookeeper啓動成功

二、zookeeper_2.sh

#!/bin/bash
#linux基本軟件和jdk1.8+zookeeper_2高可用集羣安裝腳本

echo "開始安裝vim"
rpm -qa|grep vim
yum -y install vim*
echo "vim安裝完畢"

echo "安裝telnet和xinetd和設置開機啓動"
rpm -qa telnet-server
rpm -qa xinetd
yum list |grep telnet
yum -y install telnet-server.x86_64
yum -y install telnet.x86_64
yum list |grep xinetd
yum -y install xinetd.x86_64
systemctl enable xinetd.service
systemctl enable telnet.socket
echo "telnet和xinetd和設置開機啓動安裝完畢"

echo "開啓telnet和xinetd的service"
systemctl start telnet.socket
systemctl start xinetd
echo "開啓telnet和xinetd的service完畢"

echo "開啓防火牆端口"
firewall-cmd --zone=public --add-port=23/tcp --permanent
firewall-cmd --reload
echo "開啓防火牆端口完畢"

echo "安裝net工具包"
x=`rpm -qa | grep net-tools`
if [ `rpm -qa | grep net-tools |wc -l` -ne 0 ];then
echo "net-tools已存在"
else
yum install -y net-tools
fi
echo "安裝net工具包完畢"

echo "安裝wget"
x=`rpm -qa | grep wget`
if [ `rpm -qa | grep wget |wc -l` -ne 0 ];then
echo "wget已存在"
else
yum -y install wget
fi
x=`rpm -qa | grep setup`
if [ `rpm -qa | grep setup |wc -l` -ne 0 ];then
echo "setup已存在"
else
yum -y install setup
fi
x=`rpm -qa | grep perl`
if [ `rpm -qa | grep perl |wc -l` -ne 0 ];then
echo "perl已存在"
else
yum -y install perl
fi
echo "安裝wget成功"

echo "安裝ntp時間同步"
rpm -qa|grep ntp
yum install -y ntp
echo "安裝ntp時間同步完畢"
echo "設開機啓動ntp時間同步"
systemctl start ntpd.service
systemctl enable ntpd.service
echo "設開機啓動ntp時間同步完畢"

echo "開始安裝jdk1.8"
echo "下載jdk"
x=`rpm -qa | grep jdk`
if [ `rpm -qa | grep jdk |wc -l` -ne 0 ];then
echo "jdk1.8已存在"
else
yum install java-1.8.0-openjdk* -y
fi
echo "
# /usr/share/doc/setup-*/uidgid file
if [ \$UID -gt 199 ] && [ \"\`/usr/bin/id -gn\`\" = \"\`/usr/bin/id -un\`\" ]; then
    umask 002
else
    umask 022
fi          

for i in /etc/profile.d/*.sh /etc/profile.d/sh.local ; do
    if [ -r \"\$i\" ]; then      
        if [ \"\${-#*i}\" != \"\$-\" ]; then
            . \"\$i\"
        else  
            . \"\$i\" >/dev/null
        fi
    fi
done

unset i
unset -f pathmunge
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64
export JRE_HOME=\${JAVA_HOME}/jre
export CLASSPATH=.:\${JAVA_HOME}/lib:\${JRE_HOME}/lib
export PATH=\${JAVA_HOME}/bin:\$PATH 
" > /etc/profile
echo "安裝jdk1.8完畢"


echo "開始安裝Zookeeper集羣搭建"
cd /opt
wget https://archive.apache.org/dist/zookeeper/stable/zookeeper-3.4.10.tar.gz
tar zxvf zookeeper-3.4.10.tar.gz

echo "創建/opt/zookeeper-3.4.10/conf/zoo.cfg"
echo "開始編輯/opt/zookeeper-3.4.10/conf/zoo.cfg"
echo "
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/root/zookeeper/data
dataLogDir=/root/zookeeper/logs
clientPort=2181
server.1=$1:2888:3888
server.2=$2:2888:3888
server.3=$3:2888:3888
" > /opt/zookeeper-3.4.10/conf/zoo.cfg
echo "編輯/opt/zookeeper-3.4.10/conf/zoo.cfg完畢"

echo "創建/root/zookeeper/data/myid"
cd /root
mkdir zookeeper
cd zookeeper
mkdir data
echo "2" > /root/zookeeper/data/myid
echo "創建/root/zookeeper/data/myid完畢"


echo "開啓防火牆端口"
firewall-cmd --zone=public --add-port=2181/tcp --permanent
firewall-cmd --zone=public --add-port=2888/tcp --permanent
firewall-cmd --zone=public --add-port=3888/tcp --permanent
firewall-cmd --zone=public --add-port=9092/tcp --permanent
firewall-cmd --reload
echo "開啓防火牆端口成功"


echo "啓動zookeeper"
cd /opt/zookeeper-3.4.10/bin/
./zkServer.sh start
echo "zookeeper啓動成功"

三、zookeeper_3.sh

#!/bin/bash
#linux基本軟件和jdk1.8+zookeeper_3高可用集羣安裝腳本

echo "開始安裝vim"
rpm -qa|grep vim
yum -y install vim*
echo "vim安裝完畢"

echo "安裝telnet和xinetd和設置開機啓動"
rpm -qa telnet-server
rpm -qa xinetd
yum list |grep telnet
yum -y install telnet-server.x86_64
yum -y install telnet.x86_64
yum list |grep xinetd
yum -y install xinetd.x86_64
systemctl enable xinetd.service
systemctl enable telnet.socket
echo "telnet和xinetd和設置開機啓動安裝完畢"

echo "開啓telnet和xinetd的service"
systemctl start telnet.socket
systemctl start xinetd
echo "開啓telnet和xinetd的service完畢"

echo "開啓防火牆端口"
firewall-cmd --zone=public --add-port=23/tcp --permanent
firewall-cmd --reload
echo "開啓防火牆端口完畢"

echo "安裝net工具包"
x=`rpm -qa | grep net-tools`
if [ `rpm -qa | grep net-tools |wc -l` -ne 0 ];then
echo "net-tools已存在"
else
yum install -y net-tools
fi
echo "安裝net工具包完畢"

echo "安裝wget"
x=`rpm -qa | grep wget`
if [ `rpm -qa | grep wget |wc -l` -ne 0 ];then
echo "wget已存在"
else
yum -y install wget
fi
x=`rpm -qa | grep setup`
if [ `rpm -qa | grep setup |wc -l` -ne 0 ];then
echo "setup已存在"
else
yum -y install setup
fi
x=`rpm -qa | grep perl`
if [ `rpm -qa | grep perl |wc -l` -ne 0 ];then
echo "perl已存在"
else
yum -y install perl
fi
echo "安裝wget成功"

echo "安裝ntp時間同步"
rpm -qa|grep ntp
yum install -y ntp
echo "安裝ntp時間同步完畢"
echo "設開機啓動ntp時間同步"
systemctl start ntpd.service
systemctl enable ntpd.service
echo "設開機啓動ntp時間同步完畢"


echo "開始安裝jdk1.8"
echo "下載jdk"
x=`rpm -qa | grep jdk`
if [ `rpm -qa | grep jdk |wc -l` -ne 0 ];then
echo "jdk1.8已存在"
else
yum install java-1.8.0-openjdk* -y
fi
echo "
# /usr/share/doc/setup-*/uidgid file
if [ \$UID -gt 199 ] && [ \"\`/usr/bin/id -gn\`\" = \"\`/usr/bin/id -un\`\" ]; then
    umask 002
else
    umask 022
fi          

for i in /etc/profile.d/*.sh /etc/profile.d/sh.local ; do
    if [ -r \"\$i\" ]; then      
        if [ \"\${-#*i}\" != \"\$-\" ]; then
            . \"\$i\"
        else  
            . \"\$i\" >/dev/null
        fi
    fi
done

unset i
unset -f pathmunge
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64
export JRE_HOME=\${JAVA_HOME}/jre
export CLASSPATH=.:\${JAVA_HOME}/lib:\${JRE_HOME}/lib
export PATH=\${JAVA_HOME}/bin:\$PATH 
" > /etc/profile
echo "安裝jdk1.8完畢"


echo "開始安裝Zookeeper集羣搭建"
cd /opt
wget https://archive.apache.org/dist/zookeeper/stable/zookeeper-3.4.10.tar.gz
tar zxvf zookeeper-3.4.10.tar.gz

echo "創建/opt/zookeeper-3.4.10/conf/zoo.cfg"
echo "開始編輯/opt/zookeeper-3.4.10/conf/zoo.cfg"
echo "
    tickTime=2000
    initLimit=10
    syncLimit=5
    dataDir=/root/zookeeper/data
    dataLogDir=/root/zookeeper/logs
    clientPort=2181
        server.1=$1:2888:3888
    server.2=$2:2888:3888
    server.3=$3:2888:3888
" > /opt/zookeeper-3.4.10/conf/zoo.cfg
echo "編輯/opt/zookeeper-3.4.10/conf/zoo.cfg完畢"

echo "創建/root/zookeeper/data/myid"
cd /root
mkdir zookeeper
cd zookeeper
mkdir data
echo "3" > /root/zookeeper/data/myid
echo "創建/root/zookeeper/data/myid完畢"


echo "開啓防火牆端口"
firewall-cmd --zone=public --add-port=2181/tcp --permanent
firewall-cmd --zone=public --add-port=2888/tcp --permanent
firewall-cmd --zone=public --add-port=3888/tcp --permanent
firewall-cmd --zone=public --add-port=9092/tcp --permanent
firewall-cmd --reload
echo "開啓防火牆端口成功"


echo "啓動zookeeper"
cd /opt/zookeeper-3.4.10/bin/
./zkServer.sh start
echo "zookeeper啓動成功"

  • 把zookeeper_2.sh和zookeeper_3.sh上傳到節點2虛擬機、虛擬機節點3的/home目錄下,同樣執行以上命令,把三臺zookeeper服務啓動起來

    • ./zookeeper_2.sh 192.168.25.81 192.168.25.82 192.168.25.83
    • ./zookeeper_3.sh 192.168.25.81 192.168.25.82 192.168.25.83

測試

cd /opt/zookeeper-3.4.10/bin/
./zkServer.sh status

把192.168.25.83節點的leader服務停掉

./zkServer.sh stop

可以看出192.168.25.82節點的變成leader服務

再開啓192.168.25.83節點的服務,發現變成follower

./zkServer.sh start

如果提示Starting zookeeper … already running as process ……,表示之前有啓動過zookeeper,但是沒有正常關閉。 進入和bin同層級的data目錄下,刪除zookeeper_server.pid文件,再執行啓動命令。

2、3個節點kafka服務器啓動

一、kafka_zookeeper_1.sh

#!/bin/bash
#kafka+zookeeper_1高可用集羣安裝腳本

echo "開始Kafka 集羣搭建"
cd /opt
wget http://apache.01link.hk/kafka/2.0.0/kafka_2.11-2.0.0.tgz
tar zxvf kafka_2.11-2.0.0.tgz
mv kafka_2.11-2.0.0 kafka
mkdir /root/kafka
mkdir /root/kafka/logs/
echo "開始編輯/opt/kafka/config/server.properties"
echo "
broker.id=1
listeners=PLAINTEXT://:9092
port=9092
host.name=$1
advertised.host.name=$2
advertised.port=9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/root/kafka/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000  
zookeeper.connect=$3:2181,$4:2181,$5:2181
zookeeper.connection.timeout.ms=6000
" > /opt/kafka/config/server.properties
echo "編輯/opt/kafka/config/server.properties完畢"

echo "啓動kafka"
cd /opt/kafka
bin/kafka-server-start.sh config/server.properties &
echo "kafka啓動成功"

注意!!!:

每個節點的broker.id不一樣,不然會有衝突

按照官方文檔的說法,advertised.host.name 和 advertised.port 這兩個參數用於定義集羣向 Producer 和 Consumer 廣播的節點 host 和 port,如果不定義,會默認使用 host.name 和 port 的定義。但在實際應用中,發現如果不定義 advertised.host.name 參數,使用 Java 客戶端從遠端連接集羣時,會發生連接超時,拋出異常:org.apache.kafka.common.errors.TimeoutException: Batch Expired

經過過 debug 發現,連接到集羣是成功的,但連接到集羣后更新回來的集羣 meta 信息卻是錯誤的。metadata 中的 Cluster 信息中節點的 hostname 是一串字符,而不是實際的ip地址。這串其實是遠端主機的 hostname,這說明在沒有配置 advertised.host.name 的情況下,Kafka 並沒有像官方文檔宣稱的那樣改爲廣播我們配置的 host.name,而是廣播了主機配置的 hostname 。遠端的客戶端並沒有配置 hosts,所以自然是連接不上這個 hostname 的。要解決這一問題,把 host.name 和 advertised.host.name 都配置成絕對的 ip 地址就可以了。

  • 一、把kafka_zookeeper_1.sh上傳到節點1虛擬機/home目錄下,用以下命令執行腳本權限

    • cd /home
    • chmod 777 ./*.sh
  • 執行腳本命令

    • ./kafka_zookeeper_1.sh host.name advertised.host.name 節點1虛擬機ip地址 節點2虛擬機ip地址 節點3虛擬機ip地址
    • ./kafka_zookeeper_1.sh 192.168.25.81 192.168.25.81 192.168.25.81 192.168.25.82 192.168.25.83

    參數說明 :

      host.name :節點1虛擬機ip地址
      advertised.host.name :節點1虛擬機ip地址
    

  • 二、把kafka_zookeeper_2.sh上傳到節點2虛擬機/home目錄下,用以下命令執行腳本權限

    • cd /home
    • chmod 777 ./*.sh
  • 執行腳本命令

    • ./kafka_zookeeper_2.sh host.name advertised.host.name 節點1虛擬機ip地址 節點2虛擬機ip地址 節點3虛擬機ip地址
    • ./kafka_zookeeper_2.sh 192.168.25.82 192.168.25.82 192.168.25.81 192.168.25.82 192.168.25.83

    參數說明 :

      host.name :節點2虛擬機ip地址
      advertised.host.name :節點2虛擬機ip地址
    

  • 三、把kafka_zookeeper_3.sh上傳到節點3虛擬機/home目錄下,用以下命令執行腳本權限

    • cd /home
    • chmod 777 ./*.sh
  • 執行腳本命令

    • ./kafka_zookeeper_3.sh host.name advertised.host.name 節點1虛擬機ip地址 節點2虛擬機ip地址 節點3虛擬機ip地址
    • ./kafka_zookeeper_3.sh 192.168.25.83 192.168.25.83 192.168.25.81 192.168.25.82 192.168.25.83

    參數說明 :

      host.name :節點3虛擬機ip地址
      advertised.host.name :節點3虛擬機ip地址
    

  • 四、測試

在節點1創建test的topic

cd /opt/kafka
bin/kafka-topics.sh --create --zookeeper 192.168.25.81:2181,192.168.25.82:2181,192.168.25.83:2181 --replication-factor 3 --partitions 3 --topic test

查看topic信息

cd /opt/kafka
bin/kafka-topics.sh --describe --zookeeper 192.168.25.81:2181,192.168.25.82:2181,192.168.25.83:2181 --topic test

創建生產者,並輸入信息

cd /opt/kafka
bin/kafka-console-producer.sh --broker-list 192.168.25.81:9092 -topic test

在節點2查看節點3消費者信息,也可以自己分別測試

bin/kafka-console-consumer.sh --bootstrap-server 192.168.25.83:9092 --topic test --from-beginning

繼續輸入

繼續輸出

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章