Kafka Connect:kafka connect實時ETL啓動與停止腳本

自己搭建了kafka connect實時ETL環境,並編寫了shell腳本以方便啓動和停止,具體代碼如下:
啓動腳本:

#!/bin/bash
#腳本涵蓋從啓動kafka connect集羣到schema-registry和schema-registry-ui的整個實時ETL環境的啓動

#日誌清空
echo '' > /Users/djg/env/kafka_2.12-2.2.2/start.log

#替換kafka配置文件中的對外ip
ip=`ifconfig | awk '/inet .* netmask .* broadcast/{print}' | awk -F" " '{print $2}'`;
echo $ip;
sed -i "" "s/\(advertised.listeners=PLAINTEXT:\/\/\).*\(:9092\)/\1$ip\2/" /Users/djg/env/kafka_2.12-2.2.2/config/server.properties

#啓動zookeeper
/Users/djg/env/zookeeper-3.4.11/bin/zkServer.sh start >> /Users/djg/env/kafka_2.12-2.2.2/start.log 2>&1 
sleep 5s

#啓動kafka broker
nohup /Users/djg/env/kafka_2.12-2.2.2/bin/kafka-server-start.sh /Users/djg/env/kafka_2.12-2.2.2/config/server.properties >> /Users/djg/env/kafka_2.12-2.2.2/start.log 2>&1 &
sleep 10s

#啓動kafka connect
nohup /Users/djg/env/kafka_2.12-2.2.2/bin/connect-distributed.sh /Users/djg/env/kafka_2.12-2.2.2/config/connect-distributed.properties >> /Users/djg/env/kafka_2.12-2.2.2/start.log 2>&1 &
sleep 10s

#啓動schema registry
docker run -it --rm --name schema-registry -e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=$ip:2181 -e SCHEMA_REGISTRY_HOST_NAME=schema-registry -e SCHEMA_REGISTRY_LISTENERS=http://0.0.0.0:8081 -p 8081:8081 -v /Users/djg/env/confluent-schema-registry/conf:/etc/schema-registry -d confluentinc/cp-schema-registry
sleep 5s

#啓動schema registry ui
docker run --rm -d -p 8000:8000 -e SCHEMAREGISTRY_URL=http://$ip:8081 -e PROXY=true --name schema-registry-ui landoop/schema-registry-ui

停止腳本:

#!/bin/bash
#停止schema-registry和schema-registry-ui的docker容器
docker stop schema-registry
docker stop schema-registry-ui
sleep 5s

#停止zookeeper、kafka broker、kafka connect
connectip=`jps | awk -F" " '$2=="ConnectDistributed"{print $1}'`
zookeeperip=`jps | awk -F" " '$2=="QuorumPeerMain"{print $1}'`
brokerip=`jps | awk -F" " '$2=="Kafka"{print $1}'`
kill -9 $connectip $zookeeperip $brokerip

說明:
shcema-registry和schema-registry-ui是通過docker啓動的,因爲這些服務只是輔助服務,能用就行。而kafka broker和kafka connect是非docker啓動的,方便熟悉其配置。

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章