Kafka Connect:kafka connect实时ETL启动与停止脚本

自己搭建了kafka connect实时ETL环境,并编写了shell脚本以方便启动和停止,具体代码如下:
启动脚本:

#!/bin/bash
#脚本涵盖从启动kafka connect集群到schema-registry和schema-registry-ui的整个实时ETL环境的启动

#日志清空
echo '' > /Users/djg/env/kafka_2.12-2.2.2/start.log

#替换kafka配置文件中的对外ip
ip=`ifconfig | awk '/inet .* netmask .* broadcast/{print}' | awk -F" " '{print $2}'`;
echo $ip;
sed -i "" "s/\(advertised.listeners=PLAINTEXT:\/\/\).*\(:9092\)/\1$ip\2/" /Users/djg/env/kafka_2.12-2.2.2/config/server.properties

#启动zookeeper
/Users/djg/env/zookeeper-3.4.11/bin/zkServer.sh start >> /Users/djg/env/kafka_2.12-2.2.2/start.log 2>&1 
sleep 5s

#启动kafka broker
nohup /Users/djg/env/kafka_2.12-2.2.2/bin/kafka-server-start.sh /Users/djg/env/kafka_2.12-2.2.2/config/server.properties >> /Users/djg/env/kafka_2.12-2.2.2/start.log 2>&1 &
sleep 10s

#启动kafka connect
nohup /Users/djg/env/kafka_2.12-2.2.2/bin/connect-distributed.sh /Users/djg/env/kafka_2.12-2.2.2/config/connect-distributed.properties >> /Users/djg/env/kafka_2.12-2.2.2/start.log 2>&1 &
sleep 10s

#启动schema registry
docker run -it --rm --name schema-registry -e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=$ip:2181 -e SCHEMA_REGISTRY_HOST_NAME=schema-registry -e SCHEMA_REGISTRY_LISTENERS=http://0.0.0.0:8081 -p 8081:8081 -v /Users/djg/env/confluent-schema-registry/conf:/etc/schema-registry -d confluentinc/cp-schema-registry
sleep 5s

#启动schema registry ui
docker run --rm -d -p 8000:8000 -e SCHEMAREGISTRY_URL=http://$ip:8081 -e PROXY=true --name schema-registry-ui landoop/schema-registry-ui

停止脚本:

#!/bin/bash
#停止schema-registry和schema-registry-ui的docker容器
docker stop schema-registry
docker stop schema-registry-ui
sleep 5s

#停止zookeeper、kafka broker、kafka connect
connectip=`jps | awk -F" " '$2=="ConnectDistributed"{print $1}'`
zookeeperip=`jps | awk -F" " '$2=="QuorumPeerMain"{print $1}'`
brokerip=`jps | awk -F" " '$2=="Kafka"{print $1}'`
kill -9 $connectip $zookeeperip $brokerip

说明:
shcema-registry和schema-registry-ui是通过docker启动的,因为这些服务只是辅助服务,能用就行。而kafka broker和kafka connect是非docker启动的,方便熟悉其配置。

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章