自己搭建了kafka connect實時ETL環境,並編寫了shell腳本以方便啓動和停止,具體代碼如下:
啓動腳本:
#!/bin/bash
#腳本涵蓋從啓動kafka connect集羣到schema-registry和schema-registry-ui的整個實時ETL環境的啓動
#日誌清空
echo '' > /Users/djg/env/kafka_2.12-2.2.2/start.log
#替換kafka配置文件中的對外ip
ip=`ifconfig | awk '/inet .* netmask .* broadcast/{print}' | awk -F" " '{print $2}'`;
echo $ip;
sed -i "" "s/\(advertised.listeners=PLAINTEXT:\/\/\).*\(:9092\)/\1$ip\2/" /Users/djg/env/kafka_2.12-2.2.2/config/server.properties
#啓動zookeeper
/Users/djg/env/zookeeper-3.4.11/bin/zkServer.sh start >> /Users/djg/env/kafka_2.12-2.2.2/start.log 2>&1
sleep 5s
#啓動kafka broker
nohup /Users/djg/env/kafka_2.12-2.2.2/bin/kafka-server-start.sh /Users/djg/env/kafka_2.12-2.2.2/config/server.properties >> /Users/djg/env/kafka_2.12-2.2.2/start.log 2>&1 &
sleep 10s
#啓動kafka connect
nohup /Users/djg/env/kafka_2.12-2.2.2/bin/connect-distributed.sh /Users/djg/env/kafka_2.12-2.2.2/config/connect-distributed.properties >> /Users/djg/env/kafka_2.12-2.2.2/start.log 2>&1 &
sleep 10s
#啓動schema registry
docker run -it --rm --name schema-registry -e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=$ip:2181 -e SCHEMA_REGISTRY_HOST_NAME=schema-registry -e SCHEMA_REGISTRY_LISTENERS=http://0.0.0.0:8081 -p 8081:8081 -v /Users/djg/env/confluent-schema-registry/conf:/etc/schema-registry -d confluentinc/cp-schema-registry
sleep 5s
#啓動schema registry ui
docker run --rm -d -p 8000:8000 -e SCHEMAREGISTRY_URL=http://$ip:8081 -e PROXY=true --name schema-registry-ui landoop/schema-registry-ui
停止腳本:
#!/bin/bash
#停止schema-registry和schema-registry-ui的docker容器
docker stop schema-registry
docker stop schema-registry-ui
sleep 5s
#停止zookeeper、kafka broker、kafka connect
connectip=`jps | awk -F" " '$2=="ConnectDistributed"{print $1}'`
zookeeperip=`jps | awk -F" " '$2=="QuorumPeerMain"{print $1}'`
brokerip=`jps | awk -F" " '$2=="Kafka"{print $1}'`
kill -9 $connectip $zookeeperip $brokerip
說明:
shcema-registry和schema-registry-ui是通過docker啓動的,因爲這些服務只是輔助服務,能用就行。而kafka broker和kafka connect是非docker啓動的,方便熟悉其配置。