Kafka SSL服務配置及客戶端使用(Linux+Pykafka)

內容:
  1: Kafka及Zookeeper快速安裝配置及測試
  2: Kafka SSL服務端配置
  3: Kafka 客戶端
    3.1: Linux下配置及測試
    3.2: Pykafka配置及測試
  4: 腳本
  5: 參考借鑑

1.Kafka及Zookeeper快速安裝配置及測試
見此篇： https://www.cnblogs.com/wonglu/p/8687488.html

2.kafka SSL服務端配置

[root@kafka ~]# mkdir -p /root/round1
[root@kafka ~]# cd /root/round1
[root@kafka round1]# chmod +x test.sh
[root@kafka round1]# ./test.sh          #執行所有項
[root@kafka round1]# vim /opt/kafka_2.11-1.1.0/config/server.properties
listeners=PLAINTEXT://:9092,SSL://:9093
advertised.listeners=PLAINTEXT://本機IP:9092,SSL://本機IP:9093

test.sh 腳本見第4項;腳本生成的文件均在round1，必須在當前路徑下執行腳本項，其他路徑可能會出現文件未找到(file not found)。
3.kafka客戶端
3.1Linux下配置及測試
拷貝round1文件夾下的client.keystore.jks 、client.truststore.jks、test.sh到其他機器上，三者必須在同一路徑

配置：

[root@kafka round1]# scp client.keystore.jks client.truststore.jks test.sh root@client:/root/
[root@client ~]# ./test.sh   # 只需執行腳本第1、4項即可

測試：
從官網拉取kafka.tar.gz包，並解壓

[root@client ~]# tar -zxvf kafka_2.11-1.1.0.tgz
[root@client ~]# /root/kafka_2.11-1.1.0/bin/kafka-console-producer.sh --broker-list <kafkaip>:9093 --topic test --producer.config /root/clientssl.properties              
>a
>b
>
[root@client ~]# /root/kafka_2.11-1.1.0/bin/kafka-console-consumer.sh --bootstrap-server <kafkaip>:9093 --topic test --from-beginning --consumer.config /root/clientssl.properties 
a
b

如上所示：生產消費均正常。

3.2Pykafka配置及測試
拷貝round1文件夾下的ca.crt、ca.key到D:\hhx\xxx\下

4.腳本：

#!/bin/bash
########################【Personality configuration item】#######################
BASE_DIR=`echo $PWD`"/"      
DOMAIN='machine03.zheng.com'
Password='leonzheng'
D_NAME='C=CN,ST=FJ,L=FZ,O=LEON,OU=LEON,CN=ZHENG.COM'
CA_DOMAIN='/C=CN/ST=FJ/L=FZ/O=LEON/OU=LEON/CN=ZHENG.COM'
####################################【 END】 ####################################

list_items(){
        echo "#####################################################" 
        echo "#                     kafka by ssl                  #"
        echo "#---------------------------------------------------#"
        echo "# 1.Env check         2.ssl tools      3.ssl server #"
        echo "# 4.ssl client        5.exit                        #"
        echo "#####################################################"
}
env_check(){
        java -version &> /dev/null
        if [ $? -ne 0 ];then
           echo -e "\033[31m java not found!\033[0m" 
           break
        fi
        keytool &> /dev/null
        if [ $? -ne 0 ];then
           echo -e "\033[31m java home not configure\033[0m" 
           break
        fi
        openssl version &> /dev/null
        if [ $? -ne 0 ];then
           echo -e "\033[31m openssl command not fount\033[0m" 
           break
        fi
        echo -e "\033[32m environment is OK\033[0m" 

}
create_ssl_tools(){
        echo $5
        echo "將生成本機進行雙向驗證的證書密鑰工具"
# 生成服務器keystore(密鑰和證書)
        keytool -keystore server.keystore.jks -alias $2 -validity 365 -keyalg RSA -storepass $3 -keypass $3 -genkey -dname "$4"
# 生成客戶端keystore(密鑰和證書)
        keytool -keystore client.keystore.jks -alias $2 -validity 365 -keyalg RSA -storepass $3 -keypass $3 -genkey -dname "$4"
# 創建CA證書
        openssl req -new -x509 -keyout ca.key -out ca.crt -days 365 -passout pass:$3 -subj "$5"
# 將CA證書導入到服務器truststore
        keytool -keystore server.truststore.jks -alias CARoot -import -file ca.crt -storepass $3
# 將CA證書導入到客戶端truststore
        keytool -keystore client.truststore.jks -alias CARoot -import -file ca.crt -storepass $3
# 導出服務器證書
        keytool -keystore server.keystore.jks -alias $2 -certreq -file cert-file -storepass $3
        keytool -keystore client.keystore.jks -alias $2 -certreq -file client-cert-file -storepass $3
# 用CA證書給服務器證書籤名
        openssl x509 -req -CA ca.crt -CAkey ca.key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:$3
        openssl x509 -req -CA ca.crt -CAkey ca.key -in client-cert-file -out client-cert-signed -days 365 -CAcreateserial -passin pass:$3
# 將CA證書導入服務器keystore
        keytool -keystore server.keystore.jks -alias CARoot -import -file ca.crt -storepass $3
        keytool -keystore client.keystore.jks -alias CARoot -import -file ca.crt -storepass $3
# 將已簽名的服務器證書導入服務器keystore
        keytool -keystore server.keystore.jks -alias $2 -import -file cert-signed -storepass $3
        keytool -keystore client.keystore.jks -alias $2 -import -file client-cert-signed -storepass $3
}
config_ssl_client(){
        if [ ! -f $1client.truststore.jks ];then
                echo "$1client.truststore.jks" not found
                break
        fi
        if [ ! -f $1client.keystore.jks ];then
                echo "$1client.keystore.jks" not found
                break
        fi
        echo "security.protocol=SSL
ssl.truststore.location=$1client.truststore.jks
ssl.truststore.password=$3
ssl.keystore.location=$1client.keystore.jks
ssl.keystore.password=$3
ssl.key.password=$3" > $1"clientssl.properties"
}
config_ssl_server(){
        if [ ! -n "$6" ];then
                echo -e "\033[31m [Error]\033[0m \t <server.properties> file not found" 
                break
        fi
        if [ ! -f $1server.truststore.jks ];then
                echo "$1server.truststore.jks" not found
                break
        fi
        if [ ! -f $1server.keystore.jks ];then
                echo "$1server.keystore.jks" not found
                break
        fi
        echo "ssl.client.auth=required
ssl.keystore.location=$1server.keystore.jks
ssl.keystore.password=$3
ssl.key.password=$3
ssl.truststore.location=$1server.truststore.jks
ssl.truststore.password=$3
" >> $6
}
ssl_config(){
        echo "kafka服務器也是一個客戶端，該步驟中，服務器證書將等同於客戶端證書"
        create_ssl_tools $1 $2 $3 $4 $5
}
while :
do
  clear
  list_items
  read -p "Input your option:" option
  case $option in
    1) echo -e "\033[32m Starting env check......\033[0m"
       env_check
    ;;
    2) echo -e "\033[32m localhost ssl tools for server and client\033[0m"
       echo "相關證書密鑰在該路徑下產生:" $BASE_DIR
       ssl_config $BASE_DIR $DOMAIN $Password $D_NAME $CA_DOMAIN
    ;;
    3) echo -e "\033[32m [start]\033[0m \tconfig ssl server"
       config_file=`find / -name 'server.properties'`
       config_ssl_server $BASE_DIR $DOMAIN $Password $D_NAME $CA_DOMAIN $config_file
    ;;
    4) echo -e "\033[32m [start]\033[0m \tconfig ssl client"
       config_ssl_client $BASE_DIR $DOMAIN $Password $D_NAME $CA_DOMAIN
    ;;
    5) echo -e "\033[32m [Finish]\033[0m \tprogram exit."
       break
    ;;
    *) echo -e "\033[31m [Error]\033[0m \tOption not in items!"
  esac
  read -p "Input <enter> to continue..."
done

5.參考借鑑：
1.SSL加密和認證：https://www.orchome.com/171
2.SSL認證：https://blog.csdn.net/justry_deng/article/details/88383707
3.python生產者kafka報bytes錯:https://blog.csdn.net/qq_35304570/article/details/81101395
4.Kafka Security 配置SSL: https://blog.csdn.net/difffate/article/details/53570344
5.集羣配置及java端測試，附加ACL：https://blog.csdn.net/zbdba/article/details/52458654
6.雲環境：http://www.voidcn.com/article/p-nwzohovi-bpy.html
7.雙向認證：https://blog.csdn.net/hohoo1990/article/details/79110031