Kafka安全認證：SASL/GSSAPI(kerberos)

kafka和zookeeper開啓kerberos認證

1. 環境

kafka版本：2.12-2.3.0

zookeeper版本：3.6.0

操作系統：CentOS7

2. 創建主體並生成keytab

在kerberos中，用戶和服務是平等的關係，都是以principal的形式存在

$ kadmin.local
# kafka broker，zookeeper，kafka client主體
# 其中stream.dt.local表示kafka broker所在主機的FQDN，Fully Qualified Domain Name的縮寫, 含義是完整的域名
$ kadmin.local:  addprinc kafka/[email protected]
$ kadmin.local:  addprinc zookeeper/[email protected]
$ kadmin.local:  addprinc clients/[email protected]

# 生成主體對應的keytab文件
$ kadmin.local:  xst -k /opt/third/kafka/kerberos/kafka_server.keytab
$ kadmin.local:  xst -k /opt/third/zookeeper/kerberos/kafka_zookeeper.keytab
$ kadmin.local:  xst -k /opt/third/kafka/kerberos/kafka_client.keytab

# 給keytab賦予可讀權限
$ chmod -R 777 /opt/third/kafka/kerberos/kafka_server.keytab
$ chmod -R 777 /opt/third/zookeeper/kerberos/kafka_zookeeper.keytab
$ chmod -R 777 /opt/third/kafka/kerberos/kafka_client.keytab

設置FQDN的方式

$ cat /etc/hostname
demo-db
$ vim /etc/hosts
192.168.90.88  stream.dt.local demo-db
# 192.168.90.88是本機ip，stream.dt.local是要設置的FQDN，demo-db是主機名

3. 配置jaas.conf

/opt/third/kafka/kerberos/kafka_server_jaas.conf

KafkaServer {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    keyTab="/opt/third/kafka/kerberos/kafka_server.keytab"
    principal="kafka/[email protected]";
};
# 此context名字爲ZKClient，對應kafka broker啓動參數-Dzookeeper.sasl.client=ZkClient
ZkClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    keyTab="/opt/third/kafka/kerberos/kafka_server.keytab"
    principal="kafka/[email protected]";
};

/opt/third/zookeeper/kerberos/zookeeper_jaas.conf

Server {
    com.sun.security.auth.module.Krb5LoginModule required debug=true
    useKeyTab=true
    storeKey=true
    useTicketCache=false
    keyTab="/opt/third/zookeeper/kerberos/kafka_zookeeper.keytab"
    principal="zookeeper/[email protected]";
};

/opt/third/kafka/kerberos/kafka_client_jaas.conf

# 此context名字爲KafkaClient，對應kafka consumer啓動參數-Dzookeeper.sasl.client=KafkaClient
KafkaClient {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   storeKey=true
   keyTab="/opt/third/kafka/kerberos/kafka_client.keytab"
   principal="clients/[email protected]";
};

4. 配置kafka server.properties

listeners=SASL_PLAINTEXT://stream.dt.local:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanisms=GSSAPI
sasl.kerberos.service.name=kafka

5. 配置kafka zookeeper.properties

authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=3600000

6. kafka broker+zookeeper啓動腳本

# 啓動zookeeper
export KAFKA_OPTS='-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/opt/third/zookeeper/kerberos/zookeeper_jaas.conf'

/opt/third/zookeeper/bin/zkServer.sh start >> /opt/third/kafka/start.log 2>&1

# 啓動kafka
export KAFKA_OPTS='-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/opt/third/kafka/kerberos/kafka_server_jaas.conf -Dzookeeper.sasl.client=ZkClient'

JMX_PORT=9988 nohup /opt/third/kafka/bin/kafka-server-start.sh /opt/third/kafka/config/server.properties >> /opt/third/kafka/start.log 2>&1 &

7. kafka client的使用

通過kafka client與kafka broker進行交互時，之間傳輸的TGT等信息都需要通過加密算法進行加密，當使用AES256算法加密時，由於受到美國軟件出口的管制，需要覆蓋%JAVA_HOME%\jre\lib\security下的local_policy.jar和US_export_policy.jar，下載地址

7.1 producer

# --bootstrap-server後接FQDN+port，不能接localhost
export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/opt/third/kafka/kerberos/kafka_client_jaas.conf -Dzookeeper.sasl.client=KafkaClient"

sh biu/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config /opt/third/kafka/config/producer.properties

其中producer.properties的內容如下：

security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka

7.2 consumer

# --bootstrap-server後接FQDN+port，不能接localhost
export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/opt/third/kafka/kerberos/kafka_client_jaas.conf -Dzookeeper.sasl.client=KafkaClient"

sh bin/kafka-console-consumer.sh --bootstrap-server stream.dt.local:9092 --topic test --from-beginning --consumer.config /opt/third/kafka/config/consumer.properties

其中，consumer.properties的內容如下：

security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka

參考：

https://simple.wikipedia.org/wiki/Kerberos_(protocol)

https://www.orchome.com/436

https://community.cloudera.com/t5/Support-Questions/Kafka-client-code-does-not-currently-support-obtaining-a/td-p/283879

https://www.cnblogs.com/bdicaprio/articles/10096250.html

https://stackoverflow.com/questions/43469962/kafka-sasl-zookeeper-authentication

https://docs.confluent.io/4.1.1/kafka/authentication_sasl_gssapi.html#

https://www.orchome.com/326

https://www.orchome.com/1944

https://www.orchome.com/500