Kafka SSL服务配置及客户端使用(Linux+Pykafka)

青旅半醒 2023-06-17 02:59 10阅读 0赞

内容:
      1: Kafka及Zookeeper快速安装配置及测试
      2: Kafka SSL服务端配置
      3: Kafka 客户端
        3.1: Linux下配置及测试
        3.2: Pykafka配置及测试
      4: 脚本
      5: 参考借鉴

**1.Kafka及Zookeeper快速安装配置及测试**  
[见此篇： https://www.cnblogs.com/wonglu/p/8687488.html][https_www.cnblogs.com_wonglu_p_8687488.html]

**2.kafka SSL服务端配置**

[root@kafka ~]# mkdir -p /root/round1
    [root@kafka ~]# cd /root/round1
    [root@kafka round1]# chmod +x test.sh
    [root@kafka round1]# ./test.sh #执行所有项
    [root@kafka round1]# vim /opt/kafka_2.11-1.1.0/config/server.properties
    listeners=PLAINTEXT://:9092,SSL://:9093
    advertised.listeners=PLAINTEXT://本机IP:9092,SSL://本机IP:9093

`test.sh 脚本`见第4项;脚本生成的文件均在round1，必须在当前路径下执行脚本项，其他路径可能会出现文件未找到(file not found)。  
**3.kafka客户端**  
**3.1Linux下配置及测试**  
拷贝round1文件夹下的client.keystore.jks 、client.truststore.jks、test.sh到其他机器上，三者必须在同一路径

`配置：`

[root@kafka round1]# scp client.keystore.jks client.truststore.jks test.sh root@client:/root/
    [root@client ~]# ./test.sh # 只需执行脚本第1、4项即可

`测试：`  
从官网拉取kafka.tar.gz包，并解压

[root@client ~]# tar -zxvf kafka_2.11-1.1.0.tgz
    [root@client ~]# /root/kafka_2.11-1.1.0/bin/kafka-console-producer.sh --broker-list <kafkaip>:9093 --topic test --producer.config /root/clientssl.properties 
    >a
    >b
    >
    [root@client ~]# /root/kafka_2.11-1.1.0/bin/kafka-console-consumer.sh --bootstrap-server <kafkaip>:9093 --topic test --from-beginning --consumer.config /root/clientssl.properties 
    a
    b

`如上所示：生产消费均正常。`

**3.2Pykafka配置及测试**  
拷贝round1文件夹下的ca.crt、ca.key到D:\\hhx\\xxx\\下  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4OTAwNTY1_size_16_color_FFFFFF_t_70]

**4.脚本：**

#!/bin/bash
    ########################【Personality configuration item】#######################
    BASE_DIR=`echo $PWD`"/"      
    DOMAIN='machine03.zheng.com'
    Password='leonzheng'
    D_NAME='C=CN,ST=FJ,L=FZ,O=LEON,OU=LEON,CN=ZHENG.COM'
    CA_DOMAIN='/C=CN/ST=FJ/L=FZ/O=LEON/OU=LEON/CN=ZHENG.COM'
    ####################################【 END】 ####################################
    
    list_items(){ 
            echo "#####################################################" 
            echo "# kafka by ssl #"
            echo "#---------------------------------------------------#"
            echo "# 1.Env check 2.ssl tools 3.ssl server #"
            echo "# 4.ssl client 5.exit #"
            echo "#####################################################"
    }
    env_check(){ 
            java -version &> /dev/null
            if [ $? -ne 0 ];then
               echo -e "\033[31m java not found!\033[0m" 
               break
            fi
            keytool &> /dev/null
            if [ $? -ne 0 ];then
               echo -e "\033[31m java home not configure\033[0m" 
               break
            fi
            openssl version &> /dev/null
            if [ $? -ne 0 ];then
               echo -e "\033[31m openssl command not fount\033[0m" 
               break
            fi
            echo -e "\033[32m environment is OK\033[0m" 
    
    }
    create_ssl_tools(){ 
            echo $5
            echo "将生成本机进行双向验证的证书密钥工具"
    # 生成服务器keystore(密钥和证书)
            keytool -keystore server.keystore.jks -alias $2 -validity 365 -keyalg RSA -storepass $3 -keypass $3 -genkey -dname "$4"
    # 生成客户端keystore(密钥和证书)
            keytool -keystore client.keystore.jks -alias $2 -validity 365 -keyalg RSA -storepass $3 -keypass $3 -genkey -dname "$4"
    # 创建CA证书
            openssl req -new -x509 -keyout ca.key -out ca.crt -days 365 -passout pass:$3 -subj "$5"
    # 将CA证书导入到服务器truststore
            keytool -keystore server.truststore.jks -alias CARoot -import -file ca.crt -storepass $3
    # 将CA证书导入到客户端truststore
            keytool -keystore client.truststore.jks -alias CARoot -import -file ca.crt -storepass $3
    # 导出服务器证书
            keytool -keystore server.keystore.jks -alias $2 -certreq -file cert-file -storepass $3
            keytool -keystore client.keystore.jks -alias $2 -certreq -file client-cert-file -storepass $3
    # 用CA证书给服务器证书签名
            openssl x509 -req -CA ca.crt -CAkey ca.key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:$3
            openssl x509 -req -CA ca.crt -CAkey ca.key -in client-cert-file -out client-cert-signed -days 365 -CAcreateserial -passin pass:$3
    # 将CA证书导入服务器keystore
            keytool -keystore server.keystore.jks -alias CARoot -import -file ca.crt -storepass $3
            keytool -keystore client.keystore.jks -alias CARoot -import -file ca.crt -storepass $3
    # 将已签名的服务器证书导入服务器keystore
            keytool -keystore server.keystore.jks -alias $2 -import -file cert-signed -storepass $3
            keytool -keystore client.keystore.jks -alias $2 -import -file client-cert-signed -storepass $3
    }
    config_ssl_client(){ 
            if [ ! -f $1client.truststore.jks ];then
                    echo "$1client.truststore.jks" not found
                    break
            fi
            if [ ! -f $1client.keystore.jks ];then
                    echo "$1client.keystore.jks" not found
                    break
            fi
            echo "security.protocol=SSL ssl.truststore.location=$1client.truststore.jks ssl.truststore.password=$3 ssl.keystore.location=$1client.keystore.jks ssl.keystore.password=$3 ssl.key.password=$3" > $1"clientssl.properties"
    }
    config_ssl_server(){ 
            if [ ! -n "$6" ];then
                    echo -e "\033[31m [Error]\033[0m \t <server.properties> file not found" 
                    break
            fi
            if [ ! -f $1server.truststore.jks ];then
                    echo "$1server.truststore.jks" not found
                    break
            fi
            if [ ! -f $1server.keystore.jks ];then
                    echo "$1server.keystore.jks" not found
                    break
            fi
            echo "ssl.client.auth=required ssl.keystore.location=$1server.keystore.jks ssl.keystore.password=$3 ssl.key.password=$3 ssl.truststore.location=$1server.truststore.jks ssl.truststore.password=$3 " >> $6
    }
    ssl_config(){ 
            echo "kafka服务器也是一个客户端，该步骤中，服务器证书将等同于客户端证书"
            create_ssl_tools $1 $2 $3 $4 $5
    }
    while :
    do
      clear
      list_items
      read -p "Input your option:" option
      case $option in
        1) echo -e "\033[32m Starting env check......\033[0m"
           env_check
        ;;
        2) echo -e "\033[32m localhost ssl tools for server and client\033[0m"
           echo "相关证书密钥在该路径下产生:" $BASE_DIR
           ssl_config $BASE_DIR $DOMAIN $Password $D_NAME $CA_DOMAIN
        ;;
        3) echo -e "\033[32m [start]\033[0m \tconfig ssl server"
           config_file=`find / -name 'server.properties'`
           config_ssl_server $BASE_DIR $DOMAIN $Password $D_NAME $CA_DOMAIN $config_file
        ;;
        4) echo -e "\033[32m [start]\033[0m \tconfig ssl client"
           config_ssl_client $BASE_DIR $DOMAIN $Password $D_NAME $CA_DOMAIN
        ;;
        5) echo -e "\033[32m [Finish]\033[0m \tprogram exit."
           break
        ;;
        *) echo -e "\033[31m [Error]\033[0m \tOption not in items!"
      esac
      read -p "Input <enter> to continue..."
    done

**5.参考借鉴：**  
1.SSL加密和认证：https://www.orchome.com/171  
2.SSL认证：https://blog.csdn.net/justry\_deng/article/details/88383707  
3.python生产者kafka报bytes错:https://blog.csdn.net/qq\_35304570/article/details/81101395  
4.Kafka Security 配置SSL: https://blog.csdn.net/difffate/article/details/53570344  
5.集群配置及java端测试，附加ACL：https://blog.csdn.net/zbdba/article/details/52458654  
6.云环境：http://www.voidcn.com/article/p-nwzohovi-bpy.html  
7.双向认证：https://blog.csdn.net/hohoo1990/article/details/79110031

[https_www.cnblogs.com_wonglu_p_8687488.html]: https://www.cnblogs.com/wonglu/p/8687488.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4OTAwNTY1_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191127153413332.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM4OTAwNTY1,size_16,color_FFFFFF,t_70