利用 Kubernetes DaemonSet 解决 DevOps-CD 流程中仓库自签证书分发问题

男娘i 2021-10-01 10:28 102阅读 0赞

## 原理 ##

DaemonSet方式启动Pod，将节点的/etc/docker/certs.d目录映射到Pod容器（类似于docker run -v）**/a**目录，保存有Docker仓库自签证书的ConfigMap以Volume方式挂载到Pod容器**/b**目录，通过K8S的LifeCycle控制器在Pod启动后将**/b**目录内容拷贝到**/a**目录，曲线完成Docker仓库自签名证书的分发。

## 说明 ##

*  适用无主机登陆权限的K8S节点自签证书分发.
 *  ConfigMap名为**docker-cacerts**，可修改为其它名称，可存放多个仓库自签证书，对ConfigMap数据字段有一定要求：

*Key1: Value1*

*Key2: Value2*

Key字段是“仓库地址_端口”模式，仓库地址可以是域名也可以是IP地址，端口为443的情况可以不带下划线和端口（视镜像tag决定Value为Docker仓库自签证书文本内容)

*  DaemonSet与ConfigMap系Namespace级别资源，需注意资源Namespace归属
 *  ConfigMap更新后需Recreate对应Pod

## 步骤 ##

### 1. 创建ConfigMap ###

以Harbor地址为https://10.50.32.35为例

$ obj_ns=deepexi
    $ kubectl create configmap docker-cacerts --from-file=10.50.32.35=ca.crt -n ${obj_ns}
    $ kubectl get configmap docker-cacerts -n ${obj_ns} --export
    apiVersion: v1
    data:
      ca.crt: |
        -----BEGIN CERTIFICATE-----
        MIIFrzCCA5egAwIBAgIJAIwAuuK+h8oIMA0GCSqGSIb3DQEBDQUAMG4xCzAJBgNV
        BAYTAkNOMRIwEAYDVQQIDAlHdWFuZ2RvbmcxEjAQBgNVBAcMCUd1YW5nemhvdTEQ
        MA4GA1UECgwHRGVlcGV4aTEPMA0GA1UECwwGRGV2b3BzMRQwEgYDVQQDDAtkZWVw
        ZXhpLnRvcDAeFw0yMDAzMTEwMjUzMTdaFw0zMDAzMDkwMjUzMTdaMG4xCzAJBgNV
        BAYTAkNOMRIwEAYDVQQIDAlHdWFuZ2RvbmcxEjAQBgNVBAcMCUd1YW5nemhvdTEQ
        MA4GA1UECgwHRGVlcGV4aTEPMA0GA1UECwwGRGV2b3BzMRQwEgYDVQQDDAtkZWVw
        ZXhpLnRvcDCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBANv1RGx8fcv+
        gLIVe1RiVqLyuhDRhwaTMEBvdlL8t5K9e+V1PymAzNYz9eV3UN+Ud+0+XFfW6CCr
        8FLxquhSCBTv2l+cyCZ+2Cntnk9ViV58fSW7dvrdsORD93UdSqEPMX5YjL7otY4/
        D4KOyOvMycmk761bzOqrjqVGSnmkRFSB9wGxbi59irk2EohmtKqW8MGE9NTLIUud
        chGUujIqqOk7zIEFAeNbhZNFp8W/+gEtnybkt3w5YhpiGoTS7n1OJHitpp55yCGM
        c+qU2ph2OCGvjU9baSYsK2tX2QWTeKYw/Ls1KbFe5EtJ3iz6yd2KwUl+c/r5so+d
        oeAbI+mE1bVK5H6/N3TfuWoX4+kxdPtNIKMaQTKLVfbrwPZTUrO5PXaJNoFIGWw4
        PcKh8lDeB8mcApOPJDhsLyWbdbqDcJToWGh36HSI/A9gFuxdFHEM4d7Dp2nT2+sO
        x5/qHIyEoc3u5OEqBtt3HXxkisfTPEh9vhfjLfZTGwDMQcs+EULCR/q9xp94KzK5
        bVDsVSF4remDR8iAL+u0HAEjbfcCWuils1ph2TxIQ6cvdgqVK3+wzDkpl+xZhhsa
        Z9jCoOUZ7NB8q11kSmgFna6DXG4uiyO+nQwRAYrF8zU1vsbSAbGrz8nDqdRPory2
        gGSJJ5W4Nic81Hym4yuhCa7X+QqZSYgfAgMBAAGjUDBOMB0GA1UdDgQWBBRSLuBm
        rHPUGVJBF4SUgRe8QeqyJDAfBgNVHSMEGDAWgBRSLuBmrHPUGVJBF4SUgRe8Qeqy
        JDAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBDQUAA4ICAQDHDegbJTlVUPOtPORL
        +Q9GMKAsp47gOPuJlpuZ/x2++/R1h6r5/nj8iUh4Sp5S+d441SBiZifC0dyaLgnB
        vtTP2zcPE93CB3lq1h5U70gP2iw+kUcvd7jRirkHS4847O34WtOWBEtgw/zcJZHb
        Y24ssnlTxAdZwPHps3gWvgpb1A05JZfU2E99D7BiOH4rYakYB1yeKtR8Z6NCe/Vh
        GXqytCCrYVV+Nr4LA2fICqs8TnnmxKk2yr1mVbWmOBp8zGlNwNNz1c0kOmpYGlen
        or+aUaVC6K8tHzyDFU0oqI+8rEFP0nC2oYz6/kRh0I398APFOFqvCCla9TjoXTye
        kh0RucsWXkIjbxyBsDqbnwEDRVMiEsG8bgkk9uyKCgu+pj/SKyjar55ZKZlXOSdr
        fICZT409CRiB+Ng+g7lw8CPgY2fwwdavgQinZtI+q76VldlNuL+4nMHJzSnnR1Vl
        GXAPZ+Kk5sRWIlHX1qie3f2HKcKBnJnfMeC+n6UvXvyfs8bXNPKhu/NgEwu+hGgW
        W29ZVtqMt9iF1pLtMiS074ciUTNyFN4Ehy7UfYdWH7dpP3fDFMYkdIqBJxreF6TQ
        +RClYXpxSDN6Ri5g9m5z7isxjcwoLr82Ih/tyb7Ir8FUV0Hxj2MzbRD5mxN0TBka
        laPxVfOw5qoEpF6ZFc08u8AP8A==
        -----END CERTIFICATE-----
    kind: ConfigMap
    metadata:
      name: docker-cacerts

### 2. 准备DaemonSet资源文件 ###

apiVersion: extensions/v1beta1
    kind: DaemonSet
    metadata:
      labels:
        deepexi-app: dockerca-mapper
      name: dockerca-mapper
    spec:
      selector:
        matchLabels:
          deepexi-app: dockerca-mapper
      template:
        metadata:
          labels:
            deepexi-app: dockerca-mapper
        spec:
          containers:
          - command:
            - sleep
            - 365d
            image: busybox
            imagePullPolicy: IfNotPresent
            name: dockerca-mapper
            lifecycle:
              postStart:
                exec:
                  command:
                  - /bin/sh
                  - -c
                  - for i in `ls -1 /tmp-docker-certsd`;
                    do
                      cert_dir=`echo $i | sed -e 's/$_$$[0-9]*$$/:\2/g'`;
                      if [ ! -d /docker-certsd/$cert_dir ]; then
                        mkdir -p /docker-certsd/$cert_dir;
                      fi;
                      cp /tmp-docker-certsd/$i /docker-certsd/$cert_dir/ca.crt;
                    done;
            volumeMounts:
            - mountPath: /tmp-docker-certsd
              name: tmp-docker-certsd
              readOnly: false
            - mountPath: /docker-certsd
              name: docker-certsd
              readOnly: false
          tolerations:
          - effect: NoExecute
            operator: Exists
          - effect: NoSchedule
            operator: Exists
          volumes:
          - hostPath:
              path: /etc/docker/certs.d
              type: DirectoryOrCreate
            name: docker-certsd
          - configMap:
              defaultMode: 440
              name: docker-cacerts
            name: tmp-docker-certsd

### 3. 应用配置到K8S集群 ###

$ kubectl apply -f docker-ca-mapper.yaml -n ${obj_ns}

### 4. 验证 ###

*  检查daemonset各节点Pod状态是否Running
 *  检查/etc/docker/certs.d目录下相应仓库地址:端口目录及其下ca.crt文件是否存在
 *  检查docker login仓库地址结果

$ kubectl get pod -n ${obj_ns} -o wide | grep dockerca-mapper
    $ ls -l /etc/docker/certs.d/
    $ docker login 10.50.32.5