Kubernetes监控：Dashbaord 2.0.0部署之证书创建和设定

ゝ一世哀愁。 2023-07-03 04:55 6阅读 0赞

在前面一篇文章中介绍了Dashboard 2.0.0-rc3的使用，但是证书的创建和设定使用缺省方式，在这篇文章中将进一步进行说明如何从外部创建和指定证书。

# Dashboard的部署方式 #

详细的说明可参看下文：

*  [https://liumiaocn.blog.csdn.net/article/details/104144132][https_liumiaocn.blog.csdn.net_article_details_104144132]

# 证书请求文件 #

准备如下CSR文件：

[root@host131 k8s]# cat dashboard-csr.json 
    { 
        "CN": "kubernetes-dashboard",
        "hosts": [],
        "key": { 
            "algo": "rsa",
            "size": 2048
        },
        "names": [
            { 
                "C": "CN",
                "L": "DaLian",
                "ST": "LiaoNing",
                "O": "K8S",
                "OU": "System"
            }
        ]
    }
    [root@host131 k8s]#

# 生成证书和私钥 #

使用Kubernetes集群构建时使用的CA和配置设定，使用如下命令生成dashboard的证书和私钥。

> 执行命令：cfssl gencert -ca=/etc/ssl/ca/ca.pem -ca-key=/etc/ssl/ca/ca-key.pem -config=/etc/ssl/ca/ca-config.json -profile=kubernetes dashboard-csr.json | cfssljson -bare dashboard

执行日志示例如下所示：

[root@host131 k8s]# cfssl gencert -ca=/etc/ssl/ca/ca.pem -ca-key=/etc/ssl/ca/ca-key.pem -config=/etc/ssl/ca/ca-config.json -profile=kubernetes dashboard-csr.json | cfssljson -bare dashboard
    2020/02/01 02:29:56 [INFO] generate received request
    2020/02/01 02:29:56 [INFO] received CSR
    2020/02/01 02:29:56 [INFO] generating key: rsa-2048
    2020/02/01 02:29:56 [INFO] encoded CSR
    2020/02/01 02:29:56 [INFO] signed certificate with serial number 178344385453299136705501206545903961019768969733
    2020/02/01 02:29:56 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
    websites. For more information see the Baseline Requirements for the Issuance and Management
    of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
    specifically, section 10.2.3 ("Information Requirements").
    [root@host131 k8s]# ls dashboard*
    dashboard.csr  dashboard-csr.json  dashboard-key.pem  dashboard.pem
    [root@host131 k8s]#

# YAML文件设定 #

将官方建议的YAML文件中去除如下两项设定：

*  namespace的创建
 *  名为kubernetes-dashboard-certs的secret的创建

然后添加args中如下内容

args:
                - --tls-cert-file=dashboard.pem
                - --tls-key-file=dashboard-key.pem

如下的设定可以删除，也可以直接跟在之后，作为找不到文件的处理方式，但本文为了明确使用地是挂在进来的证书，此处直接将下述设定删除

--auto-generate-certificates

还可以使用另外的做法，就是直接将证书的内容（base64编码）直接保存在kubernetes-dashboard-certs的data段内，详细可参看下文方式：

*  https://blog.csdn.net/liumiaocn/article/details/103356320  
    这种方式证书内容以硬编码的方式手动保存，和设定内容混在了一起，这里不再赘述。

# 启动Dashboard服务 #

使用如下步骤即可启动Dashboard服务

## 步骤1: 创建kubernetes-dashboard命令空间 ##

> 执行命令：kubectl create ns kubernetes-dashboard

[root@host131 dashboard]# kubectl create ns kubernetes-dashboard
    namespace/kubernetes-dashboard created
    [root@host131 dashboard]# kubectl get ns |grep dashboard
    kubernetes-dashboard   Active   8s
    [root@host131 dashboard]#

## 步骤2: 创建secrets ##

在kubernetes-dashboard命令空间中创建secrets，注意指定创建包含创建出来的证书和私钥，可使用如下命令进行创建：

> 执行命令： kubectl create secret generic kubernetes-dashboard-certs --from-file=/etc/ssl/k8s -n kubernetes-dashboard

执行日志如下所示

[root@host131 dashboard]# kubectl create secret generic kubernetes-dashboard-certs --from-file=/etc/ssl/k8s -n kubernetes-dashboard
    secret/kubernetes-dashboard-certs created
    [root@host131 dashboard]#

使用kubectl describe命令可以确认到此secret中包含所需要使用的证书和私钥文件，执行日志如下所示

[root@host131 dashboard]# kubectl get secret -n kubernetes-dashboard
    NAME                         TYPE                                  DATA   AGE
    default-token-kx9k4          kubernetes.io/service-account-token   3      3m1s
    kubernetes-dashboard-certs   Opaque                                34     16s
    [root@host131 dashboard]# 
    [root@host131 dashboard]# kubectl describe secret kubernetes-dashboard-certs -n kubernetes-dashboard
    Name:         kubernetes-dashboard-certs
    Namespace:    kubernetes-dashboard
    Labels:       <none>
    Annotations:  <none>
    
    Type:  Opaque
    
    Data
    ====
    cert-k8s.csr:                            1253 bytes
    cert-k8scm.csr:                          1062 bytes
    dashboard.csr:                           1013 bytes
    k8s-cm-csr.json:                         346 bytes
    admin-csr.json:                          287 bytes
    cert-aggregator.csr:                     1001 bytes
    kube-scheduler.kubeconfig:               6349 bytes
    dashboard.pem:                           1407 bytes
    kube-controller-manager.kubeconfig:      6401 bytes
    kubeadmin-key.pem:                       1679 bytes
    cert-k8sch-key.pem:                      1675 bytes
    cert-k8sch.csr:                          1050 bytes
    cert-k8sch.pem:                          1424 bytes
    cert-k8scm.pem:                          1436 bytes
    dashboard-key.pem:                       1679 bytes
    kubelet-client-2020-01-31-20-00-18.pem:  1228 bytes
    cert-aggregator.pem:                     1391 bytes
    cert-k8s.pem:                            1619 bytes
    cert-kubeproxy-key.pem:                  1675 bytes
    k8s-csr.json:                            535 bytes
    k8s-sch-csr.json:                        337 bytes
    bootstrap.kubeconfig:                    2169 bytes
    cert-k8scm-key.pem:                      1675 bytes
    cert-aggregator-key.pem:                 1679 bytes
    cert-k8s-key.pem:                        1679 bytes
    cert-kubeproxy.pem:                      1403 bytes
    kubelet-server-2020-01-31-20-00-56.pem:  1252 bytes
    kubeadmin.csr:                           1009 bytes
    kubeproxy-csr.json:                      230 bytes
    aggregator-csr.json:                     281 bytes
    cert-kubeproxy.csr:                      1009 bytes
    dashboard-csr.json:                      291 bytes
    kube-proxy.kubeconfig:                   6271 bytes
    kubeadmin.pem:                           1399 bytes
    [root@host131 dashboard]#

## 步骤3: 启动Dashboard服务 ##

[root@host131 dashboard]# ls
    dashboard.yml
    [root@host131 dashboard]# kubectl create -f .
    serviceaccount/kubernetes-dashboard created
    service/kubernetes-dashboard created
    secret/kubernetes-dashboard-csrf created
    secret/kubernetes-dashboard-key-holder created
    configmap/kubernetes-dashboard-settings created
    role.rbac.authorization.k8s.io/kubernetes-dashboard created
    clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
    rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
    clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
    deployment.apps/kubernetes-dashboard created
    service/dashboard-metrics-scraper created
    deployment.apps/dashboard-metrics-scraper created
    [root@host131 dashboard]#

# 结果确认 #

确认服务和pod等状态

[root@host131 dashboard]# kubectl get pods -A |grep dashboard
    kubernetes-dashboard   dashboard-metrics-scraper-7b8b58dc8b-np442   1/1     Running   0          35s
    kubernetes-dashboard   kubernetes-dashboard-6b479fc8fc-cmpg4        1/1     Running   0          35s
    [root@host131 dashboard]# kubectl get service -A |grep dashboard
    kubernetes-dashboard   dashboard-metrics-scraper   ClusterIP   10.254.56.169   <none>        8000/TCP        43s
    kubernetes-dashboard   kubernetes-dashboard        NodePort    10.254.91.222   <none>        443:33307/TCP   43s
    [root@host131 dashboard]# kubectl get deployment -A |grep dashboard
    kubernetes-dashboard   dashboard-metrics-scraper   1/1     1            1           52s
    kubernetes-dashboard   kubernetes-dashboard        1/1     1            1           52s
    [root@host131 dashboard]#

获取token信息

[root@host131 dashboard]# kubectl get secret -n kubernetes-dashboard
    NAME                               TYPE                                  DATA   AGE
    default-token-kx9k4                kubernetes.io/service-account-token   3      7m26s
    kubernetes-dashboard-certs         Opaque                                34     4m41s
    kubernetes-dashboard-csrf          Opaque                                1      94s
    kubernetes-dashboard-key-holder    Opaque                                2      94s
    kubernetes-dashboard-token-flzjd   kubernetes.io/service-account-token   3      94s
    [root@host131 dashboard]# kubectl describe secret kubernetes-dashboard-token-flzjd -n kubernetes-dashboard
    Name:         kubernetes-dashboard-token-flzjd
    Namespace:    kubernetes-dashboard
    Labels:       <none>
    Annotations:  kubernetes.io/service-account.name: kubernetes-dashboard
                  kubernetes.io/service-account.uid: 8fbee1fa-399a-4d91-b27d-dd55d119b869
    
    Type:  kubernetes.io/service-account-token
    
    Data
    ====
    ca.crt:     1359 bytes
    namespace:  20 bytes
    token:      eyJhbGciOiJSUzI1NiIsImtpZCI6Ikt5aFRwRXR6M1hMLU5GTjg0M0R1LTNfNTZLOVNhVUxZd1BSZW9HNGJrMVEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC10b2tlbi1mbHpqZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjhmYmVlMWZhLTM5OWEtNGQ5MS1iMjdkLWRkNTVkMTE5Yjg2OSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDprdWJlcm5ldGVzLWRhc2hib2FyZCJ9.M0HUoA7Xbe5HR-LgpHtqdUQIgP8LlzfqtMm-lZ8JJaVByHJDd8M7eiICNCS08cFtUkWVb8vF53nzf_MTQufiKxS4qNFwjwoP2wbi_IGx6RwO9VWCSev2EmIiuBceTqHxXZsqRLO-fK8GuGiorW6RQb202k6gZCUTCpAR8QC_vvY8UPU1Y2CzCKaD5XUDvaZ-KFDPcQlxTH89k6BlT06Ik0NNbejSw66k_vPoiDFnbGQEvtRGdqZ2XCzOF8ASUbKcalQL-9dM-vAr1KLuR4sn32hseFPDPSPCXiSFSietS2N6X_Fz3Al6iIFTsCZXfTdG3R0dbO-SKJQ6vNLO4XST5w
    [root@host131 dashboard]#

选择token方式登录，输入上述token信息  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9saXVtaWFvY24uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70]  
点击Sigin in即可登录进入Dashboard界面中了  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9saXVtaWFvY24uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 1]  
这样就使用了外部的证书进行了Dashboard服务的启动。

# 参照YAML文件 #

[root@host131 dashboard]# cat dashboard.yml 
    # Copyright 2017 The Kubernetes Authors.
    #
    # Licensed under the Apache License, Version 2.0 (the "License");
    # you may not use this file except in compliance with the License.
    # You may obtain a copy of the License at
    #
    # http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.
    
    ---
    
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      name: kubernetes-dashboard
      namespace: kubernetes-dashboard
    
    ---
    
    kind: Service
    apiVersion: v1
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      name: kubernetes-dashboard
      namespace: kubernetes-dashboard
    spec:
      type: NodePort
      ports:
        - port: 443
          targetPort: 8443
          nodePort: 33307
      selector:
        k8s-app: kubernetes-dashboard
    
    ---
    
    apiVersion: v1
    kind: Secret
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      name: kubernetes-dashboard-csrf
      namespace: kubernetes-dashboard
    type: Opaque
    data:
      csrf: ""
    
    ---
    
    apiVersion: v1
    kind: Secret
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      name: kubernetes-dashboard-key-holder
      namespace: kubernetes-dashboard
    type: Opaque
    
    ---
    
    kind: ConfigMap
    apiVersion: v1
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      name: kubernetes-dashboard-settings
      namespace: kubernetes-dashboard
    
    ---
    
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      name: kubernetes-dashboard
      namespace: kubernetes-dashboard
    rules:
      # Allow Dashboard to get, update and delete Dashboard exclusive secrets.
      - apiGroups: [""]
        resources: ["secrets"]
        resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
        verbs: ["get", "update", "delete"]
        # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
      - apiGroups: [""]
        resources: ["configmaps"]
        resourceNames: ["kubernetes-dashboard-settings"]
        verbs: ["get", "update"]
        # Allow Dashboard to get metrics.
      - apiGroups: [""]
        resources: ["services"]
        resourceNames: ["heapster", "dashboard-metrics-scraper"]
        verbs: ["proxy"]
      - apiGroups: [""]
        resources: ["services/proxy"]
        resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
        verbs: ["get"]
    
    ---
    
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      name: kubernetes-dashboard
    rules:
      # Allow Metrics Scraper to get metrics from the Metrics server
      - apiGroups: ["metrics.k8s.io"]
        resources: ["pods", "nodes"]
        verbs: ["get", "list", "watch"]
    
    ---
    
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      name: kubernetes-dashboard
      namespace: kubernetes-dashboard
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: kubernetes-dashboard
    subjects:
      - kind: ServiceAccount
        name: kubernetes-dashboard
        namespace: kubernetes-dashboard
    
    ---
    
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: kubernetes-dashboard
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: kubernetes-dashboard
    subjects:
      - kind: ServiceAccount
        name: kubernetes-dashboard
        namespace: kubernetes-dashboard
    
    ---
    
    kind: Deployment
    apiVersion: apps/v1
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      name: kubernetes-dashboard
      namespace: kubernetes-dashboard
    spec:
      replicas: 1
      revisionHistoryLimit: 10
      selector:
        matchLabels:
          k8s-app: kubernetes-dashboard
      template:
        metadata:
          labels:
            k8s-app: kubernetes-dashboard
        spec:
          containers:
            - name: kubernetes-dashboard
              image: kubernetesui/dashboard:v2.0.0-rc3
              imagePullPolicy: IfNotPresent
              ports:
                - containerPort: 8443
                  protocol: TCP
              args:
                - --tls-cert-file=dashboard.pem
                - --tls-key-file=dashboard-key.pem
                - --namespace=kubernetes-dashboard
                # Uncomment the following line to manually specify Kubernetes API server Host
                # If not specified, Dashboard will attempt to auto discover the API server and connect
                # to it. Uncomment only if the default does not work.
                # - --apiserver-host=http://my-address:port
              volumeMounts:
                - name: kubernetes-dashboard-certs
                  mountPath: /certs
                  # Create on-disk volume to store exec logs
                - mountPath: /tmp
                  name: tmp-volume
              livenessProbe:
                httpGet:
                  scheme: HTTPS
                  path: /
                  port: 8443
                initialDelaySeconds: 30
                timeoutSeconds: 30
              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
                runAsUser: 1001
                runAsGroup: 2001
          volumes:
            - name: kubernetes-dashboard-certs
              secret:
                secretName: kubernetes-dashboard-certs
            - name: tmp-volume
              emptyDir: { }
          serviceAccountName: kubernetes-dashboard
          nodeSelector:
            "beta.kubernetes.io/os": linux
          # Comment the following tolerations if Dashboard must not be deployed on master
          tolerations:
            - key: node-role.kubernetes.io/master
              effect: NoSchedule
    
    ---
    
    kind: Service
    apiVersion: v1
    metadata:
      labels:
        k8s-app: dashboard-metrics-scraper
      name: dashboard-metrics-scraper
      namespace: kubernetes-dashboard
    spec:
      ports:
        - port: 8000
          targetPort: 8000
      selector:
        k8s-app: dashboard-metrics-scraper
    
    ---
    
    kind: Deployment
    apiVersion: apps/v1
    metadata:
      labels:
        k8s-app: dashboard-metrics-scraper
      name: dashboard-metrics-scraper
      namespace: kubernetes-dashboard
    spec:
      replicas: 1
      revisionHistoryLimit: 10
      selector:
        matchLabels:
          k8s-app: dashboard-metrics-scraper
      template:
        metadata:
          labels:
            k8s-app: dashboard-metrics-scraper
          annotations:
            seccomp.security.alpha.kubernetes.io/pod: 'runtime/default'
        spec:
          containers:
            - name: dashboard-metrics-scraper
              image: kubernetesui/metrics-scraper:v1.0.3
              ports:
                - containerPort: 8000
                  protocol: TCP
              livenessProbe:
                httpGet:
                  scheme: HTTP
                  path: /
                  port: 8000
                initialDelaySeconds: 30
                timeoutSeconds: 30
              volumeMounts:
              - mountPath: /tmp
                name: tmp-volume
              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
                runAsUser: 1001
                runAsGroup: 2001
          serviceAccountName: kubernetes-dashboard
          nodeSelector:
            "beta.kubernetes.io/os": linux
          # Comment the following tolerations if Dashboard must not be deployed on master
          tolerations:
            - key: node-role.kubernetes.io/master
              effect: NoSchedule
          volumes:
            - name: tmp-volume
              emptyDir: { }
    [root@host131 dashboard]#

# 参考内容 #

https://github.com/kubernetes/dashboard/blob/master/docs/user/certificate-management.md\#self-signed-certificate  
https://github.com/kubernetes/dashboard/blob/master/docs/user/installation.md\#recommended-setup

[https_liumiaocn.blog.csdn.net_article_details_104144132]: https://liumiaocn.blog.csdn.net/article/details/104144132
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9saXVtaWFvY24uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200202203324386.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9saXVtaWFvY24uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9saXVtaWFvY24uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200202203355936.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9saXVtaWFvY24uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70