K8S HPA 基于 Prometheus Metrics

冷不防 2022-12-17 14:52 182阅读 0赞

自动扩展是一种根据资源使用情况自动扩展或缩小工作负载的方法。Kubernetes中的自动缩放有两个维度：Cluster Autoscaler处理节点扩展操作，Horizontal Pod Autoscaler自动扩展部署或副本集中的pod数量。Cluster Autoscaling与Horizontal Pod Autoscaler一起用于动态调整计算能力以及系统满足SLA所需的并行度。虽然Cluster Autoscaler高度依赖托管您的集群的云提供商的基础功能，但HPA可以独立于您的IaaS / PaaS提供商运营。

Horizontal Pod Autoscaler功能最初是在Kubernetes v1.1中引入的，并且从那时起已经发展了很多。HPA缩放容器的版本1基于观察到的CPU利用率，后来基于内存使用情况。在Kubernetes 1.6中，引入了一个新的API Custom Metrics API，使HPA能够访问任意指标。Kubernetes 1.7引入了聚合层，允许第三方应用程序通过将自己注册为API附加组件来扩展Kubernetes API。Custom Metrics API和聚合层使Prometheus等监控系统可以向HPA控制器公开特定于应用程序的指标。

Horizontal Pod Autoscaler实现为一个控制循环，定期查询Resource Metrics API以获取CPU /内存等核心指标和针对特定应用程序指标的Custom Metrics API。

![k8s-hpa][]

以下是为Kubernetes 1.9或更高版本配置HPA v2的分步指南。您将安装提供核心指标的Metrics Server附加组件，然后您将使用演示应用程序根据CPU和内存使用情况展示pod自动扩展。在本指南的第二部分中，您将部署Prometheus和自定义API服务器。您将使用聚合器层注册自定义API服务器，然后使用演示应用程序提供的自定义指标配置HPA。

在开始之前，您需要安装Go 1.8或更高版本并在GOPATH中克隆k8s-prom-hpa repo。

cd $GOPATH
    git clone https://github.com/stefanprodan/k8s-prom-hpa

### 部署 Metrics Server ###

kubernetes Metrics Server是资源使用数据的集群范围聚合器，是Heapster的后继者。度量服务器通过汇集来自kubernetes.summary\_api的数据来收集节点和pod的CPU和内存使用情况。摘要API是一种内存高效的API，用于将数据从Kubelet / cAdvisor传递到度量服务器。

![k8s-hpa-ms][]

在HPA的第一个版本中，您需要Heapster来提供CPU和内存指标，在HPA v2和Kubernetes 1.8中，只有在启用horizontal-pod-autoscaler-use-rest-clients时才需要指标服务器。默认情况下，Kubernetes 1.9中启用了HPA rest客户端。GKE 1.9附带预安装的Metrics Server。

在kube-system命名空间中部署Metrics Server：

kubectl create -f ./metrics-server

一分钟后，度量服务器开始报告节点和pod的CPU和内存使用情况。

查看nodes metrics：

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq .

结果如下：

{
      "kind": "NodeMetricsList",
      "apiVersion": "metrics.k8s.io/v1beta1",
      "metadata": {
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes"
      },
      "items": [
        {
          "metadata": {
            "name": "ip-10-1-50-61.ec2.internal",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/ip-10-1-50-61.ec2.internal",
            "creationTimestamp": "2019-02-13T08:34:05Z"
          },
          "timestamp": "2019-02-13T08:33:38Z",
          "window": "30s",
          "usage": {
            "cpu": "78322168n",
            "memory": "563180Ki"
          }
        },
        {
          "metadata": {
            "name": "ip-10-1-57-40.ec2.internal",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/ip-10-1-57-40.ec2.internal",
            "creationTimestamp": "2019-02-13T08:34:05Z"
          },
          "timestamp": "2019-02-13T08:33:42Z",
          "window": "30s",
          "usage": {
            "cpu": "48926263n",
            "memory": "554472Ki"
          }
        },
        {
          "metadata": {
            "name": "ip-10-1-62-29.ec2.internal",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/ip-10-1-62-29.ec2.internal",
            "creationTimestamp": "2019-02-13T08:34:05Z"
          },
          "timestamp": "2019-02-13T08:33:36Z",
          "window": "30s",
          "usage": {
            "cpu": "36700681n",
            "memory": "326088Ki"
          }
        }
      ]
    }

查看pods metrics:

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods" | jq .

结果如下：

{
      "kind": "PodMetricsList",
      "apiVersion": "metrics.k8s.io/v1beta1",
      "metadata": {
        "selfLink": "/apis/metrics.k8s.io/v1beta1/pods"
      },
      "items": [
        {
          "metadata": {
            "name": "kube-proxy-77nt2",
            "namespace": "kube-system",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-77nt2",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:00Z",
          "window": "30s",
          "containers": [
            {
              "name": "kube-proxy",
              "usage": {
                "cpu": "2370555n",
                "memory": "13184Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "cluster-autoscaler-n2xsl",
            "namespace": "kube-system",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/cluster-autoscaler-n2xsl",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:12Z",
          "window": "30s",
          "containers": [
            {
              "name": "cluster-autoscaler",
              "usage": {
                "cpu": "1477997n",
                "memory": "54584Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "core-dns-autoscaler-b4785d4d7-j64xd",
            "namespace": "kube-system",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/core-dns-autoscaler-b4785d4d7-j64xd",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:08Z",
          "window": "30s",
          "containers": [
            {
              "name": "autoscaler",
              "usage": {
                "cpu": "191293n",
                "memory": "7956Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "spot-interrupt-handler-8t2xk",
            "namespace": "kube-system",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/spot-interrupt-handler-8t2xk",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:04Z",
          "window": "30s",
          "containers": [
            {
              "name": "spot-interrupt-handler",
              "usage": {
                "cpu": "844907n",
                "memory": "4608Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "kube-proxy-t5kqm",
            "namespace": "kube-system",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-t5kqm",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:08Z",
          "window": "30s",
          "containers": [
            {
              "name": "kube-proxy",
              "usage": {
                "cpu": "1194766n",
                "memory": "12204Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "kube-proxy-zxmqb",
            "namespace": "kube-system",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-zxmqb",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:06Z",
          "window": "30s",
          "containers": [
            {
              "name": "kube-proxy",
              "usage": {
                "cpu": "3021117n",
                "memory": "13628Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "aws-node-rcz5c",
            "namespace": "kube-system",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/aws-node-rcz5c",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:15Z",
          "window": "30s",
          "containers": [
            {
              "name": "aws-node",
              "usage": {
                "cpu": "1217989n",
                "memory": "24976Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "aws-node-z2qxs",
            "namespace": "kube-system",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/aws-node-z2qxs",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:15Z",
          "window": "30s",
          "containers": [
            {
              "name": "aws-node",
              "usage": {
                "cpu": "1025780n",
                "memory": "46424Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "php-apache-899d75b96-8ppk4",
            "namespace": "default",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/php-apache-899d75b96-8ppk4",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:08Z",
          "window": "30s",
          "containers": [
            {
              "name": "php-apache",
              "usage": {
                "cpu": "24612n",
                "memory": "27556Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "load-generator-779c5f458c-9sglg",
            "namespace": "default",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/load-generator-779c5f458c-9sglg",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:34:56Z",
          "window": "30s",
          "containers": [
            {
              "name": "load-generator",
              "usage": {
                "cpu": "0",
                "memory": "336Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "aws-node-v9jxs",
            "namespace": "kube-system",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/aws-node-v9jxs",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:00Z",
          "window": "30s",
          "containers": [
            {
              "name": "aws-node",
              "usage": {
                "cpu": "1303458n",
                "memory": "28020Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "kube2iam-m2ktt",
            "namespace": "kube-system",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube2iam-m2ktt",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:11Z",
          "window": "30s",
          "containers": [
            {
              "name": "kube2iam",
              "usage": {
                "cpu": "1328864n",
                "memory": "9724Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "kube2iam-w9cqf",
            "namespace": "kube-system",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube2iam-w9cqf",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:03Z",
          "window": "30s",
          "containers": [
            {
              "name": "kube2iam",
              "usage": {
                "cpu": "1294379n",
                "memory": "8812Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "custom-metrics-apiserver-657644489c-pk8rb",
            "namespace": "monitoring",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/monitoring/pods/custom-metrics-apiserver-657644489c-pk8rb",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:04Z",
          "window": "30s",
          "containers": [
            {
              "name": "custom-metrics-apiserver",
              "usage": {
                "cpu": "22409370n",
                "memory": "42468Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "kube2iam-qghgt",
            "namespace": "kube-system",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube2iam-qghgt",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:11Z",
          "window": "30s",
          "containers": [
            {
              "name": "kube2iam",
              "usage": {
                "cpu": "2078992n",
                "memory": "16356Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "spot-interrupt-handler-ps745",
            "namespace": "kube-system",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/spot-interrupt-handler-ps745",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:10Z",
          "window": "30s",
          "containers": [
            {
              "name": "spot-interrupt-handler",
              "usage": {
                "cpu": "611566n",
                "memory": "4336Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "coredns-68fb7946fb-2xnpp",
            "namespace": "kube-system",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-68fb7946fb-2xnpp",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:12Z",
          "window": "30s",
          "containers": [
            {
              "name": "coredns",
              "usage": {
                "cpu": "1610381n",
                "memory": "10480Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "coredns-68fb7946fb-9ctjf",
            "namespace": "kube-system",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-68fb7946fb-9ctjf",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:13Z",
          "window": "30s",
          "containers": [
            {
              "name": "coredns",
              "usage": {
                "cpu": "1418850n",
                "memory": "9852Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "prometheus-7d4f6d4454-v4fnd",
            "namespace": "monitoring",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/monitoring/pods/prometheus-7d4f6d4454-v4fnd",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:00Z",
          "window": "30s",
          "containers": [
            {
              "name": "prometheus",
              "usage": {
                "cpu": "17951807n",
                "memory": "202316Ki"
              }
            }
          ]
        },
        {
          "metadata": {
            "name": "metrics-server-7cdd54ccb4-k2x7m",
            "namespace": "kube-system",
            "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/metrics-server-7cdd54ccb4-k2x7m",
            "creationTimestamp": "2019-02-13T08:35:19Z"
          },
          "timestamp": "2019-02-13T08:35:04Z",
          "window": "30s",
          "containers": [
            {
              "name": "metrics-server-nanny",
              "usage": {
                "cpu": "144656n",
                "memory": "5716Ki"
              }
            },
            {
              "name": "metrics-server",
              "usage": {
                "cpu": "568327n",
                "memory": "16268Ki"
              }
            }
          ]
        }
      ]
    }

### 基于CPU和内存使用情况的Auto Scaling ###

您将使用基于Golang的小型Web应用程序来测试Horizontal Pod Autoscaler（HPA）。

将podinfo部署到默认命名空间：

kubectl create -f ./podinfo/podinfo-svc.yaml,./podinfo/podinfo-dep.yaml

使用NodePort服务访问podinfo，地址为http：// <K8S\_PUBLIC\_IP>：31198。

接下来定义一个至少维护两个副本的HPA，如果CPU平均值超过80％或内存超过200Mi，则最多可扩展到10个：

apiVersion: autoscaling/v2beta1
    kind: HorizontalPodAutoscaler
    metadata:
      name: podinfo
    spec:
      scaleTargetRef:
        apiVersion: extensions/v1beta1
        kind: Deployment
        name: podinfo
      minReplicas: 2
      maxReplicas: 10
      metrics:
      - type: Resource
        resource:
          name: cpu
          targetAverageUtilization: 80
      - type: Resource
        resource:
          name: memory
          targetAverageValue: 200Mi

创建这个hpa:

kubectl create -f ./podinfo/podinfo-hpa.yaml

几秒钟后，HPA控制器联系度量服务器，然后获取CPU和内存使用情况：

kubectl get hpa
    
    NAME      REFERENCE            TARGETS                      MINPODS   MAXPODS   REPLICAS   AGE
    podinfo   Deployment/podinfo   2826240 / 200Mi, 15% / 80%   2         10        2          5m

为了增加CPU使用率，请使用rakyll / hey运行负载测试：

#install hey
    go get -u github.com/rakyll/hey
    
    #do 10K requests
    hey -n 10000 -q 10 -c 5 http://<K8S_PUBLIC_IP>:31198/

您可以使用以下方式监控HPA事件：

$ kubectl describe hpa
    
    Events:
      Type    Reason             Age   From                       Message
      ----    ------             ----  ----                       -------
      Normal  SuccessfulRescale  7m    horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
      Normal  SuccessfulRescale  3m    horizontal-pod-autoscaler  New size: 8; reason: cpu resource utilization (percentage of request) above target

暂时删除podinfo。稍后将在本教程中再次部署它：

kubectl delete -f ./podinfo/podinfo-hpa.yaml,./podinfo/podinfo-dep.yaml,./podinfo/podinfo-svc.yaml

### 部署 Custom Metrics Server ###

要根据自定义指标进行扩展，您需要拥有两个组件。一个组件，用于从应用程序收集指标并将其存储在Prometheus时间序列数据库中。第二个组件使用collect（k8s-prometheus-adapter）提供的指标扩展了Kubernetes自定义指标API。

![k8s-hpa-prom][]

您将在专用命名空间中部署Prometheus和适配器。

创建**monitoring**命名空间：

kubectl create -f ./namespaces.yaml

在**monitoring**命名空间中部署Prometheus v2：

kubectl create -f ./prometheus

生成Prometheus适配器所需的TLS证书：

make certs

生成以下几个文件：

# ls output
    apiserver.csr  apiserver-key.pem  apiserver.pem

部署Prometheus自定义指标API适配器：

kubectl create -f ./custom-metrics-api

列出Prometheus提供的自定义指标：

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .

获取**monitoring**命名空间中所有pod的FS使用情况：

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitoring/pods/*/fs_usage_bytes"| jq .

查询结果如下：

{
      "kind": "MetricValueList",
      "apiVersion": "custom.metrics.k8s.io/v1beta1",
      "metadata": {
        "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitoring/pods/%2A/fs_usage_bytes"
      },
      "items": [
        {
          "describedObject": {
            "kind": "Pod",
            "namespace": "monitoring",
            "name": "custom-metrics-apiserver-657644489c-pk8rb",
            "apiVersion": "/v1"
          },
          "metricName": "fs_usage_bytes",
          "timestamp": "2019-02-13T08:52:30Z",
          "value": "94253056"
        },
        {
          "describedObject": {
            "kind": "Pod",
            "namespace": "monitoring",
            "name": "prometheus-7d4f6d4454-v4fnd",
            "apiVersion": "/v1"
          },
          "metricName": "fs_usage_bytes",
          "timestamp": "2019-02-13T08:52:30Z",
          "value": "24576"
        }
      ]
    }

### 基于custom metrics 自动伸缩 ###

在默认命名空间中创建podinfo NodePort服务和部署：

kubectl create -f ./podinfo/podinfo-svc.yaml,./podinfo/podinfo-dep.yaml

podinfo应用程序公开名为http\_requests\_total的自定义指标。Prometheus适配器删除\_total后缀并将度量标记为计数器度量标准。

从自定义指标API获取每秒的总请求数：

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests" | jq .

{
      "kind": "MetricValueList",
      "apiVersion": "custom.metrics.k8s.io/v1beta1",
      "metadata": {
        "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests"
      },
      "items": [
        {
          "describedObject": {
            "kind": "Pod",
            "namespace": "default",
            "name": "podinfo-6b86c8ccc9-kv5g9",
            "apiVersion": "/__internal"
          },
          "metricName": "http_requests",
          "timestamp": "2018-01-10T16:49:07Z",
          "value": "901m"
        },
        {
          "describedObject": {
            "kind": "Pod",
            "namespace": "default",
            "name": "podinfo-6b86c8ccc9-nm7bl",
            "apiVersion": "/__internal"
          },
          "metricName": "http_requests",
          "timestamp": "2018-01-10T16:49:07Z",
          "value": "898m"
        }
      ]
    }

建一个HPA，如果请求数超过每秒10个，将扩展podinfo部署：

apiVersion: autoscaling/v2beta1
    kind: HorizontalPodAutoscaler
    metadata:
      name: podinfo
    spec:
      scaleTargetRef:
        apiVersion: extensions/v1beta1
        kind: Deployment
        name: podinfo
      minReplicas: 2
      maxReplicas: 10
      metrics:
      - type: Pods
        pods:
          metricName: http_requests
          targetAverageValue: 10

在默认命名空间中部署podinfo HPA：

kubectl create -f ./podinfo/podinfo-hpa-custom.yaml

几秒钟后，HPA从指标API获取http\_requests值：

kubectl get hpa
    
    NAME      REFERENCE            TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
    podinfo   Deployment/podinfo   899m / 10   2         10        2          1m

在podinfo服务上应用一些负载，每秒25个请求：

#install hey
    go get -u github.com/rakyll/hey
    
    #do 10K requests rate limited at 25 QPS
    hey -n 10000 -q 5 -c 5 http://<K8S-IP>:31198/healthz

几分钟后，HPA开始扩展部署：

kubectl describe hpa
    
    Name:                       podinfo
    Namespace:                  default
    Reference:                  Deployment/podinfo
    Metrics:                    ( current / target )
      "http_requests" on pods:  9059m / 10
    Min replicas:               2
    Max replicas:               10
    
    Events:
      Type    Reason             Age   From                       Message
      ----    ------             ----  ----                       -------
      Normal  SuccessfulRescale  2m    horizontal-pod-autoscaler  New size: 3; reason: pods metric http_requests above target

按照当前的每秒请求速率，部署永远不会达到10个pod的最大值。三个复制品足以使每个吊舱的RPS保持在10以下。

负载测试完成后，HPA会将部署缩到其初始副本：

Events:
      Type    Reason             Age   From                       Message
      ----    ------             ----  ----                       -------
      Normal  SuccessfulRescale  5m    horizontal-pod-autoscaler  New size: 3; reason: pods metric http_requests above target
      Normal  SuccessfulRescale  21s   horizontal-pod-autoscaler  New size: 2; reason: All metrics below target

您可能已经注意到自动缩放器不会立即对使用峰值做出反应。默认情况下，度量标准同步每30秒发生一次，只有在最后3-5分钟内没有重新缩放时才能进行扩展/缩小。通过这种方式，HPA可以防止快速执行冲突的决策，并为Cluster Autoscaler提供时间。

### 结论 ###

并非所有系统都可以通过单独依赖CPU /内存使用指标来满足其SLA，大多数Web和移动后端需要基于每秒请求进行自动扩展以处理任何流量突发。对于ETL应用程序，可以通过作业队列长度超过某个阈值等来触发自动缩放。通过使用Prometheus检测应用程序并公开正确的自动缩放指标，您可以对应用程序进行微调，以更好地处理突发并确保高可用性。

\- END -

[![format_png][]][format_png 1]

![format_png 2][]

[Kubernetes全栈技术培训][Kubernetes]

## 技术交流 ##

学无止境，了解更多关于kubernetes/docker/devops/openstack/openshift/linux/IaaS/PaaS相关内容，想要获取更多资料和免费视频，可按如下方式进入技术交流群

![format_png 3][]

**微信：**luckylucky421302

## 微信公众号    ##

**                                    ** **长按指纹关注公众号????**

![format_png 4][]

往期精彩文章

[kubernetes全栈技术+企业案例演示【带你快速掌握和使用k8s】][kubernetes_k8s]

[突破运维和开发瓶颈、Python、k8s、DevOps转型一网打尽！][Python_k8s_DevOps]

[python运维开发实战-基础篇  
][python_-_]

[python运维和开发实战-高级篇][python_-]

[python运维和开发实战-安装和创建Django项目][python_-_Django]

[谈谈我的IT发展之路][IT]

[Prometheus+Grafana+Alertmanager搭建全方位的监控告警系统-超详细文档][Prometheus_Grafana_Alertmanager_-]

[k8s1.18多master节点高可用集群安装-超详细中文官方文档][k8s1.18_master_-]

[linux面试题汇总][linux]

[python运维和开发实战-安装和创建Django项目][python_-_Django]

[Docker公司禁止被列入美国"实体名单"的国家、企业使用][Docker]

[Jenkis pipeline构建项目实践-编写podTemplate实现和k8s对接][Jenkis pipeline_-_podTemplate_k8s]

[安装kubernetes集群-灵活安装k8s各个版本高可用集群][kubernetes_-_k8s]

[Kubernetes v1.19 正式发布  
][Kubernetes v1.19 _]

[高效的Nginx负载均衡器][Nginx]

[5个维度对 Kubernetes 集群优化][5_ Kubernetes]

[什么是架构师？][Link 1]

[QPS、TPS、并发用户数、吞吐量关系][QPS_TPS]

[kubernetes面试题汇总][kubernetes]

[DevOps视频和资料免费领取][DevOps]

[kubernetes技术分享-可用于企业内部培训][kubernetes_-]

[谈谈我的IT发展之路][IT]

[kubernetes系列文章第一篇-k8s基本介绍][kubernetes_-k8s]

[kubernetes系列文章第二篇-kubectl][kubernetes_-kubectl]

[了解pod和pod的生命周期-这一篇文章就够了  
][pod_pod_-_]

[Kubernetes中部署MySQL高可用集群][Kubernetes_MySQL]

[k8s中蓝绿部署、金丝雀发布、滚动更新汇总][k8s]

[运维常见问题汇总-tomcat篇][-tomcat]

[关于linux内核参数的调优，你需要知道][linux 1]

[kubernetes挂载ceph rbd和cephfs][kubernetes_ceph rbd_cephfs]

[报警神器Alertmanager发送报警到多个渠道][Alertmanager]

[jenkins+kubernetes+harbor+gitlab构建企业级devops平台][jenkins_kubernetes_harbor_gitlab_devops]

[kubernetes网络插件-flannel篇][kubernetes_-flannel]

[kubernetes网络插件-calico篇][kubernetes_-calico]

[kubernetes认证、授权、准入控制][kubernetes 1]

[限制不同的用户操作k8s资源][k8s 1]

[面试真题&技术资料免费领取-覆盖面超全~][-]

[Prometheus监控MySQL][Prometheus_MySQL]

[Prometheus监控Nginx][Prometheus_Nginx]

[Prometheus监控Tomcat][Prometheus_Tomcat]

[linux面试题汇总][linux]

[测试通过storageclass动态生成pv][storageclass_pv]

[通过编写k8s的资源清单yaml文件部署gitlab服务][k8s_yaml_gitlab]

[helm安装和使用-通过helm部署k8s应用][helm_-_helm_k8s]

[k8s基于Ingress-nginx实现灰度发布][k8s_Ingress-nginx]

[k8s的Pod安全策略][k8s_Pod]

[Prometheus Operator-上篇-安装和使用篇][Prometheus Operator-_-]

[Prometheus Operator-下篇  
][Prometheus Operator-_]

[通过kubeconfig登陆k8s的dashboard ui界面][kubeconfig_k8s_dashboard ui]

[通过token令牌登陆k8s dashboard ui界面][token_k8s dashboard ui]

[kubernetes集群的etcd数据库详细介绍][kubernetes_etcd]

[Linux网络流量监控工具][Linux]

[kubernetes搭建EFK日志管理系统][kubernetes_EFK]

[prometheus operator监控k8s集群之外的haproxy组件][prometheus operator_k8s_haproxy]

[kubernetes ConfigMap存储卷][kubernetes ConfigMap]

[Python采集linux服务器数据在Django Web界面展示][Python_linux_Django Web]

[基于Kubernetes的GPU类型调度实现][Kubernetes_GPU]

[容器日志管理的最佳实践][Link 2]

[Kubernetes 数据库 Etcd 日常运维及技巧][Kubernetes _ Etcd]

[一文详解 LVS、Nginx 及 HAProxy工作原理][LVS_Nginx _ HAProxy]

[一篇文章全面了解运维监控知识体系][Link 3]

[Kubernetes容器云平台技术方案][Kubernetes 1]

[istio部署模型-单集群或者多集群][istio_-]

[用了3年Kubernetes，我们得到的5个教训][3_Kubernetes_5]

[两大容器管理平台，Kubernetes与OpenShift有什么区别？][Kubernetes_OpenShift]

[Etcd 分布式存储系统应用场景与工作原理][Etcd]

[![format_png][]][format_png 1]

![format_png 2][]

[Kubernetes全栈技术培训][Kubernetes]

[k8s-hpa]: /images/20221122/298b229acf6f455593194f4a130bee89.png
[k8s-hpa-ms]: /images/20221122/9f056b5af4db44d1a0841a43892eddf8.png
[k8s-hpa-prom]: /images/20221122/7cdbc4ef95f7416d8829c03a482c3ac9.png
[format_png]: /images/20221122/2268e3ec98094a4a8790798202ceb591.png
[format_png 1]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c35fde5f4a494f5326b3ff3c71afa608caa4c1bb58f596957f30d6516f9c10b50783d23e&idx=1&mid=2247484053&scene=21&sn=419ea3da079c947a9d65fecf7e22db6f#wechat_redirect
[format_png 2]: /images/20221122/c27f112627d54431887b90bd9c89aeff.png
[Kubernetes]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638550cc140c46f9bae11199c5652f145798f9542681f3ac80746136ec687e71ce1fb8cf21&idx=1&mid=2247486021&scene=21&sn=0bf92248ed9b80149a3fa891f3bb24b3#wechat_redirect
[format_png 3]: /images/20221122/3e742f02246642ef9d4cfc26e1e8996f.png
[format_png 4]: /images/20221122/dc6625024123451cbace77c5d5cb9023.png
[kubernetes_k8s]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638a31cc14032772d778f1a8fa5dc73f9aba6c99546c9ab25d9f91f706be3cff34572f0c1a&idx=1&mid=2247484708&scene=21&sn=2a7230ade0ce0bd9059e92752e03ba18#wechat_redirect
[Python_k8s_DevOps]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638455cc140d43dd091b09efc0c744b82f32d905c78ce9be71aae8c6d324d625f110fa1093&idx=1&mid=2247486272&scene=21&sn=e993bc259db899b79922863859d24bb2#wechat_redirect
[python_-_]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb63855acc140c4cf0346cc3433bc874e7cd37a5b17cf5254147a1b20b538cf70158f2234b0f&idx=1&mid=2247486031&scene=21&sn=58003addbf36f2858f7d0dfbf887bb2a#wechat_redirect
[python_-]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb6385eccc140cfa90bef6759c768c94ac2ca904c827fb2dcd969858ba46b818145e76e09c62&idx=1&mid=2247486201&scene=21&sn=0d45a2660a7480d73cf72ba67fe52f3a#wechat_redirect
[python_-_Django]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638417cc140d011a058d7cf83795bf75cfc327acbb3727c968058ff2807874a26f87843080&idx=1&mid=2247486210&scene=21&sn=078e172a34c22e5db1f57b17c21b2efe#wechat_redirect
[IT]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638c29cc14053fafcb62be244f5a279052b8d159c9c530e73495cfd8ab97205988780d4aaa&idx=1&mid=2247484220&scene=21&sn=72ae437836108fbecbeb86752aad888c#wechat_redirect
[Prometheus_Grafana_Alertmanager_-]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638b08cc14021e5e025f2f0b89d18ec5ef19f9dec4429a56f356e8b596d01095e313dab3e3&idx=1&mid=2247484445&scene=21&sn=bf8928e7aae815b8e7f9ac56a67289ce#wechat_redirect
[k8s1.18_master_-]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638c7dcc14056b9f0261e05e4d36a057c4511fb5806222f71b7965c3573fed86672b2c93b7&idx=1&mid=2247484264&scene=21&sn=4d74de0cb117c07ba7c73ea21c42bf94#wechat_redirect
[linux]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c0dede5f49c878898e04329f08d8adf554397476283aaa3f2554f71a8bf20986755fbcc9&idx=1&mid=2247483668&scene=21&sn=a7d0768b7eb034ef2e2f23e25b1b1845#wechat_redirect
[Docker]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb63841ecc140d08e72aa244da69b776d4470aac79f3d03610c5fec827ac24a2048178204bf0&idx=1&mid=2247486219&scene=21&sn=31fe60b84f327ac1cc24bdb98ccba6a6#wechat_redirect
[Jenkis pipeline_-_podTemplate_k8s]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638472cc140d64ef12e6fff56dfa0aea54e016940494de9293f6727a8a39e61c427e206c35&idx=1&mid=2247486311&scene=21&sn=eb4be92fc95162e7a4d44c08521c7631#wechat_redirect
[kubernetes_-_k8s]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb6384b8cc140dae0e6efba75100e29549cc440a010a5c7976bdd313e9c6139569b197e20a8f&idx=1&mid=2247486381&scene=21&sn=f07ae730cdc3ca2442462e0c7f2caed1#wechat_redirect
[Kubernetes v1.19 _]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb6384a2cc140db49957616305e55d7abb842f0a46aa7344b1d1801acfba73adbb0b389a8200&idx=1&mid=2247486391&scene=21&sn=692b1e9d5df3afe2ea639ae969c5e091#wechat_redirect
[Nginx]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb6383bccc140aaa8e044e9b56c8cd5ca63c1470b7d88625b8d6d8c1828f16a42f7c61e9e6c9&idx=1&mid=2247486633&scene=21&sn=317a4a096e4a17491d8b5fa1eeb5d7d2#wechat_redirect
[5_ Kubernetes]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb6383aecc140ab8050df002d52f6063e85ad2777f4ec0941c9d40ea49c50c11f23542f43dfc&idx=1&mid=2247486651&scene=21&sn=2fe1be9d4a335077757c46a36cb1aea2#wechat_redirect
[Link 1]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb6383decc140ac88ab1c9da0818ba89b627a96fb5a097a3ec10165c4b41e7f43f6ac562d063&idx=1&mid=2247486667&scene=21&sn=4996d3c4b5653a757256587597709c3d#wechat_redirect
[QPS_TPS]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb6383d7cc140ac1dc83e4076b5677ea33f6686129dc9154034a41a37f14be2372e89caf3b87&idx=1&mid=2247486658&scene=21&sn=ce78d0e98a572089326043db7e25af9b#wechat_redirect
[kubernetes]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638b9ccc14028a3220533e5e05f81b5b434cdc14f1cb4be6771a8afdb943f7f5f940704c61&idx=1&mid=2247484553&scene=21&sn=141700cdd2e13b87d5b756f8965a30c2#wechat_redirect
[DevOps]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638b81cc140297e292675f1e8ea21d3a9f05e01e488e642bcc1113d69d06d0afbd40446c0e&idx=1&mid=2247484564&scene=21&sn=3ec4b58056aec30acf82e84cfd9daa52#wechat_redirect
[kubernetes_-]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638b4acc14025c10c5b87f75ff8e95c89c97036f1f73250a64e2a829df191c95c34ed5b8b3&idx=1&mid=2247484511&scene=21&sn=7900c5a4eac3535be97315797d42d64b#wechat_redirect
[kubernetes_-k8s]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638c42cc1405547ea583833c46878b6c120f5f39fb3746f89a8d2e515d432422b3019ded64&idx=1&mid=2247484247&scene=21&sn=c8986ed491e6c48cb6faab087fe8472c#wechat_redirect
[kubernetes_-kubectl]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638cbacc1405ac390b6c0fc7535e1cc805877a408c5f4a94a7148a47c186a57ff4f0c8ea32&idx=3&mid=2247484335&scene=21&sn=7e5ae586dae2016f6c0f5fcb60dd7317#wechat_redirect
[pod_pod_-_]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638c30cc1405263cbd23e3fbe23f9ca9e83c1eb1800216a0e4e26e7c27d4234598def46bb1&idx=1&mid=2247484197&scene=21&sn=36ece95b266609bd769cf6a1fcf9d723#wechat_redirect
[Kubernetes_MySQL]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638c73cc1405657ed02b3126423259cb8563c18d268fa7ffd4a516327cd35148ef61d8d762&idx=1&mid=2247484262&scene=21&sn=de970288426cdca05cf6ca0f5df4ff4c#wechat_redirect
[k8s]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638c36cc140520e6905db5923afe163d7babb5d9eb6c5e8045a795c37b33a2a2e5541e3efd&idx=1&mid=2247484195&scene=21&sn=b841f2ea305acfa2996a667d4ff4d99e#wechat_redirect
[-tomcat]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638c21cc140537d10eeb8b4eb37e1451392123dd74513d704af00506c0576ba11da9907735&idx=1&mid=2247484212&scene=21&sn=923b044e9d2b4e5a612ac7c1307d166c#wechat_redirect
[linux 1]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638c5ecc14054862495a8f77d58d7483c34310839ea3ed93d8b44875a1685b3412d381f23a&idx=1&mid=2247484235&scene=21&sn=12e2e80187dbf217efff6c59da778f7c#wechat_redirect
[kubernetes_ceph rbd_cephfs]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638a07cc140311e6e111b9bb89e3613c495a26b6b496fe97c838787568c72dd1bb23088520&idx=1&mid=2247484690&scene=21&sn=32deb33cef9d91518cd8dea47da073c5#wechat_redirect
[Alertmanager]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638ab7cc1403a1f6b07cbe682548595fe683f1ffdf56d2b4d31382003907a41da1e03c1408&idx=1&mid=2247484834&scene=21&sn=7f407f5a27b03b4757c8892f1b5d6234#wechat_redirect
[jenkins_kubernetes_harbor_gitlab_devops]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638d2fcc14043990967471ba8a2aaf389ab483594fd81395b5afda1c23a9eff4371df6bca4&idx=1&mid=2247483962&scene=21&sn=595eac335f7fb523bbb60cbbede109ca#wechat_redirect
[kubernetes_-flannel]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c005de5f4913b3e81d32cf98dc4f237be04b0c19e02f615650538a9aa0aabdc025604918&idx=1&mid=2247483855&scene=21&sn=26587a359cca53d129ab7e8a7fa394e0#wechat_redirect
[kubernetes_-calico]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c073de5f49658fdaed40affb1a9ea3f059e7df3cc90ae24b194a8698a7f73089d3964cab&idx=1&mid=2247483833&scene=21&sn=c00c2b69db7d06eacaf6206805d2955a#wechat_redirect
[kubernetes 1]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c05fde5f494917570591168d9e71f507c2d124312c51dfdb6cfb30eb03eb48b73a3e0d9b&idx=1&mid=2247483797&scene=21&sn=bc208ee075c41b81ffabaefe509538ba#wechat_redirect
[k8s 1]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c0b4de5f49a2eef8cc91afa81095b71eb093e94dce6eddbc7059583e819b1ac51c596ebb&idx=1&mid=2247483774&scene=21&sn=dfc3babde045fe99da9fa168cc9f0595#wechat_redirect
[-]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c0bfde5f49a9990c672e4434f5ae7b74004dc1982cc44801c524b58067d9b245e5ba5b71&idx=1&mid=2247483765&scene=21&sn=b8444e82a93535425349b509ded4e4b6#wechat_redirect
[Prometheus_MySQL]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c0aade5f49bc7df29ecb2280ab9e1a8aa905b31e36856bf8dc19b05ef60c5e7be0333b88&idx=1&mid=2247483744&scene=21&sn=1035d73f8cf39caf8534766435497590#wechat_redirect
[Prometheus_Nginx]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c095de5f4983c249b1b556fa6f8f60a0bb6991d2872079a9d5c28ce2ee17b8fc407eeb2c&idx=1&mid=2247483743&scene=21&sn=3ad15cd693685479dc7bf5dd628796d2#wechat_redirect
[Prometheus_Tomcat]: http://mp.weixin.qq.com/s?__biz=MzIzOTQ3NDg1Mw%3D%3D&chksm=e928c086de5f499011eb542c6da8ead32b538c0bf06340ebe56ad439e2649e0b63efbcb4a611&idx=1&mid=2247483724&scene=21&sn=dd390f05270c6ac7a9c2b51fdc22f0ee#wechat_redirect
[storageclass_pv]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638837cc140121578456f6a5d679ae481421f2f3f1d5099bb7f2c56d00174fe61b2016061d&idx=1&mid=2247485218&scene=21&sn=ba698d444a029a2659091bfd81133697#wechat_redirect
[k8s_yaml_gitlab]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638814cc1401028c7ab321d2ac6579f5c6d2b029fadc5687874e41470bf95aa09c01c33d45&idx=1&mid=2247485185&scene=21&sn=2b5646119a008d69948c5a84d3724c95#wechat_redirect
[helm_-_helm_k8s]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638bbbcc1402ad16d4b65b3db26cbd324c85819934077caa41969f0c9d573b71b21e2c51ff&idx=1&mid=2247484590&scene=21&sn=2c23098269c5df1e793aa47198a7b1e6#wechat_redirect
[k8s_Ingress-nginx]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb6385fdcc140ceb4a5cf902d24499ce3480f91aa97e0e0c7a1fb12a466865ff2f8dedc8be0c&idx=1&mid=2247486184&scene=21&sn=cccb2bc67ca74a01be7b4f79e05f8bec#wechat_redirect
[k8s_Pod]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb6385c3cc140cd57c41dcdd267f2bc0f0f6476d3a56e1fa88b23a7ad3f82799bad72cbdde93&idx=1&mid=2247486166&scene=21&sn=a40899007709d1a1f3e7b5be79ed6e49#wechat_redirect
[Prometheus Operator-_-]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638626cc140f30c8d35e45239e6b559f7369b8c58bacd157a24e7b62313c826baec75f299f&idx=2&mid=2247485747&scene=21&sn=3da14a91d75f745b26c3aa8d10361d23#wechat_redirect
[Prometheus Operator-_]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb63865bcc140f4d8966d1c3745c8f69c6384337c43a1a8eda6e1ae488bd78f08e44c9553b19&idx=1&mid=2247485774&scene=21&sn=1b0301dadc4b737e19b1066219813db7#wechat_redirect
[kubeconfig_k8s_dashboard ui]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638679cc140f6f4e024f2e7739b995fc235b6f4e97983a583bc9c7e82da88cf0fbf8736fe7&idx=1&mid=2247485804&scene=21&sn=c157f0596f130bd97ec745b278546c50#wechat_redirect
[token_k8s dashboard ui]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638687cc140f918f04ca60ed188da26cd3ab37a79e6fdbf64531efd11ae60dac0a77788f7c&idx=2&mid=2247485842&scene=21&sn=163174142218e8dca3bb9a94ed173694#wechat_redirect
[kubernetes_etcd]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb63853bcc140c2d4c4da1b59a4865c193383392d92075cee80530866892d8a2a365e2b4e565&idx=1&mid=2247485998&scene=21&sn=9889882d1820ac005e5512eccb67c0f1#wechat_redirect
[Linux]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638538cc140c2e581b8c673a9c529359042a2262ec112f8c3476c7408f25e04856f0b37bbd&idx=2&mid=2247485997&scene=21&sn=ac694beaf60cbf18d133eb6ebbc7a345#wechat_redirect
[kubernetes_EFK]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb63851fcc140c09d17fe6e270524da6c3ec3c5f3969f6f4f61a54c54724168801d93b2cda9d&idx=1&mid=2247485962&scene=21&sn=87dfde054f2ff489a1ffd28dfe02a435#wechat_redirect
[prometheus operator_k8s_haproxy]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb6386f0cc140fe6dea923cb07c2164db1c894c87d8c85d828aa269cdf4ae2d92eeb201adb67&idx=1&mid=2247485925&scene=21&sn=5ae9bc969963775e347c3a1f279d541c#wechat_redirect
[kubernetes ConfigMap]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb63853acc140c2ca1dc3ca813b133ca7158c34df5cfe59cebf7621851ead91490eaabc74a1b&idx=1&mid=2247485999&scene=21&sn=f6b03745426149e498a8c7065b060e24#wechat_redirect
[Python_linux_Django Web]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb6383e6cc140af0af9c2a453a58136f4388d9ba5c919adac3dee007ba407a8b28828928f37e&idx=1&mid=2247486707&scene=21&sn=d5ce76030b250c465e273708ace5792f#wechat_redirect
[Kubernetes_GPU]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb6383c6cc140ad0da77716ce494f5e22c64d7702476fc2da4cfe1e020dd7f1740b720a43112&idx=1&mid=2247486675&scene=21&sn=6068f91ac5c527db28b98ebd0faf8bff#wechat_redirect
[Link 2]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638219cc140b0fbcd0337952428804a328eff8e65bbe32804ba81793ee54c7224f04ddfaab&idx=1&mid=2247486732&scene=21&sn=05d84ed332ea0750cd7e2f9fa342ec85#wechat_redirect
[Kubernetes _ Etcd]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb63820fcc140b19133c25d97c691b873374bb10351074901487a154210d4c3d57ea02ea4d2a&idx=1&mid=2247486746&scene=21&sn=8e920e44e5e7219cec752f5ee103f17d#wechat_redirect
[LVS_Nginx _ HAProxy]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb63825fcc140b49bb429e7c4bd29eabb792c1a8088ec069d2e9d7550d5d06a5a6f1b2d0720f&idx=1&mid=2247486794&scene=21&sn=d82096ee264c1d70b93aeada191840aa#wechat_redirect
[Link 3]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638266cc140b7041d5a32907ecaa52b5c5b6eedb4ab93b02ffe090c43bb7aad4fd12997fe9&idx=1&mid=2247486835&scene=21&sn=3b61830d0f038ee35d4a8aed626304fb#wechat_redirect
[Kubernetes 1]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb63829fcc140b893000b593bdf1dfa2eaddabe9bb686db523589709abb2b78ae35dbcb58136&idx=1&mid=2247486858&scene=21&sn=3f3568553159e34533134621b4feb0a3#wechat_redirect
[istio_-]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb6382c3cc140bd5b0b47db355c79499b8b2e8f30c32cc5812c733c44ba3132cc0b2441c6dd4&idx=1&mid=2247486934&scene=21&sn=f018d3b800c4999d19abb72c21fe206d#wechat_redirect
[3_Kubernetes_5]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb6382e8cc140bfeb945d0ec6523b3356301840540ef877e8ee13c1ff7e38ccbe4a3aa71bf88&idx=1&mid=2247486973&scene=21&sn=cffae13ce2c00732e38a50bb7431090e#wechat_redirect
[Kubernetes_OpenShift]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638171cc14086765f2aa627f09b1dcdb9a5aeda4f35e033322e75a0a07a9549d2682a890ac&idx=1&mid=2247487076&scene=21&sn=7e86c8f8054c87a7830d112230c8df23#wechat_redirect
[Etcd]: http://mp.weixin.qq.com/s?__biz=MzU0NjEwMTg4Mg%3D%3D&chksm=fb638166cc1408702268b035f84a23931b664766f6f9d0854c0182961d7fb2b8722b8529780e&idx=1&mid=2247487091&scene=21&sn=8b81ce5e02285eeeb0b1c2cbec67ac72#wechat_redirect