Prometheus + Grafana on k8s部署

谁践踏了优雅 2022-10-19 05:33 252阅读 0赞

一键部署链接 [点我跳转][Link 1]  
使用k8s方式部署Prometheus + grafana 监控pod 和 node 节点信息

环境基础：  
k8s集群  
创建命名空间方便管理

kubectl create namespace monitoring

## 3个组件 ##

### 1.数据源pod（任选其一） ###

node节点基础数据，cpu，io，内存，磁盘，网络等  
偏向使用DaemonSet部署的node-exporter

#### \- **node-exporter.yaml** ####

一键部署默认是这种pod

apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: node-exporter
      namespace: monitoring
      labels:
        k8s-app: node-exporter
    spec:
      selector:
        matchLabels:
          k8s-app: node-exporter
      template:
        metadata:
          labels:
            k8s-app: node-exporter
        spec:
          containers:
          - image: prom/node-exporter:v1.0.1
            name: node-exporter
            ports:
            - containerPort: 9100
              protocol: TCP
              name: http
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        k8s-app: node-exporter
      name: node-exporter
      namespace: monitoring
    spec:
      ports:
     - name: http
        port: 9100
        nodePort: 31672
        protocol: TCP
      type: NodePort
      selector:
        k8s-app: node-exporter

更为详细的k8s内部信息，pod，svc，namespace，等集群信息  
推荐deployment部署kube-state-metrics方式  
项目地址 [点我跳转][Link 2]

**本次实验使用这种模式**

#### \- **kube-state-metrics.yaml** ####

包含ClusterRoleBinding-ClusterRole-sa-svc-dep 一系列资源创建作用是收集集群数据，所以不用在同一个namespace。

apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      labels:
        app.kubernetes.io/name: kube-state-metrics
        app.kubernetes.io/version: v1.9.7
      name: kube-state-metrics
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: kube-state-metrics
    subjects:
    - kind: ServiceAccount
      name: kube-state-metrics
      namespace: ops-monit
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      labels:
        app.kubernetes.io/name: kube-state-metrics
        app.kubernetes.io/version: v1.9.7
      name: kube-state-metrics
    rules:
    - apiGroups:
      - ""
      resources:
      - configmaps
      - secrets
      - nodes
      - pods
      - services
      - resourcequotas
      - replicationcontrollers
      - limitranges
      - persistentvolumeclaims
      - persistentvolumes
      - namespaces
      - endpoints
      verbs:
      - list
      - watch
    - apiGroups:
      - extensions
      resources:
      - daemonsets
      - deployments
      - replicasets
      - ingresses
      verbs:
      - list
      - watch
    - apiGroups:
      - apps
      resources:
      - statefulsets
      - daemonsets
      - deployments
      - replicasets
      verbs:
      - list
      - watch
    - apiGroups:
      - batch
      resources:
      - cronjobs
      - jobs
      verbs:
      - list
      - watch
    - apiGroups:
      - autoscaling
      resources:
      - horizontalpodautoscalers
      verbs:
      - list
      - watch
    - apiGroups:
      - authentication.k8s.io
      resources:
      - tokenreviews
      verbs:
      - create
    - apiGroups:
      - authorization.k8s.io
      resources:
      - subjectaccessreviews
      verbs:
      - create
    - apiGroups:
      - policy
      resources:
      - poddisruptionbudgets
      verbs:
      - list
      - watch
    - apiGroups:
      - certificates.k8s.io
      resources:
      - certificatesigningrequests
      verbs:
      - list
      - watch
    - apiGroups:
      - storage.k8s.io
      resources:
      - storageclasses
      - volumeattachments
      verbs:
      - list
      - watch
    - apiGroups:
      - admissionregistration.k8s.io
      resources:
      - mutatingwebhookconfigurations
      - validatingwebhookconfigurations
      verbs:
      - list
      - watch
    - apiGroups:
      - networking.k8s.io
      resources:
      - networkpolicies
      verbs:
      - list
      - watch
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      labels:
        app.kubernetes.io/name: kube-state-metrics
        app.kubernetes.io/version: v1.9.7
      name: kube-state-metrics
      namespace: ops-monit
    ---
    apiVersion: v1
    kind: Service
    metadata:
    # annotations:
    # prometheus.io/scrape: 'true'
      labels:
        app.kubernetes.io/name: kube-state-metrics
        app.kubernetes.io/version: v1.9.7
      name: kube-state-metrics
      namespace: ops-monit
    spec:
      clusterIP: None
      ports:
      - name: http-metrics
        port: 8080
        targetPort: http-metrics
      - name: telemetry
        port: 8081
        targetPort: telemetry
      selector:
        app.kubernetes.io/name: kube-state-metrics
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app.kubernetes.io/name: kube-state-metrics
        app.kubernetes.io/version: v1.9.7
      name: kube-state-metrics
      namespace: ops-monit
    spec:
      replicas: 1
      selector:
        matchLabels:
          app.kubernetes.io/name: kube-state-metrics
      template:
        metadata:
          labels:
            app.kubernetes.io/name: kube-state-metrics
            app.kubernetes.io/version: v1.9.7
        spec:
          containers:
          - image: quay.mirrors.ustc.edu.cn/coreos/kube-state-metrics:v1.9.7
            livenessProbe:
              httpGet:
                path: /healthz
                port: 8080
              initialDelaySeconds: 5
              timeoutSeconds: 5
            name: kube-state-metrics
            ports:
            - containerPort: 8080
              name: http-metrics
            - containerPort: 8081
              name: telemetry
            readinessProbe:
              httpGet:
                path: /
                port: 8081
              initialDelaySeconds: 5
              timeoutSeconds: 5
          nodeSelector:
            beta.kubernetes.io/os: linux
          serviceAccountName: kube-state-metrics

### 2.Prometheus相关 ###

*  rbac-setup.yaml

apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: prometheus
    rules:
    - apiGroups: [""]
      resources:
      - nodes
      - nodes/proxy
      - services
      - endpoints
      - pods
      verbs: ["get", "list", "watch"]
    - apiGroups:
      - extensions
      resources:
      - ingresses
      verbs: ["get", "list", "watch"]
    - nonResourceURLs: ["/metrics"]
      verbs: ["get"]
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: prometheus
      namespace: monitoring
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: prometheus
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: prometheus
    subjects:
    - kind: ServiceAccount
      name: prometheus
      namespace: monitoring

*  configmap.yaml  
    用与重新定义收集到的数据源提供给grafan展现

apiVersion: v1
    kind: ConfigMap
    metadata:
      name: prometheus-config
      namespace: monitoring
    data:
      prometheus.yml: |
        global:
          scrape_interval:     15s
          evaluation_interval: 15s
        scrape_configs:
        - job_name: 'k8s-cadvisor'
          metrics_path: /metrics/cadvisor
          kubernetes_sd_configs:
          - role: node
          relabel_configs:
          - source_labels: [__address__]
            regex: '(.*):10250'
            replacement: '${1}:10255'
            target_label: __address__
            action: replace
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
    
          metric_relabel_configs:
          - source_labels: [instance]
            separator: ;
            regex: (.+)
            target_label: node
            replacement: $1
            action: replace
    
          - source_labels: [pod_name]
            separator: ;
            regex: (.+)
            target_label: pod
            replacement: $1
            action: replace
          - source_labels: [container_name]
            separator: ;
            regex: (.+)
            target_label: container
            replacement: $1
            action: replace
    
        - job_name: kube-state-metrics
          kubernetes_sd_configs:
          - role: endpoints
            namespaces:
              names:
              - ops-monit
          relabel_configs:
          - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
            regex: kube-state-metrics
            replacement: $1
            action: keep
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: k8s_namespace
          - source_labels: [__meta_kubernetes_service_name]
            action: replace
            target_label: k8s_sname

*  prometheus-deploy.yaml

apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        name: prometheus-deployment
      name: prometheus
      namespace: monitoring
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: prometheus
      template:
        metadata:
          labels:
            app: prometheus
        spec:
          containers:
          - image: prom/prometheus:v2.7.1
            name: prometheus
            command:
            - "/bin/prometheus"
            args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus"
            - "--storage.tsdb.retention=24h"
            ports:
            - containerPort: 9090
              protocol: TCP
            volumeMounts:
            - mountPath: "/prometheus"
              name: data
            - mountPath: "/etc/prometheus"
              name: config-volume
            resources:
              requests:
                cpu: 100m
                memory: 100Mi
              limits:
                cpu: 500m
                memory: 2500Mi
          serviceAccountName: prometheus    
          volumes:
          - name: data
            emptyDir: { }
          - name: config-volume
            configMap:
              name: prometheus-config
    ---
    kind: Service
    apiVersion: v1
    metadata:
      labels:
        app: prometheus
      name: prometheus
      namespace: monitoring
    spec:
      type: NodePort
      ports:
      - port: 9090
        targetPort: 9090
        nodePort: 30003
      selector:
        app: prometheus

### 3.grafana展示相关 ###

*  grafana.yaml

apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: grafana
      namespace: monitoring
    spec:
      selector:
        matchLabels:
          app: grafana
      replicas: 1
      template:
        metadata:
          labels:
            app: grafana
        spec:
          containers:
          - name: grafana
            image: grafana/grafana:6.6.0
            imagePullPolicy: IfNotPresent
            ports:
            - containerPort: 3000
              protocol: TCP
            env:
            - name: GF_AUTH_ANONYMOUS_ENABLED
              value: "true"
            - name: GF_AUTH_ANONYMOUS_ORG_ROLE
              value: "Viewer"
            - name: GF_SECURITY_ALLOW_EMBEDDING
              value: "true"
          volumes:
          - name: grafana-storage
            emptyDir: { }
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: grafana
      namespace: monitoring
    spec:
      type: NodePort
      ports:
      - port: 80
        targetPort: 3000
        nodePort: 30006
      selector:
        app: grafana

执行yaml文件

kubectl create -f xxx.yaml

查看相关资源状态

kubectl get all -n ops-monit ##查看kube-state-metrics服务

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70]

kubectl get all -n monitoring ##查看监控相关

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70 1]  
相关地址：  
http://:30003 \#\#Prometheus地址第一步查看是否有数据源  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70 2]  
http://192.168.17.130:30006/ \#\#grafana地址 默认密码admin/addmin  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70 3]

### 开始配置grafana ###

添加数据源  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70 4]  
添加上面的ip:30003端口  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70 5]  
导入grafan模板  
地址 [点我跳转][Link 3]  
13105 官网 Dashboard-id  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70 6]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70 7]  
最后效果展示  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70 8]

[Link 1]: http://docs.dubhe.ai/docs/setup/monitor-pod-indicator-information/
[Link 2]: https://github.com/starsliao/Prometheus/tree/master/kubernetes
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70]: /images/20221005/ed3a00229af945c78b9da1cf5704db9a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70 1]: /images/20221005/4387ef0bfe284df88756a121781123db.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70 2]: /images/20221005/d3a575fd0191477ea9152e09fac688e0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70 3]: /images/20221005/42da3a13e7d243a587c6b97ffb22a478.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70 4]: /images/20221005/e2ed24fa71d949d79d4f6e62e126eb73.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70 5]: /images/20221005/eb060198b7c74f5aaf9bde2f4051d17c.png
[Link 3]: https://grafana.com/grafana/dashboards/13105
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70 6]: /images/20221005/13edf6587008465d96630f7abb8bea5a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70 7]: /images/20221005/032c336b54cb4a7dac187d4b5b86beff.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h0OTk5OWk_size_16_color_FFFFFF_t_70 8]: /images/20221005/188df7014e9441aebbd65a72aa4befa7.png