k8s日志预研—收集Kubernetes event（容器日志）

￡神魔★判官ぃ 2023-05-31 06:56 30阅读 0赞

以下内容根据[简书链接][Link 1]，并结合自己的日志收集系统整理而成

#### （一）使用收集event方案 ####

开源项目eventrouter，地址为：https://github.com/heptiolabs/eventrouter

#### （二）收集流程 ####

大概流程为：  
1、启动eventrouter容器,挂载/data/log/eventrouter目录  
2、启动filebeat容器,挂载/data/log/eventrouter目录  
3、filebeat收集/data/log/eventrouter目录下的日志  
4、filebeat数据发送到logstash进行处理  
5、logstash处理后的数据存储到Elasticsearch  
6、kibana可视化平台进行索引展示

#### （三）测试集群 ####

<table> 
 <thead> 
  <tr> 
   <th>IP</th> 
   <th>角色</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>10.0.3.239</td> 
   <td>master</td> 
  </tr> 
  <tr> 
   <td>10.0.3.247</td> 
   <td>node</td> 
  </tr> 
  <tr> 
   <td>10.0.3.248</td> 
   <td>node</td> 
  </tr> 
  <tr> 
   <td>10.0.3.104</td> 
   <td>logstash、es、kibana</td> 
  </tr> 
 </tbody> 
</table>

[ k8s笔记8-部署filebeat+ELK日志方案][k8s_8-_filebeat_ELK]，参考该博客部署服务器10.0.3.104上的ELK环境。

##### 小福利： #####

由于elasticsearch-6.3.2.rpm、kibana-6.3.2-x86\_64.rpm外网下载很慢，分享2个地址，大家可以在迅雷里面下载，节约时间：  
**kibana地址**：https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-x86\_64.rpm  
**elasticsearch地址**：https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.rpm  
logstash地址：各个版本的链接查看地址：https://www.elastic.co/downloads/past-releases?page=6\#logstash  
下载地址：https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.rpm  
安装RPM的命令：

rpm -ivh *.rpm --force --nodeps

#### （四）修改过的eventrouter配置文件，cat filebeat-eventrouter.yaml ####

apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: eventrouter 
      namespace: kube-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRole
    metadata:
      name: eventrouter 
    rules:
    - apiGroups: [""]
      resources: ["events"]
      verbs: ["get", "watch", "list"]
    ---
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRoleBinding
    metadata:
      name: eventrouter 
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: eventrouter
    subjects:
    - kind: ServiceAccount
      name: eventrouter
      namespace: kube-system
    ---
    apiVersion: v1
    data:
      config.json: |- 
        { 
          "sink": "glog"
        }
    kind: ConfigMap
    metadata:
      name: eventrouter-cm
      namespace: kube-system
    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: filebeat-config
      namespace: kube-system
    data:
      filebeat.yml: |-
        filebeat.prospectors:
        - input_type: log
          paths:
            - "/data/log/eventrouter/*"
          fields:
            app: www
            type: eventrouter-log
          fields_under_root: true
          multiline:
            pattern: '^\['
            negate: true
            match: after
        output.logstash:
          hosts: ["10.0.3.104:5044"]
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: eventrouter
      namespace: kube-system
      labels:
        app: eventrouter
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: eventrouter
      template:
        metadata:
          labels:
            app: eventrouter
            tier: control-plane-addons
        spec:
          containers:
            - name: kube-eventrouter
              image: baiyongjie/eventrouter:v0.2
              imagePullPolicy: IfNotPresent
              command:
                - "/bin/sh"
              args:
                - "-c"
                - "/eventrouter -v 3 -log_dir /data/log/eventrouter"
              volumeMounts:
              - name: eventrouter-cm
                mountPath: /etc/eventrouter
              - name: log-path
                mountPath: /data/log/eventrouter
            - name: filebeat
              image: test.com/project/filebeat:6.3.2
              command:
                - "/bin/sh"
              args:
                - "-c"
                - "filebeat -c /etc/filebeat/filebeat.yml"
              volumeMounts:
              - name: filebeat-config
                mountPath: /etc/filebeat/
              - name: log-path
                mountPath: /data/log/eventrouter
          serviceAccount: eventrouter
          volumes:
            - name: eventrouter-cm
              configMap:
                name: eventrouter-cm
            - name: filebeat-config
              configMap:
                name: filebeat-config
            - name: log-path
              emptyDir: { }

修改/etc/logstash/conf.d/logstash-to-es.conf文件，添加logstash的内容：

else if [type] == "eventrouter-log" {
               elasticsearch {
                  hosts => ["http://127.0.0.1:9200"]
                  index => "eventrouter-log-%{+YYYY.MM.dd}"
               }
            }

#### （五）在kibana的页面展示添加index Patterns，展示 ####

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA4MDE5OTQ_size_16_color_FFFFFF_t_70]  
在服务器中查看即时的event的语句为

kubectl -n kube-system describe pod grafana-0 | grep -A 10 Events:

当集群中有pod的contrainer重启或者出错时，会在该界面看到对应时间戳的日志展示，方便与获取容器日志与应用日志之间的匹配关系。

后续需要加的：  
将日志内容进行持久化存储

[Link 1]: https://www.jianshu.com/p/335572c2c236
[k8s_8-_filebeat_ELK]: https://blog.csdn.net/u010801994/article/details/88634102
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA4MDE5OTQ_size_16_color_FFFFFF_t_70]: /images/20230531/da167ef8a8024ce1a69cd0f6b3c15af9.png