gitlab集成k8s的问题

迈不过友情╰ 2022-12-13 04:54 316阅读 0赞

**1 k8s中metrics不能工作影响gitlab的推进**  
下方是k8s中的日志

status:
      conditions:
        - lastTransitionTime: '2020-07-16T04:10:58Z'
          message: >-
            failing or missing response from
            https://10.101.30.104:4443/apis/metrics.k8s.io/v1beta1: Get
            https://10.101.30.104:4443/apis/metrics.k8s.io/v1beta1: dial tcp
            10.101.30.104:4443: connect: no route to host
          reason: FailedDiscoveryCheck
          status: 'False'
          type: Available

下方是metrics-server的日志

unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:worker3.sh-cluster.sh-k8s.com: unable to get CPU for container "copyright-v1" in pod spider/copyright-v1-dcd6f785b-4ztkn on node "10.101.30.106", discarding data: missing cpu usage metric

参考[kubernetes metrics 不能工作问题][kubernetes metrics]，这里面说解决方案只适用于测试环境，不适合生产环境，这是为啥呢？  
根据[ Metrics server 常见问题][Metrics server]中说明，提示了两个错误原因

1.  使用二进制安装方式安装 Kubernetes 集群，但是未在 Master 节点上启用 kube-proxy 组件；
2.  master 节点与 metrics-server 所在的 worker 节点之间有防火墙，或者不在同一个安全组；

那么检查master节点是否启用了kube-proxy组件呢  
根据[K8S 资源指标监控-部署metrics-server][K8S _-_metrics-server]，执行命令`kubectl get pods -n kube-system`，我的服务器上确实没有看到`kube-proxy`  
![1][]  
在仔细阅读[metrics-server采集数据失败问题排查][metrics-server],执行`kubectl describe apiservice v1beta1.metrics.k8s.io`查看`apiService`的状态,我报的错跟他的一模一样，但是解决方案却不是一样的，我没有增加什么额外的配置

[root@worker1 ~]# kubectl describe apiservice v1beta1.metrics.k8s.io
    Name:         v1beta1.metrics.k8s.io
    Namespace:    
    Labels:       <none>
    Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                    {"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{},"name":"v1beta1.metrics.k8s.io"},"spec":{"...
    API Version:  apiregistration.k8s.io/v1
    Kind:         APIService
    Metadata:
      Creation Timestamp:  2020-07-16T03:33:42Z
      Resource Version:    20595406
      Self Link:           /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
      UID:                 65ebc093-4ca4-4f70-a6df-8d3babfe7a1d
    Spec:
      Group:                     metrics.k8s.io
      Group Priority Minimum:    100
      Insecure Skip TLS Verify:  true
      Service:
        Name:            metrics-server
        Namespace:       kube-system
        Port:            443
      Version:           v1beta1
      Version Priority:  100
    Status:
      Conditions:
        Last Transition Time:  2020-07-16T04:10:58Z
        Message:               failing or missing response from https://10.101.30.104:4443/apis/metrics.k8s.io/v1beta1: Get https://10.101.30.104:4443/apis/metrics.k8s.io/v1beta1: dial tcp 10.101.30.104:4443: connect: connection timed out
        Reason:                FailedDiscoveryCheck
        Status:                False
        Type:                  Available
    Events:                    <none>

查看[部署METRICS-SERVER时遇到的问题][METRICS-SERVER]  
按照[metrics-server部署后服务不可用  
][metrics-server_]，均无效果  
![1][1 1]  
参考[【k8s错误处理】——unable to retrieve the complete list of server APIs][k8s_unable to retrieve the complete list of server APIs],执行`kubectl get apiservice`,`kube-system/metrics-server`的状态确实是不可用，一气之下我执行了`kubectl delete apiservce <service-name>`,删掉了`v1beta1.metrics.k8s.io`

[root@master1 gitlab-runner]# kubectl get apiservice
    NAME                                   SERVICE                      AVAILABLE                      AGE
    v1.                                    Local                        True                           88d
    v1.admissionregistration.k8s.io        Local                        True                           88d
    v1.apiextensions.k8s.io                Local                        True                           88d
    v1.apps                                Local                        True                           88d
    v1.authentication.k8s.io               Local                        True                           88d
    v1.authorization.k8s.io                Local                        True                           88d
    v1.autoscaling                         Local                        True                           88d
    v1.batch                               Local                        True                           88d
    v1.coordination.k8s.io                 Local                        True                           88d
    v1.kuboard.cn                          Local                        True                           50m
    v1.networking.k8s.io                   Local                        True                           88d
    v1.rbac.authorization.k8s.io           Local                        True                           88d
    v1.scheduling.k8s.io                   Local                        True                           88d
    v1.storage.k8s.io                      Local                        True                           88d
    v1beta1.admissionregistration.k8s.io   Local                        True                           88d
    v1beta1.apiextensions.k8s.io           Local                        True                           88d
    v1beta1.authentication.k8s.io          Local                        True                           88d
    v1beta1.authorization.k8s.io           Local                        True                           88d
    v1beta1.batch                          Local                        True                           88d
    v1beta1.certificates.k8s.io            Local                        True                           88d
    v1beta1.coordination.k8s.io            Local                        True                           88d
    v1beta1.events.k8s.io                  Local                        True                           88d
    v1beta1.extensions                     Local                        True                           88d
    v1beta1.metrics.k8s.io                 kube-system/metrics-server   False (FailedDiscoveryCheck)   88d
    v1beta1.networking.k8s.io              Local                        True                           88d
    v1beta1.node.k8s.io                    Local                        True                           88d
    v1beta1.policy                         Local                        True                           88d
    v1beta1.rbac.authorization.k8s.io      Local                        True                           88d
    v1beta1.scheduling.k8s.io              Local                        True                           88d
    v1beta1.storage.k8s.io                 Local                        True                           88d
    v2alpha1.batch                         Local                        True                           88d
    v2beta1.autoscaling                    Local                        True                           88d
    v2beta2.autoscaling                    Local                        True                           88d

删掉之后，在执行下面的语句，就没有出现异常了。后面有时间在对metrics-server进行重新安装吧

helm install --name-template gitlab-runner -f values.yaml . --namespace gitlab-runners

Error: Could not get apiVersions from Kubernetes: 
    unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

**2 gitlab-ci.yml配置scrapy爬虫**

ERROR: Job failed (system failure): prepare environment: image pull failed: Back-off pulling image "scrapy_env:0.01". 
    Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

[kubernetes metrics]: https://blog.csdn.net/taoli1986/article/details/95776291
[Metrics server]: https://kuboard.cn/install/faq/metrics-server.html
[K8S _-_metrics-server]: https://blog.csdn.net/wangmiaoyan/article/details/102868728
[1]: /images/20221123/d5eb4a8c71e047d99424a4f9c1d15996.png
[metrics-server]: https://www.jianshu.com/p/8f84b6769063
[METRICS-SERVER]: http://blog.ljmict.com/?p=98
[metrics-server_]: https://github.com/opsnull/follow-me-install-kubernetes-cluster/issues/417
[1 1]: /images/20221123/1413d3758e974108ad7f2f75863f67ba.png
[k8s_unable to retrieve the complete list of server APIs]: https://blog.csdn.net/weixin_40449300/article/details/108348420