kubernetes 从v1.7.6升级到v1.11.2版本--kube-apiserver错误分析

墨蓝 2022-05-13 11:06 322阅读 0赞

**错误日志**

[root@master-47-35 ~]# kubectl logs -f kube-apiserver-master-47-34 -n kube-system 
    ...
    E0912 06:36:05.215312       1 controller.go:111] loading OpenAPI spec for "v1alpha1.custom-metrics.metrics.k8s.io" failed with: OpenAPI spec does not exists
    ...

查原因

**查看api-versions**

[root@master-47-35 ~]# kubectl api-versions
    admissionregistration.k8s.io/v1beta1
    apiextensions.k8s.io/v1beta1
    apiregistration.k8s.io/v1
    apiregistration.k8s.io/v1beta1
    apps/v1
    apps/v1beta1
    apps/v1beta2
    authentication.k8s.io/v1
    authentication.k8s.io/v1beta1
    authorization.k8s.io/v1
    authorization.k8s.io/v1beta1
    autoscaling/v1
    autoscaling/v2beta1
    batch/v1
    batch/v1beta1
    certificates.k8s.io/v1beta1
    custom-metrics.metrics.k8s.io/v1alpha1
    events.k8s.io/v1beta1
    extensions/v1beta1
    monitoring.coreos.com/v1
    networking.k8s.io/v1
    policy/v1beta1
    rbac.authorization.k8s.io/v1
    rbac.authorization.k8s.io/v1beta1
    scheduling.k8s.io/v1beta1
    storage.k8s.io/v1
    storage.k8s.io/v1beta1
    v1

虽然有创建这个`custom-metrics.metrics.k8s.io/v1alpha1`，但是在v1.11.1,已经不支持这个版本了[kubernetes release-1.11][]，在v1.7.6版本支持[kubernetes release-1.7][]  
对比以下两幅图，就可以看出了，在k8s v1.11.x已经  
custom-metrics.metrics.k8s.io/v1alpha1  
改为  
custom-metrics.metrics.k8s.io/v2beta1  
以及custom-metrics.metrics.k8s.io/v2beta2  
所以就会报错

![v1.7.6][]

![v1.11.2][]

**解决办法**

删除`v1alpha1.custom-metrics.metrics.k8s.io`api组

vi v1alpha1.custom-metrics.metrics.k8s.io.yaml
    
    apiVersion: apiregistration.k8s.io/v1beta1
    kind: APIService
    metadata:
      name: v1alpha1.custom-metrics.metrics.k8s.io
    spec:
      insecureSkipTLSVerify: true
      group: custom-metrics.metrics.k8s.io
      groupPriorityMinimum: 1000
      versionPriority: 15
      service:
        name: api
        namespace: custom-metrics
      version: v1alpha1

kubectl delete -f v1alpha1.custom-metrics.metrics.k8s.io.yaml
    
    
    apiservice.apiregistration.k8s.io "v1alpha1.custom-metrics.metrics.k8s.io" deleted

查看api-versions

[root@master-47-35 custom-metrics]# kubectl api-versions
    admissionregistration.k8s.io/v1beta1
    apiextensions.k8s.io/v1beta1
    apiregistration.k8s.io/v1
    apiregistration.k8s.io/v1beta1
    apps/v1
    apps/v1beta1
    apps/v1beta2
    authentication.k8s.io/v1
    authentication.k8s.io/v1beta1
    authorization.k8s.io/v1
    authorization.k8s.io/v1beta1
    autoscaling/v1
    autoscaling/v2beta1
    batch/v1
    batch/v1beta1
    certificates.k8s.io/v1beta1
    events.k8s.io/v1beta1
    extensions/v1beta1
    monitoring.coreos.com/v1
    networking.k8s.io/v1
    policy/v1beta1
    rbac.authorization.k8s.io/v1
    rbac.authorization.k8s.io/v1beta1
    scheduling.k8s.io/v1beta1
    storage.k8s.io/v1
    storage.k8s.io/v1beta1
    v1

删除成功，验证  
重启kube-apiserver

[root@master-47-35 custom-metrics]# docker ps | grep api
    b5ce2cb0371f        harbor.enncloud.cn/enncloud/hyperkube-amd64@sha256:2466e6219aba9d88dcef04c0f5578a6df5e9cc0d9f1913357585bc71436b4237   "/apiserver --enab..."   About a minute ago   Up About a minute                       k8s_kube-apiserver_kube-apiserver-master-47-35_kube-system_497141a7e067956aaaa0199ce88060ed_2
    dee06861bece        harbor.enncloud.cn/paas/pause-amd64:3.1                                                                               "/pause"                 24 hours ago         Up 24 hours                             k8s_POD_kube-apiserver-master-47-35_kube-system_497141a7e067956aaaa0199ce88060ed_2
    [root@master-47-35 custom-metrics]# docker logs -f b5ce2cb0371f
    Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version.
    Flag --insecure-port has been deprecated, This flag will be removed in a future version.
    [restful] 2018/09/12 07:29:35 log.go:33: [restful/swagger] listing is available at https://10.39.47.35:6443/swaggerapi
    [restful] 2018/09/12 07:29:35 log.go:33: [restful/swagger] https://10.39.47.35:6443/swaggerui/ is mapped to folder /swagger-ui/
    [restful] 2018/09/12 07:29:37 log.go:33: [restful/swagger] listing is available at https://10.39.47.35:6443/swaggerapi
    [restful] 2018/09/12 07:29:37 log.go:33: [restful/swagger] https://10.39.47.35:6443/swaggerui/ is mapped to folder /swagger-ui/
    E0912 07:29:41.072852       1 available_controller.go:311] v1.monitoring.coreos.com failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1.monitoring.coreos.com": StorageError: invalid object, Code: 4, Key: /registry/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 6e717648-b65d-11e8-89a3-5254e98192ae, UID in object meta: 
    E0912 07:29:41.119161       1 autoregister_controller.go:190] v1.monitoring.coreos.com failed with : apiservices.apiregistration.k8s.io "v1.monitoring.coreos.com" already exists
    E0912 07:29:44.605439       1 controller.go:111] loading OpenAPI spec for "v1alpha1.custom-metrics.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
    , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
    E0912 07:30:44.605888       1 controller.go:111] loading OpenAPI spec for "v1alpha1.custom-metrics.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
    , Header: map[X-Content-Type-Options:[nosniff] Content-Type:[text/plain; charset=utf-8]]

还有报错 但是不影响使用

[kubernetes release-1.11]: https://github.com/kubernetes/metrics/tree/release-1.11/pkg/apis/custom_metrics
[kubernetes release-1.7]: https://github.com/kubernetes/metrics/tree/release-1.7/pkg/apis/custom_metrics
[v1.7.6]: /images/20220513/b44c61de7dbd44ff974d2f137e7981ec.png
[v1.11.2]: /images/20220513/a242a5ec7d2c4b2cba428e18abae4888.png