kubernetes v1.9.0中 RBAC DENY 解决办法

ゝ一世哀愁。 2022-06-01 03:25 129阅读 0赞

# 1 问题 #

在node1上，安装kubelet后，查看日志：

root@node1:~# tailf /var/log/syslog
    Jan 15 15:01:07 node1 kubelet[17646]: I0115 15:01:07.080370   17646 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach
    Jan 15 15:01:07 node1 kubelet[17646]: I0115 15:01:07.084014   17646 kubelet_node_status.go:431] Recording NodeHasSufficientDisk event message for node 192.168.122.3
    Jan 15 15:01:07 node1 kubelet[17646]: I0115 15:01:07.084061   17646 kubelet_node_status.go:431] Recording NodeHasSufficientMemory event message for node 192.168.122.3
    Jan 15 15:01:07 node1 kubelet[17646]: I0115 15:01:07.084084   17646 kubelet_node_status.go:431] Recording NodeHasNoDiskPressure event message for node 192.168.122.3
    Jan 15 15:01:07 node1 kubelet[17646]: I0115 15:01:07.084106   17646 kubelet_node_status.go:82] Attempting to register node 192.168.122.3
    Jan 15 15:01:07 node1 kubelet[17646]: E0115 15:01:07.086790   17646 kubelet_node_status.go:106] Unable to register node "192.168.122.3" with API server: nodes is forbidden: User "system:node:192.168.122.3" cannot create nodes at the cluster scope
    Jan 15 15:01:07 node1 kubelet[17646]: E0115 15:01:07.309503   17646 eviction_manager.go:238] eviction manager: unexpected err: failed to get node info: node "192.168.122.3" not found

在master结点上查看日志，apiserver日志报错，大量的组件都是RBAC DENY状态。：

root@master:~# tailf /var/log/syslog
    Jan 18 14:39:39 master kube-apiserver[2638]: I0118 14:39:39.885925    2638 rbac.go:116] RBAC DENY: user "system:node:192.168.122.3" groups ["system:nodes" "system:authenticated"] cannot "list" resource "services" cluster-wide
    Jan 18 14:39:39 master kube-apiserver[2638]: I0118 14:39:39.886076    2638 wrap.go:42] GET /api/v1/services?limit=500&resourceVersion=0: (490.18µs) 403 [[kubelet/v1.9.0 (linux/amd64) kubernetes/925c127] 192.168.122.3:33354]
    Jan 18 14:39:40 master kube-apiserver[2638]: I0118 14:39:40.097660    2638 rbac.go:116] RBAC DENY: user "system:node:192.168.122.3" groups ["system:nodes" "system:authenticated"] cannot "list" resource "pods" cluster-wide
    Jan 18 14:39:40 master kube-apiserver[2638]: I0118 14:39:40.098047    2638 wrap.go:42] GET /api/v1/pods?fieldSelector=spec.nodeName%3D192.168.122.3&limit=500&resourceVersion=0: (930.577µs) 403 [[kubelet/v1.9.0 (linux/amd64) kubernetes/925c127] 192.168.122.3:33354]
    Jan 18 14:39:40 master kube-apiserver[2638]: I0118 14:39:40.359008    2638 wrap.go:42] GET /api/v1/namespaces/kube-system/endpoints/kube-scheduler: (16.594085ms) 200 [[kube-scheduler/v1.9.0 (linux/amd64) kubernetes/925c127/leader-election] 192.168.122.2:57472]
    Jan 18 14:39:40 master kube-apiserver[2638]: I0118 14:39:40.367652    2638 wrap.go:42] PUT /api/v1/namespaces/kube-system/endpoints/kube-scheduler: (7.997847ms) 200 [[kube-scheduler/v1.9.0 (linux/amd64) kubernetes/925c127/leader-election] 192.168.122.2:57472]
    Jan 18 14:39:40 master kube-apiserver[2638]: I0118 14:39:40.749509    2638 rbac.go:116] RBAC DENY: user "system:node:192.168.122.3" groups ["system:nodes" "system:authenticated"] cannot "list" resource "nodes" cluster-wide

# 2 原因： #

按照k8s官方文档(https://kubernetes.io/docs/admin/authorization/rbac/\#service-account-permissions)，存在如下的clusterrolebing。

![20180118170858191][]

1.8版本之前.开启rbac后,apiserver默认绑定system:nodes组到system:node的clusterrole。v1.8之后,此绑定默认不存在,需要手工绑定,否则kubelet启动后会报认证错误，使用kubectl get nodes查看无法成为Ready状态。

## 默认角色与默认角色绑定 ##

API Server 会创建一组默认的  ClusterRole 和  ClusterRoleBinding 对象。 这些默认对象中有许多包含  system: 前缀，表明这些资源由 Kubernetes 基础组件”拥有”。 对这些资源的修改可能导致非功能性集群（ non-functional cluster ） 。一个例子是  system:node ClusterRole 对象。这个角色定义了 kubelets 的权限。如果这个角色被修改，可能会导致 kubelets 无法正常工作。  
所有默认的ClusterRole 和 ClusterRoleBinding 对象都会被标记为 kubernetes.io/bootstrapping=rbac-defaults 。

使用命令kubectl get clusterrolebinding和kubectl get clusterrole可以查看系统中的角色与角色绑定

使用命令kubectl get clusterrolebindings system:node -o yaml或kubectl describe clusterrolebindings system:node查看system:node角色绑定的详细信息：

root@master:~# kubectl describe clusterrolebindings system:node
    Name:         system:node
    Labels:       kubernetes.io/bootstrapping=rbac-defaults
    Annotations:  rbac.authorization.kubernetes.io/autoupdate=true
    Role:
      Kind:  ClusterRole
      Name:  system:node
    Subjects:
      Kind  Name  Namespace
      ----  ----  ---------

system:node角色默认绑定为空。

## 创建角色绑定 ##

在整个集群中授予 ClusterRole ，包括所有命名空间。

从错误日志中可看出信息：user "system:node:192.168.122.3" groups \["system:nodes" "system:authenticated"\]或者从kubelet.kubeconfig查看使用的用户。

在整个集群范围内将 system:node ClusterRole 授予用户”system:node:192.168.122.3”或组”system:nodes”：

root@master:~# kubectl create clusterrolebinding kubelet-node-clusterbinding --clusterrole=system:node --user=system:node:192.168.122.3
    clusterrolebinding "kubelet-node-clusterbinding" created
    root@master:~# kubectl describe clusterrolebindings kubelet-node-clusterbinding
    Name:         kubelet-node-clusterbinding
    Labels:       <none>
    Annotations:  <none>
    Role:
      Kind:  ClusterRole
      Name:  system:node
    Subjects:
      Kind  Name                       Namespace
      ----  ----                       ---------
      User  system:node:192.168.122.3  
    root@master:~#
    root@master:~# kubectl delete clusterrolebindings kubelet-node-clusterbinding
    clusterrolebinding "kubelet-node-clusterbinding" deleted
    root@master:~#
    root@master:~# kubectl create clusterrolebinding kubelet-node-clusterbinding --clusterrole=system:node --group=system:nodes
    clusterrolebinding "kubelet-node-clusterbinding" created
    root@master:~#
    root@master:~# kubectl describe clusterrolebindings kubelet-node-clusterbinding
    Name:         kubelet-node-clusterbinding
    Labels:       <none>
    Annotations:  <none>
    Role:
      Kind:  ClusterRole
      Name:  system:node
    Subjects:
      Kind   Name          Namespace
      ----   ----          ---------
      Group  system:nodes

此时节点状态变更为Ready

root@master:~# kubectl get nodes
    NAME            STATUS    ROLES     AGE       VERSION
    192.168.122.3   Ready     <none>    1h        v1.9.0

查看日志都已经正常

[20180118170858191]: /images/20220601/600dff49831b4696ad1749deff7f3952.png