kubernetes v1.18.6 高可用集群集群安装说明

男娘i 2023-03-05 03:18 12阅读 0赞

### 前言 ###

> 之前我有写过普通集群搭建，没有给master做高可用，于是很多给我评论如何弄ha。今天我们就开始怎么去搭建一个高可用的Kuberneters 集群吧！

### 准备机器或虚拟机 ###

我这里使用六台CentOS虚拟机，你们机器少的话可以删减两台worker，我这里主要还是讲解master的高可用。

镜像地址：[http://mirrors.aliyun.com/centos/7/isos/x86\_64/CentOS-7-x86\_64-Minimal-2003.iso][http_mirrors.aliyun.com_centos_7_isos_x86_64_CentOS-7-x86_64-Minimal-2003.iso]

<table> 
 <tbody> 
  <tr> 
   <td style="width:139px;">IP</td> 
   <td style="width:198px;">HOSTNAME</td> 
   <td style="width:214px;">CPU</td> 
   <td style="width:196px;">MEMORY</td> 
   <td style="width:196px;">DISK</td> 
  </tr> 
  <tr> 
   <td style="width:139px;">172.16.0.77</td> 
   <td style="width:198px;">master00</td> 
   <td style="width:214px;">&gt;=2</td> 
   <td style="width:196px;">&gt;=4G</td> 
   <td style="width:196px;">&gt;=8G</td> 
  </tr> 
  <tr> 
   <td style="width:139px;">172.16.0.78</td> 
   <td style="width:198px;">master01</td> 
   <td style="width:214px;">&gt;=2</td> 
   <td style="width:196px;">&gt;=4G</td> 
   <td style="width:196px;">&gt;=8G</td> 
  </tr> 
  <tr> 
   <td style="width:139px;">172.16.0.79</td> 
   <td style="width:198px;">master02</td> 
   <td style="width:214px;">&gt;=2</td> 
   <td style="width:196px;">&gt;=4G</td> 
   <td style="width:196px;">&gt;=8G</td> 
  </tr> 
  <tr> 
   <td style="width:139px;">172.16.0.80</td> 
   <td style="width:198px;">worker00</td> 
   <td style="width:214px;">&gt;=2</td> 
   <td style="width:196px;">&gt;=4G</td> 
   <td style="width:196px;">&gt;=16G</td> 
  </tr> 
  <tr> 
   <td style="width:139px;">172.16.0.81</td> 
   <td style="width:198px;">worker01</td> 
   <td style="width:214px;">&gt;=2</td> 
   <td style="width:196px;">&gt;=4G</td> 
   <td style="width:196px;">&gt;=16G</td> 
  </tr> 
  <tr> 
   <td style="width:139px;">172.16.0.82</td> 
   <td style="width:198px;">worker02</td> 
   <td style="width:214px;">&gt;=2</td> 
   <td style="width:196px;">&gt;=4G</td> 
   <td style="width:196px;">&gt;=16G</td> 
  </tr> 
 </tbody> 
</table>

### 详细安装步骤与[上一篇][Link 1]大同小异 ###

我这里不多做太多介绍了，直接上脚本。需要注意的是我这里的机器与外网隔离的，你们自己ip不一定要按照我的来，自己设置成桥接有网络的配置。

每台机器或虚拟机都执行如下脚本：

systemctl disable postfix --now && systemctl disable firewalld --now
    setenforce 0 && sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
    swapoff -a && sed -i 's/.*swap.*/#&/' /etc/fstab
    cat > /etc/sysctl.d/k8s.conf <<EOF
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    EOF
    sysctl --system
    cd /etc/yum.repos.d/
    mkdir backup
    mv ./*.repo backup
    curl -O http://mirrors.aliyun.com/repo/Centos-7.repo
    curl -O https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
    cat <<EOF > /etc/yum.repos.d/kubernetes.repo
    [kubernetes]
    name=Kubernetes
    baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
    enabled=1
    gpgcheck=0
    EOF
    yum clean all && yum makecache
    yum -y update
    yum -y install docker-ce docker-ce-cli containerd.io kubelet kubeadm kubectl
    systemctl enable docker --now
    cat > /etc/docker/daemon.json <<EOF
    {
      "exec-opts": ["native.cgroupdriver=systemd"],
      "log-driver": "json-file",
      "log-opts": {
        "max-size": "100m"
      },
      "storage-driver": "overlay2",
      "storage-opts": [
        "overlay2.override_kernel_check=true"
      ]
    }
    EOF
    mkdir -p /etc/systemd/system/docker.service.d
    systemctl daemon-reload && systemctl restart docker
    systemctl enable kubelet --now

在各master节点执行如下脚本：

yum -y install keepalived

在master00节点编辑kubeadm-config.yaml文件并保存：

apiVersion: kubeadm.k8s.io/v1beta2
    kind: ClusterConfiguration
    kubernetesVersion: v1.18.6
    imageRepository: registry.cn-shanghai.aliyuncs.com/k8sgcrio_containers
    apiServer:
      certSANs:
      - master00
      - master01
      - master02
      - vip
      - 172.16.0.77
      - 172.16.0.78
      - 172.16.0.79
      - 172.16.0.76
    controlPlaneEndpoint: "172.16.0.76:6443"
    #etcd:
        #external:
            #endpoints:
            #- http://172.16.0.160:2379
            #- http://172.16.0.161:2379
            #- http://172.16.0.162:2379
    networking:
      dnsDomain: cluster.local
      podSubnet: 10.244.0.0/16
      serviceSubnet: 10.96.0.0/12

其中 apiServer certSANS 填写master host名和ip以及虚拟ip(VIP)  
controlPlaneEndpoint: 可以填VIP，也可以填master其中一个ip。其实个人推荐在准备两台机器搭建nginx一主一备，直接负载均衡转发到三台master即可。  
etcd默认使用本地，也可以使用外部etcd集群,搭建方式可以参考我我的[Etcd集群搭建][Etcd]：。我这里使用堆叠etcd方式。

在master00节点执行如下脚本：

mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.bak
    cat << EOF > /etc/keepalived/keepalived.conf
    ! Configuration File for keepalived
    
    global_defs {
       router_id master00
    }
    
    vrrp_instance VI_1 {
        state MASTER
        interface eth0
        virtual_router_id 50
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            172.16.0.76
        }
    }
    
    EOF
    systemctl enable keepalived --now
    kubeadm init --config=kubeadm-config.yaml
    输出结果(一定要记得保存哦，后面会用到！！！)：
    此处忽略前半部分
    Your Kubernetes control-plane has initialized successfully!
    
    To start using your cluster, you need to run the following as a regular user:
    
      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config
    
    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/
    
    You can now join any number of control-plane nodes by copying certificate authorities
    and service account keys on each node and then running the following as root:
    
      kubeadm join 172.16.0.76:6443 --token ymrykq.ra79tq8u29ovhjky \
        --discovery-token-ca-cert-hash sha256:a0f42468b84d3826e92de594416652a5d9b2f757923b1a8ac2a0d7accc80ca75 \
        --control-plane 
    
    Then you can join any number of worker nodes by running the following on each as root:
    
    kubeadm join 172.16.0.76:6443 --token ymrykq.ra79tq8u29ovhjky \
        --discovery-token-ca-cert-hash sha256:a0f42468b84d3826e92de594416652a5d9b2f757923b1a8ac2a0d7accc80ca75

上面其实输出结果都告诉你了，我这里再贴一遍：

mkdir -p $HOME/.kube
    cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    chown $(id -u):$(id -g) $HOME/.kube/config

此时我们使用客户端获取节点信息：

[root@master00 ~]# kubectl get nodes
    NAME       STATUS     ROLES    AGE     VERSION
    master00   NotReady   master   5m24s   v1.18.6

上面我们看到状态为未就绪，原因是还有一步网络插件没部署，现在我们部署网络插件：

这里的kube-flannel.yml文件请看我上篇文章。

[root@master00 ~]# kubectl apply -f kube-flannel.yml
    podsecuritypolicy.policy/psp.flannel.unprivileged created
    clusterrole.rbac.authorization.k8s.io/flannel created
    clusterrolebinding.rbac.authorization.k8s.io/flannel created
    serviceaccount/flannel created
    configmap/kube-flannel-cfg created
    daemonset.apps/kube-flannel-ds-amd64 created
    daemonset.apps/kube-flannel-ds-arm64 created
    daemonset.apps/kube-flannel-ds-arm created
    daemonset.apps/kube-flannel-ds-ppc64le created
    daemonset.apps/kube-flannel-ds-s390x created

免密登录设置在master00上操作：

ssh-keygen
    ssh-copy-id master01
    ssh-copy-id master02

复制证书相关文件到其他master节点：

ssh master01 'mkdir -p /etc/kubernetes/pki/etcd/'
    ssh master02 'mkdir -p /etc/kubernetes/pki/etcd/'
    cd /etc/kubernetes/pki/
    scp -r ca.* sa.* front-proxy-ca.* master01:$PWD
    scp -r ca.* sa.* front-proxy-ca.* master02:$PWD
    cd /etc/kubernetes/pki/etcd/
    scp -r ca.* master01:$PWD
    scp -r ca.* master02:$PWD

在master01节点执行如下脚本：

mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.bak
    cat << EOF > /etc/keepalived/keepalived.conf
    ! Configuration File for keepalived
    
    global_defs {
       router_id master01
    }
    
    vrrp_instance VI_1 {
        state BACKUP
        interface eth0
        virtual_router_id 50
        priority 90
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            172.16.0.76
        }
    }
    EOF
    systemctl enable keepalived --now
    kubeadm join 172.16.0.76:6443 --token ymrykq.ra79tq8u29ovhjky \
        --discovery-token-ca-cert-hash sha256:a0f42468b84d3826e92de594416652a5d9b2f757923b1a8ac2a0d7accc80ca75 \
        --control-plane
    输出结果：
    忽略前半部分
    This node has joined the cluster and a new control plane instance was created:
    
    * Certificate signing request was sent to apiserver and approval was received.
    * The Kubelet was informed of the new secure connection details.
    * Control plane (master) label and taint were applied to the new node.
    * The Kubernetes control plane instances scaled up.
    * A new etcd member was added to the local/stacked etcd cluster.
    
    To start administering your cluster from this node, you need to run the following as a regular user:
    
    	mkdir -p $HOME/.kube
    	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    	sudo chown $(id -u):$(id -g) $HOME/.kube/config
    
    Run 'kubectl get nodes' to see this node join the cluster.

在master02节点执行如下脚本：

mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.bak
    cat << EOF > /etc/keepalived/keepalived.conf
    ! Configuration File for keepalived
    
    global_defs {
       router_id master02
    }
    
    vrrp_instance VI_1 {
        state BACKUP
        interface eth0
        virtual_router_id 50
        priority 80
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            172.16.0.76
        }
    }
    EOF
    systemctl enable keepalived --now
    kubeadm join 172.16.0.76:6443 --token ymrykq.ra79tq8u29ovhjky \
        --discovery-token-ca-cert-hash sha256:a0f42468b84d3826e92de594416652a5d9b2f757923b1a8ac2a0d7accc80ca75 \
        --control-plane
    输出结果：
    忽略前半部分
    This node has joined the cluster and a new control plane instance was created:
    
    * Certificate signing request was sent to apiserver and approval was received.
    * The Kubelet was informed of the new secure connection details.
    * Control plane (master) label and taint were applied to the new node.
    * The Kubernetes control plane instances scaled up.
    * A new etcd member was added to the local/stacked etcd cluster.
    
    To start administering your cluster from this node, you need to run the following as a regular user:
    
    	mkdir -p $HOME/.kube
    	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    	sudo chown $(id -u):$(id -g) $HOME/.kube/config
    
    Run 'kubectl get nodes' to see this node join the cluster.

在master00上检查集群情况：

[root@master00 ~]# kubectl get nodes
    NAME      STATUS   ROLES    AGE     VERSION
    master00  Ready    master   37m     v1.18.6
    master01  Ready    master   10m     v1.18.6
    master02  Ready    master   9m53s   v1.18.6

大家可以看到master都成功加入集群了，这里是三个master，当master00挂掉的时候，keepalived的lvs技术会根据优先级虚拟vip。也就是我们前面配置到的172.16.0.76！当master00恢复的时候lvs会把虚拟vip交给master00。后面worker加入就更加简单了，直接复制上面加入命令。

在各个worker节点执行如下命令：

[root@worker00 ~]# kubeadm join 172.16.0.76:6443 --token ymrykq.ra79tq8u29ovhjky \
    >     --discovery-token-ca-cert-hash sha256:a0f42468b84d3826e92de594416652a5d9b2f757923b1a8ac2a0d7accc80ca75 
    W0804 16:19:40.260197    1645 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
    [preflight] Running pre-flight checks
    	[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
    [preflight] Reading configuration from the cluster...
    [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
    [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
    [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
    [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
    [kubelet-start] Starting the kubelet
    [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
    
    This node has joined the cluster:
    * Certificate signing request was sent to apiserver and a response was received.
    * The Kubelet was informed of the new secure connection details.
    
    Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

在返回到master00执行如下命令：

[root@master00 ~]# kubectl get nodes
    NAME      STATUS   ROLES    AGE   VERSION
    master00  Ready    master   77m   v1.18.6
    master01  Ready    master   50m   v1.18.6
    master02  Ready    master   50m   v1.18.6
    worker00  Ready    <none>   42s   v1.18.6
    worker01  Ready    <none>   19s   v1.18.6
    worker02  Ready    <none>   10s   v1.18.6

好了，高可用集群搭建完毕，有什么问题继续下面评论。

[http_mirrors.aliyun.com_centos_7_isos_x86_64_CentOS-7-x86_64-Minimal-2003.iso]: http://mirrors.aliyun.com/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-2003.iso
[Link 1]: https://blog.csdn.net/ZouChengli/article/details/107241595
[Etcd]: https://blog.csdn.net/ZouChengli/article/details/108884039