Kubernetes高可用集群二进制部署（一）主机准备和负载均衡器安装

红太狼 2023-10-14 13:12 1阅读 0赞

[Kubernetes概述][Kubernetes]  
[使用kubeadm快速部署一个k8s集群][kubeadm_k8s]  
[Kubernetes高可用集群二进制部署（一）主机准备和负载均衡器安装][Kubernetes 1]  
[Kubernetes高可用集群二进制部署（二）ETCD集群部署][Kubernetes_ETCD]  
[Kubernetes高可用集群二进制部署（三）部署api-server][Kubernetes_api-server]  
[Kubernetes高可用集群二进制部署（四）部署kubectl和kube-controller-manager、kube-scheduler][Kubernetes_kubectl_kube-controller-manager_kube-scheduler]  
[Kubernetes高可用集群二进制部署（五）kubelet、kube-proxy、Calico、CoreDNS][Kubernetes_kubelet_kube-proxy_Calico_CoreDNS]  
[Kubernetes高可用集群二进制部署（六）Kubernetes集群节点添加][Kubernetes_Kubernetes]

Kubernetes（简称为：k8s）是Google在2014年6月开源的一个容器集群管理系统，使用Go语言开发，用于管理云平台中多个主机上的容器化的应用，Kubernetes的目标是让部署容器化的应用简单并且高效,Kubernetes提供了资源调度、部署管理、服务发现、扩容缩容、监控，维护等一整套功能，努力成为跨主机集群的自动部署、扩展以及运行应用程序容器的平台。 它支持一系列容器工具, 包括Docker、Containerd等。

## 一、集群环境准备 ##

### 1.1 主机规划 ###

<table> 
 <thead> 
  <tr> 
   <th>主机IP地址</th> 
   <th>主机名</th> 
   <th>主机配置</th> 
   <th>主机角色</th> 
   <th>软件列表</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>192.168.10.103</td> 
   <td>k8s-master1</td> 
   <td>2C4G</td> 
   <td>master + worker</td> 
   <td>kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker-ce</td> 
  </tr> 
  <tr> 
   <td>192.168.10.104</td> 
   <td>k8s-master2</td> 
   <td>2C4G</td> 
   <td>master + worker</td> 
   <td>kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker-ce</td> 
  </tr> 
  <tr> 
   <td>192.168.10.105</td> 
   <td>k8s-master3</td> 
   <td>2C4G</td> 
   <td>master + worker</td> 
   <td>kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker-ce</td> 
  </tr> 
  <tr> 
   <td>192.168.10.106</td> 
   <td>k8s-worker1</td> 
   <td>2C4G</td> 
   <td>worker1</td> 
   <td>kubelet、kube-proxy、docker-ce</td> 
  </tr> 
  <tr> 
   <td>192.168.10.107</td> 
   <td>k8s-worker2</td> 
   <td>2C4G</td> 
   <td>worker2(备用工作节点，集群搭建完成后作为新节点加入)</td> 
   <td>kubelet、kube-proxy、docker-ce</td> 
  </tr> 
  <tr> 
   <td>192.168.10.101</td> 
   <td>ha1</td> 
   <td>1C2G</td> 
   <td>LB</td> 
   <td>haproxy、keepalived</td> 
  </tr> 
  <tr> 
   <td>192.168.10.102</td> 
   <td>ha2</td> 
   <td>1C2G</td> 
   <td>LB</td> 
   <td>haproxy、keepalived</td> 
  </tr> 
  <tr> 
   <td>192.168.10.100</td> 
   <td>/</td> 
   <td>/</td> 
   <td>VIP(虚拟IP)</td> 
   <td></td> 
  </tr> 
 </tbody> 
</table>

### 1.2 软件版本 ###

<table> 
 <thead> 
  <tr> 
   <th>软件名称</th> 
   <th>版本</th> 
   <th>备注</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>CentOS7</td> 
   <td>kernel版本：5.16</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>kubernetes</td> 
   <td>v1.21.10</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>etcd</td> 
   <td>v3.5.2</td> 
   <td>最新版本</td> 
  </tr> 
  <tr> 
   <td>calico</td> 
   <td>v3.19.4</td> 
   <td>网络插件</td> 
  </tr> 
  <tr> 
   <td>coredns</td> 
   <td>v1.8.4</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>docker-ce</td> 
   <td>20.10.13</td> 
   <td>YUM源默认</td> 
  </tr> 
  <tr> 
   <td>haproxy</td> 
   <td>5.18</td> 
   <td>YUM源默认</td> 
  </tr> 
  <tr> 
   <td>keepalived</td> 
   <td>3.5</td> 
   <td>YUM源默认</td> 
  </tr> 
 </tbody> 
</table>

### 1.3 网络分配 ###

<table> 
 <thead> 
  <tr> 
   <th>网络名称</th> 
   <th>网段</th> 
   <th>备注</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>Node网络</td> 
   <td>192.168.10.101/107</td> 
   <td>集群节点网络</td> 
  </tr> 
  <tr> 
   <td>Service网络</td> 
   <td>10.96.0.0/16</td> 
   <td>实现服务发现时所使用的网络</td> 
  </tr> 
  <tr> 
   <td>Pod网络</td> 
   <td>10.244.0.0/16</td> 
   <td></td> 
  </tr> 
 </tbody> 
</table>

## 二、集群部署 ##

### 2.1主机准备 ###

#### 2.1.1 主机名设置 ####

hostnamectl set-hostname xxx

关于主机名参见1.1小节主机规划表

#### 2.1.2 主机与IP地址解析 ####

cat >> /etc/hosts << EOF
    192.168.10.101 ha1
    192.168.10.102 ha2
    192.168.10.103 k8s-master1
    192.168.10.104 k8s-master2
    192.168.10.105 k8s-master3
    192.168.10.106 k8s-worker1
    EOF

#### 2.1.3 主机安全设置 ####

##### 2.1.3.1 关闭防火墙 #####

systemctl stop firewalld
    systemctl disable firewalld
    firewall-cmd --state

##### 2.1.3.2 关闭selinux #####

setenforce 0
    sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
    sestatus

#### 2.1.4 交换分区设置 ####

swapoff -a
    sed -ri 's/.*swap.*/#&/' /etc/fstab
    echo "vm.swappiness=0" >> /etc/sysctl.conf
    sysctl -p

#### 2.1.5 主机系统时间同步 ####

安装软件
    yum -y install ntpdate
    
    制定时间同步计划任务
    crontab -e
    0 */1 * * * ntpdate time1.aliyun.com

#### 2.1.6 主机系统优化 ####

> limit优化

ulimit -SHn 65535

cat <<EOF >> /etc/security/limits.conf
    * soft nofile 655360
    * hard nofile 131072
    * soft nproc 655350
    * hard nproc 655350
    * soft memlock unlimited
    * hard memlock unlimited
    EOF

#### 2.1.7 ipvs管理工具安装及模块加载 ####

> 为集群节点安装，负载均衡节点不用安装

yum -y install ipvsadm ipset sysstat conntrack libseccomp

所有节点配置ipvs模块，在内核4.19+版本nf_conntrack_ipv4已经改为nf_conntrack， 4.18以下使用nf_conntrack_ipv4即可： 
     
    modprobe -- ip_vs 
    modprobe -- ip_vs_rr 
    modprobe -- ip_vs_wrr 
    modprobe -- ip_vs_sh 
    modprobe -- nf_conntrack

创建 /etc/modules-load.d/ipvs.conf 并加入以下内容： 
    cat >/etc/modules-load.d/ipvs.conf <<EOF 
    ip_vs 
    ip_vs_lc 
    ip_vs_wlc 
    ip_vs_rr 
    ip_vs_wrr 
    ip_vs_lblc 
    ip_vs_lblcr 
    ip_vs_dh 
    ip_vs_sh 
    ip_vs_fo 
    ip_vs_nq 
    ip_vs_sed 
    ip_vs_ftp 
    ip_vs_sh 
    nf_conntrack 
    ip_tables 
    ip_set 
    xt_set 
    ipt_set 
    ipt_rpfilter 
    ipt_REJECT 
    ipip 
    EOF

设置为开机启动
    systemctl enable --now systemd-modules-load.service

如果执行开机启动失败了，提示如下信息：

Job for systemd-modules-load.service failed because the control process exited with error code. See "systemctl status systemd-modules-load.service" and "journalctl -xe" for details.
    
    Failed to find module 'ip_vs_fo'

具体原因是内核版本问题，不过也可以将文件中的ip\_vs\_fo 去掉，然后继续执行

#### 2.1.8 Linux内核升级 ####

> 在所有节点中安装,需要重新操作系统更换内核。

[root@localhost ~]# yum -y install perl

[root@localhost ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

[root@localhost ~]# yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm

[root@localhost ~]# yum  --enablerepo="elrepo-kernel"  -y install kernel-ml.x86_64

[root@localhost ~]# grub2-set-default 0

[root@localhost ~]# grub2-mkconfig -o /boot/grub2/grub.cfg

#### 2.1.9 Linux内核优化 ####

cat <<EOF > /etc/sysctl.d/k8s.conf
    net.ipv4.ip_forward = 1
    net.bridge.bridge-nf-call-iptables = 1
    net.bridge.bridge-nf-call-ip6tables = 1
    fs.may_detach_mounts = 1
    vm.overcommit_memory=1
    vm.panic_on_oom=0
    fs.inotify.max_user_watches=89100
    fs.file-max=52706963
    fs.nr_open=52706963
    net.netfilter.nf_conntrack_max=2310720
    
    net.ipv4.tcp_keepalive_time = 600
    net.ipv4.tcp_keepalive_probes = 3
    net.ipv4.tcp_keepalive_intvl =15
    net.ipv4.tcp_max_tw_buckets = 36000
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_max_orphans = 327680
    net.ipv4.tcp_orphan_retries = 3
    net.ipv4.tcp_syncookies = 1
    net.ipv4.tcp_max_syn_backlog = 16384
    net.ipv4.ip_conntrack_max = 131072
    net.ipv4.tcp_max_syn_backlog = 16384
    net.ipv4.tcp_timestamps = 0
    net.core.somaxconn = 16384
    EOF

sysctl --system

所有节点配置完内核后，重启服务器，保证重启后内核依旧加载
    reboot -h now

重启后查看结果：
    lsmod | grep --color=auto -e ip_vs -e nf_conntrack

#### 2.1.10 其它工具安装(选装) ####

yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git lrzsz -y

### 2.2 负载均衡器准备 ###

#### 2.2.1 安装haproxy与keepalived ####

在负载均衡两台服务器上安装

yum -y install haproxy keepalived

#### 2.2.2 HAProxy配置 ####

cat >/etc/haproxy/haproxy.cfg<<"EOF"
    global
     maxconn 2000
     ulimit-n 16384
     log 127.0.0.1 local0 err
     stats timeout 30s
    
    defaults
     log global
     mode http
     option httplog
     timeout connect 5000
     timeout client 50000
     timeout server 50000
     timeout http-request 15s
     timeout http-keep-alive 15s
    
    frontend monitor-in
     bind *:33305
     mode http
     option httplog
     monitor-uri /monitor
    
    frontend k8s-master
     bind 0.0.0.0:6443
     bind 127.0.0.1:6443
     mode tcp
     option tcplog
     tcp-request inspect-delay 5s
     default_backend k8s-master
    
    backend k8s-master
     mode tcp
     option tcplog
     option tcp-check
     balance roundrobin #负载均衡策略
     default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
     server  k8s-master1  192.168.10.103:6443 check
     server  k8s-master2  192.168.10.104:6443 check
     server  k8s-master3  192.168.10.105:6443 check
    EOF

#### 2.2.3 KeepAlived ####

> 主从配置不一致，需要注意。

KeepAlived主要是对haproxy进行监控

ha1:
    
    cat >/etc/keepalived/keepalived.conf<<"EOF"
    ! Configuration File for keepalived
    global_defs {
        
       router_id LVS_DEVEL
    script_user root
       enable_script_security
    }
    vrrp_script chk_apiserver {
        
       script "/etc/keepalived/check_apiserver.sh"
       interval 5
       weight -5
       fall 2 
    rise 1
    }
    vrrp_instance VI_1 {
        
       state MASTER #指定主节点
       interface eth0 #网卡
       mcast_src_ip 192.168.10.101 #本机ip
       virtual_router_id 51
       priority 100 #优先级
       advert_int 2
       authentication {
        
           auth_type PASS
           auth_pass K8SHA_KA_AUTH
       }
       virtual_ipaddress {
        
           192.168.10.100 #虚拟ip 主节点和备份节点一样的
       }
       track_script {
        
          chk_apiserver #通过脚本实现监控
       }
    }
    EOF

ha2:
    
    cat >/etc/keepalived/keepalived.conf<<"EOF"
    ! Configuration File for keepalived
    global_defs {
        
       router_id LVS_DEVEL
    script_user root
       enable_script_security
    }
    vrrp_script chk_apiserver {
        
       script "/etc/keepalived/check_apiserver.sh"
      interval 5
       weight -5
       fall 2 
    rise 1
    }
    vrrp_instance VI_1 {
        
       state BACKUP
       interface eth0
       mcast_src_ip 192.168.10.102
       virtual_router_id 51
       priority 99
       advert_int 2
       authentication {
        
           auth_type PASS
           auth_pass K8SHA_KA_AUTH
       }
       virtual_ipaddress {
        
           192.168.10.100
       }
       track_script {
        
          chk_apiserver
       }
    }
    EOF

#### 2.2.4 健康检查脚本 ####

> ha1及ha2均要配置（相同）

cat > /etc/keepalived/check_apiserver.sh <<"EOF"
    #!/bin/bash
    err=0 #定义变量
    for k in $(seq 1 3)
    do
       check_code=$(pgrep haproxy) #检查haproxy进程
       if [[ $check_code == "" ]]; then
           err=$(expr $err + 1)
           sleep 1
           continue
       else
           err=0
           break
       fi
    done
    
    if [[ $err != "0" ]]; then
       echo "systemctl stop keepalived"
       /usr/bin/systemctl stop keepalived
       exit 1
    else
       exit 0
    fi
    EOF

chmod +x /etc/keepalived/check_apiserver.sh

#### 2.2.5 启动服务并验证 ####

systemctl daemon-reload
    systemctl enable --now haproxy
    systemctl enable --now keepalived

ip address show

ss -anput | grep ":6443"

http://192.168.10.101:33305/monitor

[Kubernetes]: https://xy5462.blog.csdn.net/article/details/132093671?spm=1001.2014.3001.5502
[kubeadm_k8s]: https://xy5462.blog.csdn.net/article/details/132093723?spm=1001.2014.3001.5502
[Kubernetes 1]: https://xy5462.blog.csdn.net/article/details/132093826?spm=1001.2014.3001.5502
[Kubernetes_ETCD]: https://xy5462.blog.csdn.net/article/details/132093851?spm=1001.2014.3001.5502
[Kubernetes_api-server]: https://xy5462.blog.csdn.net/article/details/132093879?spm=1001.2014.3001.5502
[Kubernetes_kubectl_kube-controller-manager_kube-scheduler]: https://xy5462.blog.csdn.net/article/details/132093931?spm=1001.2014.3001.5502
[Kubernetes_kubelet_kube-proxy_Calico_CoreDNS]: https://xy5462.blog.csdn.net/article/details/132094015?spm=1001.2014.3001.5502
[Kubernetes_Kubernetes]: https://xy5462.blog.csdn.net/article/details/132094054?spm=1001.2014.3001.5502