kubernetes 1.15.3部署windows结点

深藏阁楼爱情的钟 2024-04-18 07:49 16阅读 0赞

注意：windows结点需要使用windows server 2019 1809以上的**英文版**，最新的中文版也许可以，但是老一点的中文版装不上KB4489899补丁包

## master端 ##

1. 安装所需包  
执行如下命令

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
    [kubernetes]
    name=Kubernetes
    baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
    enabled=1
    gpgcheck=1
    repo_gpgcheck=1
    gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
    EOF
    
    echo "nameserver 114.114.114.114" > /etc/resolv.conf
    wget http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-11.noarch.rpm
    rpm -ivh epel-release-7-11.noarch.rpm
    cd /etc/yum.repos.d/
    wget http://mirrors.163.com/.help/CentOS7-Base-163.repo
    yum clean all
    yum install -y conntrack-tools
    yum install -y kubelet kubeadm kubectl ipvsadm  kubernetes-cni

2. 准备环境配置

###############
    #配置环境
    # 临时禁用selinux
    # 永久关闭 修改/etc/sysconfig/selinux文件设置
    sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux
    setenforce 0
     
    # 临时关闭swap
    # 永久关闭 注释/etc/fstab文件里swap相关的行
    swapoff -a
     
    # 开启forward
    # Docker从1.13版本开始调整了默认的防火墙规则
    # 禁用了iptables filter表中FOWARD链
    # 这样会引起Kubernetes集群中跨Node的Pod无法通信
     
    iptables -P FORWARD ACCEPT
    iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
     
    # 配置转发相关参数，否则可能会出错
    cat <<EOF >  /etc/sysctl.d/k8s.conf
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    vm.swappiness=0
    EOF
    sysctl --system 
    
    # 加载ipvs相关内核模块
    # 如果重新开机，需要重新加载
    yum install ipset ipvsadm -y
    cat > /etc/sysconfig/modules/ipvs.modules <<EOF
    #!/bin/bash
    modprobe -- ip_vs
    modprobe -- ip_vs_rr
    modprobe -- ip_vs_wrr
    modprobe -- ip_vs_sh
    modprobe -- nf_conntrack_ipv4
    EOF
    chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
    
    lsmod | grep ip_vs

从[https://download.docker.com/linux/centos/7/x86\_64/stable/Packages/][https_download.docker.com_linux_centos_7_x86_64_stable_Packages]下载rpm包安装docker

3. 安装kubernetes

docker pull mirrorgooglecontainers/kube-proxy:v1.15.3
    docker tag mirrorgooglecontainers/kube-proxy:v1.15.3 k8s.gcr.io/kube-proxy:v1.15.3
    
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1
    docker tag  registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1   k8s.gcr.io/coredns:1.3.1
    
    docker pull mirrorgooglecontainers/kube-controller-manager:v1.15.3
    docker tag mirrorgooglecontainers/kube-controller-manager:v1.15.3  k8s.gcr.io/kube-controller-manager:v1.15.3
    
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.15.3
    docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.15.3  k8s.gcr.io/kube-apiserver:v1.15.3
    
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.15.3
    docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.15.3 k8s.gcr.io/kube-scheduler:v1.15.3
    
    docker pull mirrorgooglecontainers/pause:3.1
    docker tag  mirrorgooglecontainers/pause:3.1 k8s.gcr.io/pause:3.1
    
    docker pull mirrorgooglecontainers/etcd:3.3.10
    docker tag mirrorgooglecontainers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
    
    kubeadm init --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12
    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
    mkdir -p kube/yaml && cd kube/yaml
    kubectl get ds/kube-proxy -o go-template='{
       {.spec.updateStrategy.type}}{
       {"\n"}}' --namespace=kube-system
    wget https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/l2bridge/manifests/node-selector-patch.yml
    kubectl patch ds/kube-proxy --patch "$(cat node-selector-patch.yml)" -n=kube-system

4. 使用flanne的vxlan方式组网（关键步骤）

sudo sysctl net.bridge.bridge-nf-call-iptables=1
    wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

使用vi命令修改该文件，  
找到net-conf.json项，加入  
        "VNI" : 4096,  
        "Port": 4789  
使其看来如下：

net-conf.json: |
        {
          "Network": "10.244.0.0/16",
          "Backend": {
            "Type": "vxlan",
            "VNI" : 4096,
            "Port": 4789
          }
        }

找到cni-conf.json项，修改Name为vxlan0（其默认名为cbr0）  
使其看来如下：

cni-conf.json: |
        {
          "name": "vxlan0",
          "plugins": [
            {
              "type": "flannel",
              "delegate": {
                "hairpinMode": true,
                "isDefaultGateway": true
              }
            },
            {
              "type": "portmap",
              "capabilities": {
                "portMappings": true
              }
            }
          ]
        }

针对很多系统的docker镜像  
修改保存退出后，执行

kubectl apply -f kube-flannel.yml
    kubectl patch ds/kube-flannel-ds-amd64 --patch "$(cat node-selector-patch.yml)" -n=kube-system

5. 加入其他结点：  
其他需要加入的结点执行，通常命令会在kubeadm执行完成时给出提示  
kubeadm join <Master\_IP>:6443 --token <some\_token> --discovery-token-ca-cert-hash <some\_hash>  
例如：  
kubeadm join 192.168.110.100:6443 --token jsfh39.4855pr1w1234231 \\  
    --discovery-token-ca-cert-hash sha256:c55eb95acfc056ccaaf62e71ee4033a47b49eb9b103f1233214567

systemctl enable docker.service

systemctl enable kubelet.service

##    windows端 ##

1. 安装docker  
Install-Module -Name DockerMsftProvider -Repository PSGallery -Force  
Install-Package -Name docker -ProviderName DockerMsftProvider  
\#如果下载失败，就手动下载：  
https://dockermsft.blob.core.windows.net/dockercontainer/docker-19-03-1.zip  
之后，将其拷贝到C:\\Users\\Administrator\\AppData\\Local\\Temp\\2\\DockerMsftProvider\\目录下，执行如下操作

Get-FileHash -Path C:\\Users\\Administrator\\AppData\\Local\\Temp\\2\\DockerMsftProvider\\Docker-19-03-1.zip -Algorithm SHA256  
 Install-Package -Name docker -ProviderName DockerMsftProvider -Verbose  
最后重启系统，执行：

Restart-Computer -Force  
之后，重启docker的命令为  
net stop docker  
net start docker  
启动docker服务（自启动）  
Restart-Service docker

2. 启动hyper-v  
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All

3. 安装补丁  
https://www.catalog.update.microsoft.com/Search.aspx?q=KB4489899  
下载server 2019对应的补丁包

命令行执行  
sconfig  
菜单选择6，在Search for for (A)ll updates or (R)ecommended updates only?选择A，安装所以更新包

4. 下载镜像  
docker pull mcr.microsoft.com/k8s/core/pause:1.0.0  
docker pull mcr.microsoft.com/k8s/core/pause:1.2.0  
docker pull mcr.microsoft.com/windows/nanoserver:1809  
docker tag mcr.microsoft.com/windows/nanoserver:1809 microsoft/nanoserver:latest

5. 部署程序  
mkdir c:\\k  
$env:Path += ";C:\\k"  
\[Environment\]::SetEnvironmentVariable("Path", $env:Path + ";C:\\k", \[EnvironmentVariableTarget\]::Machine)

下载windows程序  
进入https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.15.md，例如  
https://dl.k8s.io/v1.15.3/kubernetes-node-windows-amd64.tar.gz，解压后拷贝到c:\\k下

将linux的master结点上的/etc/kubernetes/admin.conf文件拷贝为config到k目录下  
$env:KUBECONFIG="C:\\k\\config"  
\[Environment\]::SetEnvironmentVariable("KUBECONFIG", "C:\\k\\config", \[EnvironmentVariableTarget\]::User)

下载启动脚本  
\[Net.ServicePointManager\]::SecurityProtocol = \[Net.SecurityProtocolType\]::Tls12  
wget https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/start.ps1 -o c:\\k\\start.ps1

6. 进入控制面板，关闭防火墙

7. 启动程序

docker run microsoft/nanoserver:latest

docker run -d mcr.microsoft.com/k8s/core/pause:1.0.0  \#这一步很重要，如果漏掉这一步，可能会导致vxlan的网络创建失败，可以通过Get-NetAdapter | Format-Table Name,InterfaceDescription,ifIndex,Status,LinkSpeed,MediaConnectionState来检查  
cd c:\\k  
.\\start.ps1 -ManagementIP <Windows Node IP> -NetworkMode <network mode>  -ClusterCIDR <Cluster CIDR> -ServiceCIDR <Service CIDR> -KubeDnsServiceIP <Kube-dns Service IP> -LogDir <Log directory> -KubeletFeatureGates "WinOverlay=true"  
<Windows Node IP> 为windows本机的ip  
<network mode>为网络模式，这里使用了vxlan方式，填入overlay  
<Cluster CIDR>输入10.244.0.0/16其值是在kubeadm初始化的过程中定义的  
<Service CIDR>输入10.96.0.0/12  
<Kube-dns Service IP>输入10.96.0.10  
<Log directory>输入日志路径c:\\var\\k8s

例如  
.\\start.ps1 -ManagementIP 192.168.110.159 -NetworkMode overlay  -ClusterCIDR 10.244.0.0/16 -ServiceCIDR 10.96.0.0/12 -KubeDnsServiceIP 10.96.0.10 -LogDir c:\\var\\k8s\\ -KubeletFeatureGates "WinOverlay=true"  
第一次执行时，耗时较多，如果中间卡顿，可以找到具体位置，手动下载文件，并拷贝到相应路径下

[https_download.docker.com_linux_centos_7_x86_64_stable_Packages]: https://download.docker.com/linux/centos/7/x86_64/stable/Packages/