K8s：先跑起来，部署Kubernetes Cluster集群

水深无声 2022-12-17 08:56 442阅读 0赞

> 更新中

本文为Kubernetes系列的第一篇文章。

> 越是门槛高的知识，就越需要一个最小化可用的系统。如果直接上来就只学习理论知识和概念，就很容易造成：从入门到放弃。  
> 按照一贯的学习思路在短时间内搭建出来一个可用的系统。这样可以尽快建立起对学习对象的感性认识。先用起来，然后快速了解基本概念、功能和使用场景，进而不断加深研究与开发。

参考链接  
[使用kubeadm创建Kubernetes集群][kubeadm_Kubernetes]

# 0x00环境准备 #

本文使用主机的系统皆为CentOS 7，所有机器之间网络连接正常。  
确保3台主机的 /etc/hostname 和/etc/hosts文件中已经修改为正确的主机名。以及相应的主机名称解析，修改后，请重启系统。  
注意：**请确保CPU至少2核，内存2G**  
所有命令皆在root用户下运行

Master：k8s-master  
Node：k8s-node1，k8s-node2

<table> 
 <thead> 
  <tr> 
   <th>主机名</th> 
   <th>IP地址</th> 
   <th>作用</th> 
   <th>配置</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>k8s-master</td> 
   <td>192.168.2.103</td> 
   <td>主节点Master</td> 
   <td>2U4G64G</td> 
  </tr> 
  <tr> 
   <td>k8s-node1</td> 
   <td>192.168.2.123</td> 
   <td>从节点Node</td> 
   <td>2U4G64G</td> 
  </tr> 
  <tr> 
   <td>k8s-node2</td> 
   <td>192.168.2.130</td> 
   <td>从节点Node</td> 
   <td>2U4G64G</td> 
  </tr> 
 </tbody> 
</table>

## 1.host相关配置 ##

### （1）文件修改 ###

配置文件/etc/hosts

**Master节点**

hostnamectl set-hostname k8s-master
    hostname
    hostnamectl
    cat /etc/hosts
    
    echo 192.168.2.103 k8s-master >> /etc/hosts
    echo 192.168.2.123 k8s-node1 >> /etc/hosts
    echo 192.168.2.130 k8s-node2 >> /etc/hosts
    cat /etc/hosts

**Node1节点**

hostnamectl set-hostname k8s-node1
    hostname
    hostnamectl
    cat /etc/hosts
    
    echo 192.168.2.103 k8s-master >> /etc/hosts
    echo 192.168.2.123 k8s-node1 >> /etc/hosts
    echo 192.168.2.130 k8s-node2 >> /etc/hosts
    cat /etc/hosts

**Node2节点**

hostnamectl set-hostname k8s-node2
    hostname
    hostnamectl
    cat /etc/hosts
    
    echo 192.168.2.103 k8s-master >> /etc/hosts
    echo 192.168.2.104 k8s-node1 >> /etc/hosts
    echo 192.168.2.125 k8s-node2 >> /etc/hosts
    cat /etc/hosts

注：  
①某些发行版本在/etc/hosts 文件中添加了多余的条目，该条目将实际的主机名解析为另一个回送IP地址，例如127.0.1.1。如果有的话，则必须将其注释或删除，以防止名称解析问题。  
②不要删除127.0.0.1条目。  
③每一个节点都需要对其他节点进行主机名的解析配置。  
④主机名修改完成后，退出当前终端，重新登录服务器。

### （2）验证连通性 ###

在每一个节点执行

ping -c 4 k8s-master
    ping -c 4 k8s-node1
    ping -c 4 k8s-node2
    ping -c 4 qq.com

## 2.防火墙配置 ##

在实验环境中可以选择直接禁用防火墙，但是在生产环境中根据实际的需求进行规则的配置。  
禁用防火墙

### （1）关闭firewalld ###

在CentOS7上面防火墙是firewalld服务，停止并且禁止开机启动firewalld.service

systemctl stop firewalld.service
    systemctl disable firewalld.service
    systemctl status firewalld.service

### （2）关闭 selinux ###

setenforce 0
    getenforce
    sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
    sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
    grep SELINUX=disabled /etc/sysconfig/selinux
    grep SELINUX=disabled /etc/selinux/config

## 3.配置k8s.conf ##

参考[链接][Link 1]

cat <<EOF >  /etc/sysctl.d/k8s.conf
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    EOF

sysctl --system

执行如下命令使修改生效：

modprobe br_netfilter
    sysctl -p /etc/sysctl.d/k8s.conf

## 4.swap配置 ##

禁用swap  
Linux swapoff命令用于关闭系统交换区(swap area)。swapoff实际上为swapon的符号连接，可用来关闭系统的交换区。

sed -i '/swap/ s/^/#/' /etc/fstab
    swapoff -a
    free -m

> \-a 将/etc/fstab文件中所有设置为swap的设备关闭  
> \-h 帮助信息  
> \-V 版本信息

## 5.通用YUM源配置 ##

注：博主使用阿里云的ISO镜像，已经默认为阿里云的镜像源，故跳过此步骤。  
（1）备份域名原repo文件，在CentOS中配置使用网易和阿里的开源镜像。

cd
    mkdir ori_repo-config
    mv /etc/yum.repos.d/* ./ori_repo-config/
    cd /etc/yum.repos.d/
    wget http://mirrors.aliyun.com/repo/Centos-7.repo
    wget http://mirrors.163.com/.help/CentOS7-Base-163.repo

（2）更新源

yum clean all
    yum makecache
    yum repolist
    cd

## 6.基础安装 ##

### （1）Linux常用工具 ###

yum install -y lsof vim net-tools wget git curl

yum install -y yum-utils device-mapper-persistent-data lvm2

### （2）NTP时间同步 ###

使用下面的命令一次性完成配置，详细配置附后。

yum  -y install chrony
    sed -i '3s/0.centos.pool.ntp.org/ntp1.aliyun.com/g' /etc/chrony.conf
    sed -i '4s/1.centos.pool.ntp.org/ntp2.aliyun.com/g' /etc/chrony.conf
    sed -i '5,6s/^/#/g' /etc/chrony.conf
    sed -i "s|#allow 192.168.0.0/16|allow 192.168.2.0/24|g" /etc/chrony.conf
    systemctl restart chronyd.service
    systemctl status chronyd.service
    systemctl enable chronyd.service
    systemctl list-unit-files |grep chronyd.service
    timedatectl set-timezone Asia/Shanghai
    chronyc sources
    timedatectl status

（1）编辑chrony.conf文件，使用下面的命令完成添加。

yum  -y install chrony
    vim /etc/chrony.conf

# add follow
    server ntp1.aliyun.com iburst
    server ntp2.aliyun.com iburst
    allow 192.168.232.2/24

注意：  
①替换NTP\_SERVER为合适的更准确（较低层）NTP服务器的主机名或IP地址。  
②该配置支持多个server键。默认情况下，控制器节点通过公共服务器池同步时间。但是也可以选择配置备用服务器，例如组织提供的服务器。  
③要使其他节点能够连接到控制器节点上的chrony守护程序，请将此密钥添加到上述相同的chrony.conf文件中：allow 10.0.0.0/24，如有必要，请替换10.0.0.0/24为您的子网描述。

（2）重新启动NTP服务，并配置开机启动：

systemctl restart chronyd.service
    systemctl status chronyd.service
    systemctl enable chronyd.service
    systemctl list-unit-files |grep chronyd.service

（3）设置时区，同步时间

timedatectl set-timezone Asia/Shanghai
    chronyc sources
    timedatectl status

# 0x01安装步骤 #

## 1.安装docker环境 ##

所有节点都需要安装docker，  
docker官方[安装指南][Link 2]  
菜鸟教程[参考][Link 3]

阿里教程：  
https://developer.aliyun.com/article/110806  
https://yq.aliyun.com/articles/626118

### （1）卸载旧的版本 ###

yum remove docker \
                      docker-client \
                      docker-client-latest \
                      docker-common \
                      docker-latest \
                      docker-latest-logrotate \
                      docker-logrotate \
                      docker-engine

### （2）官方脚本自动安装 ###

手动安装，请参考菜鸟教程的[安装指南][Link 3]  
通过阿里云安装。

curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun

docker version

systemctl start docker
    systemctl status docker
    systemctl enable docker
    systemctl list-unit-files |grep chronyd.service

### （3）配置Docker加速 ###

快速配置

cat > /etc/docker/daemon.json <<EOF
    {
        "registry-mirrors":["https://docker.mirrors.ustc.edu.cn/"]
    }
    EOF

详细情况如下

touch /etc/docker/daemon.json
    vi /etc/docker/daemon.json

{
        
        "registry-mirrors":["https://docker.mirrors.ustc.edu.cn/"]
    }

重新启动Docker并查看Docker信息

systemctl daemon-reload
    docker info
    systemctl restart docker
    systemctl status docker

至此，Docker在所有节点安装完成。

## 2.安装Kubernetes工具 ##

在所有节点安装kubectl、kubelet、kubeadm

### （1）端口开放说明 ###

**Master主节点**

<table> 
 <thead> 
  <tr> 
   <th align="left">端口范围</th> 
   <th align="left">用途</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td align="left">6443 *</td> 
   <td align="left">Kubernetes API server</td> 
  </tr> 
  <tr> 
   <td align="left">2379-2380</td> 
   <td align="left">etcd server client API</td> 
  </tr> 
  <tr> 
   <td align="left">10250</td> 
   <td align="left">Kubelet API</td> 
  </tr> 
  <tr> 
   <td align="left">10251</td> 
   <td align="left">kube-scheduler</td> 
  </tr> 
  <tr> 
   <td align="left">10252</td> 
   <td align="left">kube-controller-manager</td> 
  </tr> 
  <tr> 
   <td align="left">10255</td> 
   <td align="left">Read-only Kubelet API (Heapster)</td> 
  </tr> 
 </tbody> 
</table>

**Node从节点**

<table> 
 <thead> 
  <tr> 
   <th align="left">端口范围</th> 
   <th align="left">用途</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td align="left">10250</td> 
   <td align="left">Kubelet API</td> 
  </tr> 
  <tr> 
   <td align="left">10255</td> 
   <td align="left">Read-only Kubelet API (Heapster)</td> 
  </tr> 
  <tr> 
   <td align="left">30000-32767</td> 
   <td align="left"><a href="https://kubernetes.io/docs/concepts/services-networking/service" rel="nofollow">NodePort Services</a>默认端口范围。</td> 
  </tr> 
 </tbody> 
</table>

**参考链接**  
http://docs.kubernetes.org.cn/457.html  
https://www.kubernetes.org.cn/installkubectl  
ubuntu：  
https://www.cnblogs.com/mouseleo/p/11930784.html  
https://blog.csdn.net/shykevin/article/details/98811021

centos：  
https://yq.aliyun.com/articles/626118

*  kubectl  
    kubectl是Kubernetes命令行工具。通过运行kubectl可以部署和管理应用，查看各类资源，创建、删除和更新各种组件。
 *  kublet  
    kublet运行在Cluster的所有节点上，负责启动Pod和容器。
 *  kubeadm  
    kubeadm用于初始化Cluster。

### （2）配置Kubernetes的阿里云YUM ###

参考[链接][Link 1]

https://yq.aliyun.com/articles/626118

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
    [kubernetes]
    name=Kubernetes
    baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
    enabled=1
    gpgcheck=1
    repo_gpgcheck=1
    gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
    EOF

### （3）安装kubectl、kubelet、kubeadm ###

参考[指南][Link 4]  
https://yq.aliyun.com/articles/626118

使用Kubernetes命令行工具kubectl在Kubernetes上部署和管理应用程序。使用kubectl，可以检查集群资源; 创建，删除和更新组件。  
kubectl可作为快速应用程序使用。

yum clean all
    yum makecache fast -y
    yum install -y kubelet-1.19.3 kubeadm-1.19.3 kubectl-1.19.3

启动kubelet服务

systemctl start kubelet
    systemctl status kubelet
    systemctl enable kubelet

此时status应该是无法正常启动的，**可以忽略**，具体的解决措施请继续下一节的部署操作。

systemctl status kubelet

问题解释：[kubelet服务启动失败，错误代码255][kubelet_255]

至此，所有节点，Kubernetes工具安装完成。

## 3.安装Kubernetes集群 ##

### （1）配置 kubelet启动参数 ###

**解决上一节的kubulet启动问题**

[https://www.kubernetes.org.cn/installkubectl][Link 4]

[CentOS7安装kubernetes1.11.2图文完整版][CentOS7_kubernetes1.11.2]

[用 kubeadm 搭建集群环境][kubeadm]

安装完成后，我们还需要对kubelet进行配置，因为用yum源的方式安装的kubelet生成的配置文件将参数–cgroup-driver改成了systemd，而 docker 的cgroup-driver是cgroupfs，这二者必须一致才行，我们可以通过docker info命令查看：

docker info |grep Cgroup

修改文件 kubelet 的配置文件，将其中的KUBELET\_CGROUP\_ARGS参数更改成cgroupfs：

以前的版本的路径：/etc/systemd/system/kubelet.service.d/10-kubeadm.conf，

操作：

sed -i "s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

当前版本的路径为：/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

在`[service]`添加一行`Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"`

sed -i 'N;4iEnvironment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"' /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

修改完成后，重新加载我们的配置文件即可：

systemctl daemon-reload
    systemctl restart kubelet
    systemctl status kubelet

此时status应该是无法正常启动的，**可以忽略**，具体的解决措施请继续下一节的部署操作。

systemctl status kubelet

问题解释：[kubelet服务启动失败，错误代码255][kubelet_255]

> 看Kubernets官方文档，其实里面已经写了很清楚:  
> The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do. This crashloop is expected and normal, please proceed with the next step and the kubelet will start running normally.

简单地说就是在kubeadm init 之前kubelet会不断重启。

### （2）准备镜像 ###

K8s每个版本需要的镜像版本号在kubeadm Setup Tool Reference Guide这个文档的的Running kubeadm without an internet connection一节里有写。所以可以根据安装的实际版本来调整这个脚本的参数。可以使用命令获取需要的镜像版本如下:  
[国内拉取google kubernetes镜像][google kubernetes]

kubeadm config images list

[root@k8s-master ~]# kubeadm config images list
    W1106 15:04:10.306351    2206 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
    k8s.gcr.io/kube-apiserver:v1.19.3
    k8s.gcr.io/kube-controller-manager:v1.19.3
    k8s.gcr.io/kube-scheduler:v1.19.3
    k8s.gcr.io/kube-proxy:v1.19.3
    k8s.gcr.io/pause:3.2
    k8s.gcr.io/etcd:3.4.13-0
    k8s.gcr.io/coredns:1.7.0
    [root@k8s-master ~]#

根据镜像仓库获取地址调整脚本，如下:

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.19.3
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.19.3
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.19.3
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.19.3
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0

sh push.sh

docker images

准备好的镜像如下,与kubeadm config images list时输出的镜像名称版本一致。  
我们这里是k8s.gcr.io所以 脚本中pull 时重命名使用的是

docker tag ${username}/${image} k8s.gcr.io/${image}

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.19.3 k8s.gcr.io/kube-apiserver:v1.19.3
    docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.19.3 k8s.gcr.io/kube-controller-manager:v1.19.3
    docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.19.3 k8s.gcr.io/kube-scheduler:v1.19.3
    docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.19.3 k8s.gcr.io/kube-proxy:v1.19.3
    docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2 k8s.gcr.io/pause:3.2
    docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0 k8s.gcr.io/etcd:3.4.13-0
    docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0 k8s.gcr.io/coredns:1.7.0

kubeadm config images list
    docker images

### （3）初始化Master ###

到这里我们的准备工作就完成了，接下来我们就可以在**master**节点上用kubeadm命令来初始化我们的集群了：  
master 是控制组件运行的机器，包括etcd和API Server等（使用kubectl CLI与之通信启动）。

参考[链接][Link 1]

#### 方法一、使用如下命令 ####

注意：多次运行kubeadm init命令，需要reset

kubeadm reset

kubeadm init --kubernetes-version=v1.19.3 --pod-network-cidr=192.169.0.0/16 --image-repository=registry.aliyuncs.com/google_containers --apiserver-advertise-address=192.168.2.103

参数解释：

> – image-repository 指定镜像源，指定为阿里云的源，这样就会避免在拉取镜像超时，如果没问题，过几分钟就能看到成功的日志输入  
> –kubernetes-version 指定版本  
> –pod-network-cidr 指定pod网络地址。设置为内网网段！

命令非常简单，就是kubeadm init，后面的参数是需要安装的集群版本，因为我们这里选择flannel作为 Pod 的网络插件，所以需要指定–pod-network-cidr=10.244.0.0/16，然后是 apiserver 的通信地址，这里就是我们 master 节点的 IP 地址。执行上面的命令，如果出现running with swap on is not supported. Please disable swap之类的错误，则我们还需要增加一个参数–ignore-preflight-errors=Swap来忽略 swap 的错误提示信息：

执行情况如下

[root@k8s-master ~]# kubeadm reset
    [reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
    [reset] Are you sure you want to proceed? [y/N]: y
    [preflight] Running pre-flight checks
    W1106 15:56:46.414897   14265 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
    [reset] No etcd config found. Assuming external etcd
    [reset] Please, manually reset etcd to prevent further issues
    [reset] Stopping the kubelet service
    [reset] Unmounting mounted directories in "/var/lib/kubelet"
    [reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
    [reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
    [reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]
    
    The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d
    
    The reset process does not reset or clean up iptables rules or IPVS tables.
    If you wish to reset iptables, you must do so manually by using the "iptables" command.
    
    If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
    to reset your system's IPVS tables.
    
    The reset process does not clean your kubeconfig files and you must remove them manually.
    Please, check the contents of the $HOME/.kube/config file.

[root@k8s-master ~]# 
    [root@k8s-master ~]# kubeadm init --kubernetes-version=v1.19.3 --pod-network-cidr=192.169.0.0/16 --image-repository=registry.aliyuncs.com/google_containers --apiserver-advertise-address=192.168.2.103
    W1106 15:56:52.839787   14278 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
    [init] Using Kubernetes version: v1.19.3
    [preflight] Running pre-flight checks
    	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
    [preflight] Pulling images required for setting up a Kubernetes cluster
    [preflight] This might take a minute or two, depending on the speed of your internet connection
    [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
    [certs] Using certificateDir folder "/etc/kubernetes/pki"
    [certs] Generating "ca" certificate and key
    [certs] Generating "apiserver" certificate and key
    [certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.2.103]
    [certs] Generating "apiserver-kubelet-client" certificate and key
    [certs] Generating "front-proxy-ca" certificate and key
    [certs] Generating "front-proxy-client" certificate and key
    [certs] Generating "etcd/ca" certificate and key
    [certs] Generating "etcd/server" certificate and key
    [certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.2.103 127.0.0.1 ::1]
    [certs] Generating "etcd/peer" certificate and key
    [certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.2.103 127.0.0.1 ::1]
    [certs] Generating "etcd/healthcheck-client" certificate and key
    [certs] Generating "apiserver-etcd-client" certificate and key
    [certs] Generating "sa" key and public key
    [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
    [kubeconfig] Writing "admin.conf" kubeconfig file
    [kubeconfig] Writing "kubelet.conf" kubeconfig file
    [kubeconfig] Writing "controller-manager.conf" kubeconfig file
    [kubeconfig] Writing "scheduler.conf" kubeconfig file
    [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
    [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
    [kubelet-start] Starting the kubelet
    [control-plane] Using manifest folder "/etc/kubernetes/manifests"
    [control-plane] Creating static Pod manifest for "kube-apiserver"
    [control-plane] Creating static Pod manifest for "kube-controller-manager"
    [control-plane] Creating static Pod manifest for "kube-scheduler"
    [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
    [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
    [apiclient] All control plane components are healthy after 10.002039 seconds
    [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
    [kubelet] Creating a ConfigMap "kubelet-config-1.19" in namespace kube-system with the configuration for the kubelets in the cluster
    [upload-certs] Skipping phase. Please see --upload-certs
    [mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
    [mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
    [bootstrap-token] Using token: do7j65.ku7srin3exqu68p5
    [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
    [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
    [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
    [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
    [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
    [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
    [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
    [addons] Applied essential addon: CoreDNS
    [addons] Applied essential addon: kube-proxy
    
    Your Kubernetes control-plane has initialized successfully!
    
    To start using your cluster, you need to run the following as a regular user:
    
      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config
    
    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/
    
    Then you can join any number of worker nodes by running the following on each as root:
    
    kubeadm join 192.168.2.103:6443 --token do7j65.ku7srin3exqu68p5 \
        --discovery-token-ca-cert-hash sha256:f9848158920157dbf87f58383c0b7c28f195d1a4fcad5f205bcb68564f77f4ed 
    [root@k8s-master ~]#

要注意将上面的加入集群的命令保存下面，如果忘记保存上面的 token 和 sha256 值的话也不用担心，我们可以使用下面的命令来查找：

[root@k8s-master ~]# kubeadm token list
    TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
    do7j65.ku7srin3exqu68p5   23h         2020-11-07T15:57:08+08:00   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token
    [root@k8s-master ~]#

[root@k8s-master ~]# kubeadm init
    W1106 15:45:44.005765   11612 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
    [init] Using Kubernetes version: v1.19.3
    [preflight] Running pre-flight checks
    	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
    error execution phase preflight: [preflight] Some fatal errors occurred:
    	[ERROR Port-6443]: Port 6443 is in use
    	[ERROR Port-10259]: Port 10259 is in use
    	[ERROR Port-10257]: Port 10257 is in use
    	[ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
    	[ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
    	[ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
    	[ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
    	[ERROR Port-10250]: Port 10250 is in use
    	[ERROR Port-2379]: Port 2379 is in use
    	[ERROR Port-2380]: Port 2380 is in use
    	[ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
    [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
    To see the stack trace of this error execute with --v=5 or higher
    [root@k8s-master ~]#

#### 方法二、配置初始化文件kubeadm-init.yaml ####

查看默认配置文件的编写

kubeadm config print init-defaults > init-defaults.yaml

自行编写初始化配置文件kubeadm-init.yaml  
文件内容如下

apiVersion: kubeadm.k8s.io/v1beta2
    kind: MasterConfiguration
    kubernetesVersion: v1.19.3    # kubernetes的版本
    api:
      advertiseAddress: 192.168.2.103 # Master的IP地址
    networking:
      podSubnet: 192.168.0.0/16    # pod网络的网段
    imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers # image的仓库源

编写配置命令

cat <<EOF > /root/kubeadm-init.yaml
    apiVersion: kubeadm.k8s.io/v1beta2
    kind: MasterConfiguration
    kubernetesVersion: v1.19.3
    api:
      advertiseAddress: 192.168.2.103
    networking:
      podSubnet: 192.168.0.0/16
    imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
    EOF

开始初始化，仅master节点运行

kubeadm init --config kubeadm-init.yaml

### （4）使kubectl正常工作 ###

上面的信息记录了 kubeadm 初始化整个集群的过程，生成相关的各种证书、kubeconfig 文件、bootstraptoken 等等，  
后边是使用kubeadm join往集群中添加节点时用到的命令.  
在初始化的操作中，我们看到Linux推荐使用普通用户执行kubectl命令，因为使用root用户执行会有一些问题。

To start using your cluster, you need to run (as a regular user):

非root用户

su centos	#切换至普通用户

mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config

执行情况如下：

[root@k8s-master ~]# su centos
    [centos@k8s-master root]$ cd
    [centos@k8s-master ~]$ pwd
    /home/centos
    [centos@k8s-master ~]$ mkdir -p $HOME/.kube
    [centos@k8s-master ~]$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    
    We trust you have received the usual lecture from the local System
    Administrator. It usually boils down to these three things:
    
        #1) Respect the privacy of others.
        #2) Think before you type.
        #3) With great power comes great responsibility.
    
    [sudo] password for centos:
    [centos@k8s-master ~]$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
    [centos@k8s-master ~]$

root用户

export KUBECONFIG=/etc/kubernetes/admin.conf

执行情况如下：

[centos@k8s-master ~]$ exit
    exit
    [root@k8s-master ~]# export KUBECONFIG=/etc/kubernetes/admin.conf
    [root@k8s-master ~]#

为了使用的便捷，启用kubectl的自动补全功能。  
参考：https://blog.csdn.net/shykevin/article/details/98811021

yum install -y bash-completion
    source <(kubectl completion bash)
    echo "source <(kubectl completion bash)" >> ~/.bashrc
    source  ~/.bashrc

最后给出了将节点加入集群的命令：  
暂时不需要执行

Then you can join any number of worker nodes by running the following on each as root:
    
    kubeadm join 192.168.2.103:6443 --token eiocuu.39w009hfxddoktvh \
        --discovery-token-ca-cert-hash sha256:7910428d0ce7ec23d28c05dc9e4a4703da30eece45119d0e4aedd9f71ab6f4d0
    [root@k8s-master ~]#

我们根据上面的提示配置好 kubectl 后，就可以使用 kubectl 来查看集群的信息了：

kubectl get cs
    kubectl get csr
    kubectl get nodes

遇到问题解决记录  
https://blog.csdn.net/BigData\_Mining/article/details/87887860  
https://www.cnblogs.com/potato-chip/p/13973760.html

执行情况如下：

[root@k8s-master ~]# kubectl get cs
    Warning: v1 ComponentStatus is deprecated in v1.19+
    NAME                 STATUS      MESSAGE                                                                                       ERROR
    scheduler            Unhealthy   Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused   
    controller-manager   Unhealthy   Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused   
    etcd-0               Healthy     {
        "health":"true"}                                                                             
    [root@k8s-master ~]# kubectl get csr
    NAME        AGE     SIGNERNAME                                    REQUESTOR                CONDITION
    csr-qlmh7   8m28s   kubernetes.io/kube-apiserver-client-kubelet   system:node:k8s-master   Approved,Issued
    [root@k8s-master ~]# kubectl get nodes
    NAME         STATUS     ROLES    AGE     VERSION
    k8s-master   NotReady   master   8m28s   v1.19.3
    [root@k8s-master ~]#

### （5）安装Pod网络 ###

POD网络有多种实现方式  
http://docs.kubernetes.org.cn/459.html\#34pod

结合初始化《每天五分钟玩转kubernetes》,第3.3.1节

**在Master节点**

接下来我们来安装flannel网络插件，很简单，和安装普通的 POD 没什么两样：  
参考[链接][Link 5]

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

kubectl apply -f  kube-flannel.yml

或者

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

执行情况如下

[root@k8s-master system]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
    --2020-10-16 16:21:02--  https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
    Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.108.133
    Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.108.133|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 4813 (4.7K) [text/plain]
    Saving to: 'kube-flannel.yml'
    
    100%[================================================================================================>] 4,813       --.-K/s   in 0s
    
    2020-10-16 16:21:03 (15.6 MB/s) - 'kube-flannel.yml' saved [4813/4813]
    
    [root@k8s-master system]# kubectl apply -f  kube-flannel.yml
    podsecuritypolicy.policy/psp.flannel.unprivileged created
    clusterrole.rbac.authorization.k8s.io/flannel created
    clusterrolebinding.rbac.authorization.k8s.io/flannel created
    serviceaccount/flannel created
    configmap/kube-flannel-cfg created
    daemonset.apps/kube-flannel-ds created
    [root@k8s-master system]#

另外需要注意的是如果你的节点有多个网卡的话，需要在 kube-flannel.yml 中使用–iface参数指定集群主机内网网卡的名称，否则可能会出现 dns 无法解析。flanneld 启动参数加上`--iface=<iface-name>`

args:
    - --ip-masq
    - --kube-subnet-mgr
    - --iface=eth0

安装完成后使用 kubectl get pods 命令可以查看到我们集群中的组件运行状态，如果都是Running 状态的话，那么恭喜你，你的 master 节点安装成功了。

kubectl get pods --all-namespaces

[root@k8s-master system]# kubectl get pods --all-namespaces
    NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
    kube-system   coredns-6d56c8448f-gp6fz             1/1     Running   0          44m
    kube-system   coredns-6d56c8448f-rcxf9             1/1     Running   0          44m
    kube-system   etcd-k8s-master                      1/1     Running   0          45m
    kube-system   kube-apiserver-k8s-master            1/1     Running   0          45m
    kube-system   kube-controller-manager-k8s-master   1/1     Running   0          45m
    kube-system   kube-flannel-ds-qvgmn                1/1     Running   0          106s
    kube-system   kube-proxy-5nbfw                     1/1     Running   0          44m
    kube-system   kube-scheduler-k8s-master            1/1     Running   0          45m
    [root@k8s-master system]#

### （6）添加Node节点 ###

同样的上面的环境配置、docker 安装、kubeadmin、kubelet、kubectl 这些都在Node(10.151.30.62)节点安装配置好过后，我们就可以直接在 Node 节点上执行kubeadm join命令了（上面初始化的时候有），同样加上参数`--ignore-preflight-errors=Swap`

[root@k8s-master ~]# kubeadm init --kubernetes-version=v1.19.3 --pod-network-cidr=192.169.0.0/16 --image-repository=registry.aliyuncs.com/google_containers --apiserver-advertise-address=192.168.2.103

> Your Kubernetes control-plane has initialized successfully!
> 
> To start using your cluster, you need to run the following as a  
> regular user:
> 
> mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf  
> $HOME/.kube/config sudo chown  ( i d − u ) : (id -u): (id−u):(id -g) $HOME/.kube/config
> 
> You should now deploy a pod network to the cluster. Run “kubectl apply  
> \-f \[podnetwork\].yaml” with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/
> 
> Then you can join any number of worker nodes by running the following  
> on each as root:
> 
> kubeadm join 192.168.2.103:6443 --token do7j65.ku7srin3exqu68p5  
> –discovery-token-ca-cert-hash sha256:f9848158920157dbf87f58383c0b7c28f195d1a4fcad5f205bcb68564f77f4ed  
> `[root@k8s-master ~]#`

**在k8s-node1、k8s-node2中分别执行**

kubeadm join 192.168.2.103:6443 --token do7j65.ku7srin3exqu68p5 \
        --discovery-token-ca-cert-hash sha256:f9848158920157dbf87f58383c0b7c28f195d1a4fcad5f205bcb68564f77f4ed

[root@k8s-node1 ~]# kubeadm join 192.168.2.103:6443 --token do7j65.ku7srin3exqu68p5 \
    >     --discovery-token-ca-cert-hash sha256:f9848158920157dbf87f58383c0b7c28f195d1a4fcad5f205bcb68564f77f4ed
    [preflight] Running pre-flight checks
    	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
    [preflight] Reading configuration from the cluster...
    [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
    [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
    [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
    [kubelet-start] Starting the kubelet
    [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
    
    This node has joined the cluster:
    * Certificate signing request was sent to apiserver and a response was received.
    * The Kubelet was informed of the new secure connection details.
    
    Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
    
    [root@k8s-node1 ~]#

[root@k8s-node2 ~]# kubeadm join 192.168.2.103:6443 --token do7j65.ku7srin3exqu68p5 \
    >     --discovery-token-ca-cert-hash sha256:f9848158920157dbf87f58383c0b7c28f195d1a4fcad5f205bcb68564f77f4ed
    [preflight] Running pre-flight checks
    	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
    	[WARNING Hostname]: hostname "k8s-node2" could not be reached
    	[WARNING Hostname]: hostname "k8s-node2": lookup k8s-node2 on 192.168.2.1:53: no such host
    [preflight] Reading configuration from the cluster...
    [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
    [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
    [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
    [kubelet-start] Starting the kubelet
    [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
    
    This node has joined the cluster:
    * Certificate signing request was sent to apiserver and a response was received.
    * The Kubelet was informed of the new secure connection details.
    
    Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
    
    [root@k8s-node2 ~]#

Kubernetes-kubectl命令出现错误  
【The connection to the server localhost:8080 was refused - did you specify the right host or port?】

ll /etc/kubernetes/kubelet.conf
    echo export KUBECONFIG=/etc/kubernetes/kubelet.conf >> ~/.bash_profile
    source ~/.bash_profile

## 4.验证Kubernetes集群 ##

### （1）节点校验 ###

kubectl get nodes
    kubectl get pods --all-namespaces

[root@k8s-node1 ~]# kubectl get nodes
    NAME         STATUS   ROLES    AGE     VERSION
    k8s-master   Ready    master   23m     v1.19.3
    k8s-node1    Ready    <none>   2m21s   v1.19.3
    k8s-node2    Ready    <none>   2m10s   v1.19.3
    [root@k8s-node1 ~]# kubectl get pods --all-namespaces
    NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
    kube-system   coredns-6d56c8448f-79ml8             1/1     Running   0          23m
    kube-system   coredns-6d56c8448f-cnrjg             1/1     Running   0          23m
    kube-system   etcd-k8s-master                      1/1     Running   0          23m
    kube-system   kube-apiserver-k8s-master            1/1     Running   0          23m
    kube-system   kube-controller-manager-k8s-master   1/1     Running   0          23m
    kube-system   kube-flannel-ds-7sgnw                1/1     Running   0          10m
    kube-system   kube-flannel-ds-gqn7b                1/1     Running   0          2m21s
    kube-system   kube-flannel-ds-krpqf                1/1     Running   0          2m10s
    kube-system   kube-proxy-ljc6m                     1/1     Running   0          23m
    kube-system   kube-proxy-vbcm2                     1/1     Running   0          2m10s
    kube-system   kube-proxy-ws7tf                     1/1     Running   0          2m21s
    kube-system   kube-scheduler-k8s-master            1/1     Running   0          23m
    [root@k8s-node1 ~]#

[root@k8s-master ~]# kubectl get pods --all-namespaces
    NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
    kube-system   coredns-6d56c8448f-79ml8             1/1     Running   0          23m
    kube-system   coredns-6d56c8448f-cnrjg             1/1     Running   0          23m
    kube-system   etcd-k8s-master                      1/1     Running   0          23m
    kube-system   kube-apiserver-k8s-master            1/1     Running   0          23m
    kube-system   kube-controller-manager-k8s-master   1/1     Running   0          23m
    kube-system   kube-flannel-ds-7sgnw                1/1     Running   0          10m
    kube-system   kube-flannel-ds-gqn7b                1/1     Running   0          2m19s
    kube-system   kube-flannel-ds-krpqf                1/1     Running   0          2m8s
    kube-system   kube-proxy-ljc6m                     1/1     Running   0          23m
    kube-system   kube-proxy-vbcm2                     1/1     Running   0          2m8s
    kube-system   kube-proxy-ws7tf                     1/1     Running   0          2m19s
    kube-system   kube-scheduler-k8s-master            1/1     Running   0          23m
    [root@k8s-master ~]# kubectl get nodes
    NAME         STATUS   ROLES    AGE     VERSION
    k8s-master   Ready    master   24m     v1.19.3
    k8s-node1    Ready    <none>   3m9s    v1.19.3
    k8s-node2    Ready    <none>   2m58s   v1.19.3
    [root@k8s-master ~]#

[root@k8s-node2 ~]# kubectl get nodes
    NAME         STATUS   ROLES    AGE     VERSION
    k8s-master   Ready    master   23m     v1.19.3
    k8s-node1    Ready    <none>   2m23s   v1.19.3
    k8s-node2    Ready    <none>   2m12s   v1.19.3
    [root@k8s-node2 ~]# kubectl get pods --all-namespaces
    NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
    kube-system   coredns-6d56c8448f-79ml8             1/1     Running   0          23m
    kube-system   coredns-6d56c8448f-cnrjg             1/1     Running   0          23m
    kube-system   etcd-k8s-master                      1/1     Running   0          23m
    kube-system   kube-apiserver-k8s-master            1/1     Running   0          23m
    kube-system   kube-controller-manager-k8s-master   1/1     Running   0          23m
    kube-system   kube-flannel-ds-7sgnw                1/1     Running   0          10m
    kube-system   kube-flannel-ds-gqn7b                1/1     Running   0          2m23s
    kube-system   kube-flannel-ds-krpqf                1/1     Running   0          2m12s
    kube-system   kube-proxy-ljc6m                     1/1     Running   0          23m
    kube-system   kube-proxy-vbcm2                     1/1     Running   0          2m12s
    kube-system   kube-proxy-ws7tf                     1/1     Running   0          2m23s
    kube-system   kube-sched

### （2）服务校验 ###

kubectl version
    systemctl status kubelet

## 5.常见问题处理 ##

运行kubectl version以验证您安装的版本是否已经是最新。

kubectl version

“The connection to the server localhost:8080 was refused - did you specify the right host or port?”

ll /etc/kubernetes/kubelet.conf 
    echo export KUBECONFIG=/etc/kubernetes/kubelet.conf  ~/.bash_profile
    source ~/.bash_profile

阻止自动更新  
apt-mark hold kubelet=1.15.2-00 kubeadm=1.15.2-00 kubectl=1.15.2-00\`

hold – 标记指定软件包为保留(held back)，阻止软件自动更新

# 0x02部署应用 #

# 0x03Kubernetes Dashboard #

## 1.安装dashboard可视化插件 ##

参考链接：  
[kubernetes部署dashboard可视化插件][kubernetes_dashboard]  
[官方Kubernetes Web UI][Kubernetes Web UI]  
[github项目地址][github]

当前部署dashboard版本：v2.0.4，注意检查dashboard版本与kubernetes版本兼容性：  
v2.0.4

<table> 
 <thead> 
  <tr> 
   <th>Kubernetes version</th> 
   <th>1.16</th> 
   <th>1.17</th> 
   <th>1.18</th> 
   <th>1.19</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>Compatibility</td> 
   <td>?</td> 
   <td>?</td> 
   <td>?</td> 
   <td>✓</td> 
  </tr> 
 </tbody> 
</table>

*  `✓` Fully supported version range.
 *  `?` Due to breaking changes between Kubernetes API versions, some features might not work correctly in the Dashboard.

执行yaml文件直接部署：

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.4/aio/deploy/recommended.yaml

[root@k8s-master ~]# kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.4/aio/deploy/recommended.yaml
    namespace/kubernetes-dashboard created
    serviceaccount/kubernetes-dashboard created
    service/kubernetes-dashboard created
    secret/kubernetes-dashboard-certs created
    secret/kubernetes-dashboard-csrf created
    secret/kubernetes-dashboard-key-holder created
    configmap/kubernetes-dashboard-settings created
    role.rbac.authorization.k8s.io/kubernetes-dashboard created
    clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
    rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
    clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
    deployment.apps/kubernetes-dashboard created
    service/dashboard-metrics-scraper created
    deployment.apps/dashboard-metrics-scraper created
    [root@k8s-master ~]#

查看dashboard运行状态，以deployment方式部署，运行2个pod及2个service：

kubectl -n kubernetes-dashboard get pods
    kubectl -n kubernetes-dashboard get svc

[root@k8s-master ~]# kubectl -n kubernetes-dashboard get pods
    NAME                                         READY   STATUS              RESTARTS   AGE
    dashboard-metrics-scraper-7b59f7d4df-fddnw   1/1     Running             0          22s
    kubernetes-dashboard-665f4c5ff-gvsw4         0/1     ContainerCreating   0          22s
    [root@k8s-master ~]# kubectl -n kubernetes-dashboard get svc 
    NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
    dashboard-metrics-scraper   ClusterIP   10.98.228.65    <none>        8000/TCP   2m45s
    kubernetes-dashboard        ClusterIP   10.104.230.65   <none>        443/TCP    2m45s
    [root@k8s-master ~]#

## 2.配置访问dashboard端口 ##

这里作为演示，使用nodeport方式将dashboard服务暴露在集群外，指定使用30443端口，可自定义：

kubectl  patch svc kubernetes-dashboard -n kubernetes-dashboard \
    -p '{"spec":{"type":"NodePort","ports":[{"port":443,"targetPort":8443,"nodePort":30443}]}}'

[root@k8s-master ~]# kubectl  patch svc kubernetes-dashboard -n kubernetes-dashboard \
    > -p '{"spec":{"type":"NodePort","ports":[{"port":443,"targetPort":8443,"nodePort":30443}]}}'
    service/kubernetes-dashboard patched
    [root@k8s-master ~]#

查看暴露的service,已修改为nodeport类型：

kubectl -n kubernetes-dashboard get svc

[root@k8s-master ~]# kubectl -n kubernetes-dashboard get svc
    NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
    dashboard-metrics-scraper   ClusterIP   10.98.228.65    <none>        8000/TCP        4m37s
    kubernetes-dashboard        NodePort    10.104.230.65   <none>        443:30443/TCP   4m37s
    [root@k8s-master ~]#

> 或者下载yaml文件手动修改service部分，详细请[参考][kubernetes_dashboard]的修改service部分。

## 3.登录dashboard ##

https://github.com/kubernetes/dashboard/blob/master/docs/user/accessing-dashboard/README.md

https://<dashboard_ip>:30443

[kubeadm_Kubernetes]: http://docs.kubernetes.org.cn/459.html
[Link 1]: https://www.qikqiak.com/k8s-book/docs/16.%E7%94%A8%20kubeadm%20%E6%90%AD%E5%BB%BA%E9%9B%86%E7%BE%A4%E7%8E%AF%E5%A2%83.html
[Link 2]: https://docs.docker.com/engine/install/
[Link 3]: https://www.runoob.com/docker/centos-docker-install.html
[Link 4]: https://www.kubernetes.org.cn/installkubectl
[kubelet_255]: https://blog.csdn.net/nklinsirui/article/details/80583971
[CentOS7_kubernetes1.11.2]: https://blog.csdn.net/zzq900503/article/details/81710319
[kubeadm]: https://www.qikqiak.com/k8s-book/docs/16.%E7%94%A8%20kubeadm%20%E6%90%AD%E5%BB%BA%E9%9B%86%E7%BE%A4%E7%8E%AF%E5%A2%83.html#%E9%85%8D%E7%BD%AE-kubelet
[google kubernetes]: https://blog.csdn.net/networken/article/details/84571373
[Link 5]: https://www.qikqiak.com/k8s-book/docs/16.%E7%94%A8%20kubeadm%20%E6%90%AD%E5%BB%BA%E9%9B%86%E7%BE%A4%E7%8E%AF%E5%A2%83.html#%E5%AE%89%E8%A3%85-pod-network
[kubernetes_dashboard]: https://blog.csdn.net/networken/article/details/85607593
[Kubernetes Web UI]: https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/
[github]: https://github.com/kubernetes/dashboard