openshift 3.10部署文档

客官°小女子只卖身不卖艺 2022-04-22 12:52 203阅读 0赞

机器

<table> 
 <thead> 
  <tr> 
   <th>ip</th> 
   <th>hostname</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>10.39.47.63</td> 
   <td>openshift-master</td> 
  </tr> 
  <tr> 
   <td>10.39.47.64</td> 
   <td>openshift-node-64</td> 
  </tr> 
  <tr> 
   <td>10.39.47.65</td> 
   <td>openshift-node-65</td> 
  </tr> 
  <tr> 
   <td>10.39.47.66</td> 
   <td>openshift-node-66</td> 
  </tr> 
 </tbody> 
</table>

系统

[root@openshift-master ansible]# cat /etc/redhat-release 
    CentOS Linux release 7.2.1511 (Core)

--------------------

每台都执行yum升级并安装必要的软件包

yum clean all && yum update -y && yum install epel-release -y && yum install lrzsz tree -y

每台都配置系统时区为GMT+8

timedatectl set-timezone Asia/Shanghai

每台都安装docker，并配置docker mirror源

yum install docker   -y

cat /etc/docker/daemon.json
    {
    "registry-mirrors": ["https://registry.docker-cn.com"]
    }

每个节点都要执行  
添加host映射  
vi /etc/hosts

10.39.47.63 openshift-master openshift-master.example.com
    10.39.47.64 openshift-node-64 openshift-node-64.example.com 
    10.39.47.65 openshift-node-65 openshift-node-65.example.com 
    10.39.47.66 openshift-node-66 openshift-node-66.example.com

格式化磁盘(每个节点)

[root@openshift-node-66 ~]# fdisk /dev/vdc 
    Welcome to fdisk (util-linux 2.23.2).
    
    Changes will remain in memory only, until you decide to write them.
    Be careful before using the write command.
    
    Device does not contain a recognized partition table
    Building a new DOS disklabel with disk identifier 0x658c9b83.
    
    Command (m for help): n
    Partition type:
       p   primary (0 primary, 0 extended, 4 free)
       e   extended
    Select (default p): 
    Using default response p
    Partition number (1-4, default 1): 
    First sector (2048-545259519, default 2048): 
    Using default value 2048
    Last sector, +sectors or +size{K,M,G} (2048-545259519, default 545259519): 
    Using default value 545259519
    Partition 1 of type Linux and of size 260 GiB is set
    
    Command (m for help): w
    The partition table has been altered!
    
    Calling ioctl() to re-read partition table.
    Syncing disks.
    [root@openshift-node-66 ~]# lsblk 
    NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
    vda    253:0    0  100G  0 disk 
    └─vda1 253:1    0  100G  0 part /
    vdb    253:16   0   16G  0 disk [SWAP]
    vdc    253:32   0  260G  0 disk 
    └─vdc1 253:33   0  260G  0 part 
    [root@openshift-node-66 ~]# bi
    bind         biosdecode   biosdevname  
    [root@openshift-node-66 ~]# b
    badblocks             bg                    blkid                 btrfs                 btrfs-map-logical     bunzip2               bzip2
    base64                bind                  blockdev              btrfsck               btrfs-select-super    busctl                bzip2recover
    basename              biosdecode            bond2team             btrfs-convert         btrfstune             bzcat                 bzless
    bash                  biosdevname           bootctl               btrfs-debug-tree      btrfs-zero-log        bzcmp                 bzmore
    bashbug               blkdeactivate         break                 btrfs-find-root       build-locale-archive  bzdiff                
    bashbug-64            blkdiscard            bridge                btrfs-image           builtin               bzgrep                
    [root@openshift-node-66 ~]# bl
    blkdeactivate  blkdiscard     blkid          blockdev       
    [root@openshift-node-66 ~]# bl
    blkdeactivate  blkdiscard     blkid          blockdev       
    [root@openshift-node-66 ~]# blkid 
    /dev/vda1: UUID="277fd82b-0856-4e23-8371-1f11823281b7" TYPE="ext4" 
    /dev/vdb: LABEL="YUNIFYSWAP" UUID="48eb1df6-1663-4a52-ab30-040d552c2d76" TYPE="swap" 
    [root@openshift-node-66 ~]# mkfs.xfs /dev/vdc1
    meta-data=/dev/vdc1              isize=512    agcount=4, agsize=17039296 blks
             =                       sectsz=512   attr=2, projid32bit=1
             =                       crc=1        finobt=0, sparse=0
    data     =                       bsize=4096   blocks=68157184, imaxpct=25
             =                       sunit=0      swidth=0 blks
    naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
    log      =internal log           bsize=4096   blocks=33279, version=2
             =                       sectsz=512   sunit=0 blks, lazy-count=1
    realtime =none                   extsz=4096   blocks=0, rtextents=0
    [root@openshift-node-66 ~]# blkid 
    /dev/vda1: UUID="277fd82b-0856-4e23-8371-1f11823281b7" TYPE="ext4" 
    /dev/vdb: LABEL="YUNIFYSWAP" UUID="48eb1df6-1663-4a52-ab30-040d552c2d76" TYPE="swap" 
    /dev/vdc1: UUID="ade4f022-1f2d-4854-a34e-876d874b71f8" TYPE="xfs" 
    [root@openshift-node-66 ~]# vi /etc/fstab 
    [root@openshift-node-66 ~]# 
    [root@openshift-node-66 ~]# 
    [root@openshift-node-66 ~]# 
    [root@openshift-node-66 ~]# 
    [root@openshift-node-66 ~]# 
    [root@openshift-node-66 ~]# 
    [root@openshift-node-66 ~]# systemctl stop docker 
    [root@openshift-node-66 ~]# mount -a
    [root@openshift-node-66 ~]# systemctl start docker 
    [root@openshift-node-66 ~]# lsblk 
    NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
    vda    253:0    0  100G  0 disk 
    └─vda1 253:1    0  100G  0 part /
    vdb    253:16   0   16G  0 disk [SWAP]
    vdc    253:32   0  260G  0 disk 
    └─vdc1 253:33   0  260G  0 part /var/lib/docker

需要打开selinux

[root@openshift-master ~]# cat /etc/selinux/config 
    
    # This file controls the state of SELinux on the system.
    # SELINUX= can take one of these three values:
    #     enforcing - SELinux security policy is enforced.
    #     permissive - SELinux prints warnings instead of enforcing.
    #     disabled - No SELinux policy is loaded.
    SELINUX=enforcing
    # SELINUXTYPE= can take one of these two values:
    #     targeted - Targeted processes are protected,
    #     minimum - Modification of targeted policy. Only selected processes are protected. 
    #     mls - Multi Level Security protection.
    SELINUXTYPE=targeted

--------------------

## Master节点 ##

在master节点上生成免密码登录的ssh key

[root@openshift-master ansible] yum install -y ansible
    [root@openshift-master ansible]# ssh-keygen 
    Generating public/private rsa key pair.
    Enter file in which to save the key (/root/.ssh/id_rsa): 
    Created directory '/root/.ssh'.
    Enter passphrase (empty for no passphrase): 
    Enter same passphrase again: 
    Your identification has been saved in /root/.ssh/id_rsa.
    Your public key has been saved in /root/.ssh/id_rsa.pub.
    The key fingerprint is:
    6e:63:84:52:ae:ca:0e:9f:7c:e8:3e:fa:cd:e9:9e:21 root@openshift-master
    The key's randomart image is:
    +--[ RSA 2048]----+
    |                 |
    |                 |
    |      .          |
    |     o .         |
    |    . o S        |
    |     o o         |
    |. E.o   =        |
    | =o*.+ o .       |
    |.*@=B            |
    +-----------------+
    [root@openshift-master ansible]# ssh-copy-id 10.39.47.63
    /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
    /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
    Warning: Permanently added '10.39.47.63' (ECDSA) to the list of known hosts.
    root@10.39.47.63's password: 
    
    Number of key(s) added: 1
    
    Now try logging into the machine, with:   "ssh '10.39.47.63'"
    and check to make sure that only the key(s) you wanted were added.
    
    [root@openshift-master ansible]# ssh-copy-id 10.39.47.64
    /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
    /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
    Warning: Permanently added '10.39.47.64' (ECDSA) to the list of known hosts.
    root@10.39.47.64's password: 
    
    Number of key(s) added: 1
    
    Now try logging into the machine, with:   "ssh '10.39.47.64'"
    and check to make sure that only the key(s) you wanted were added.
    
    [root@openshift-master ansible]# ssh-copy-id 10.39.47.65
    /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
    /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
    Warning: Permanently added '10.39.47.65' (ECDSA) to the list of known hosts.
    root@10.39.47.65's password: 
    
    Number of key(s) added: 1
    
    Now try logging into the machine, with:   "ssh '10.39.47.65'"
    and check to make sure that only the key(s) you wanted were added.

安装ansible-openshift脚本

yum install centos-release-openshift-origin310 -y
    
    yum install openshift-ansible -y

每台都安装docker，并配置docker mirror源

yum install docker  ansible -y

升级ansible(openshift对ansible要求严格)

yum install python-pip python-devel -y
    yum install gcc glbc-devel zlib-devel rpm-build openssl-devel -y
    pip install -U pip
    pip install cryptography -U
    pip install ansible --upgrade

需要升级到ansible 2.7.1  
升级ansible的文档参考[Ansible系列（一） Centos7 安装ansible ，解决cryptography版本低引起的故障问题][Ansible_ Centos7 _ansible _cryptography]

TASK [openshift_control_plane : Wait for all control plane pods to become ready] *****************************************************************************
    FAILED - RETRYING: Wait for all control plane pods to become ready (60 retries left).
    FAILED - RETRYING: Wait for all control plane pods to become ready (59 retries left).
    FAILED - RETRYING: Wait for all control plane pods to become ready (58 retries left).
    FAILED - RETRYING: Wait for all control plane pods to become ready (57 retries left).
    FAILED - RETRYING: Wait for all control plane pods to become ready (56 retries left).
    FAILED - RETRYING: Wait for all control plane pods to become ready (55 retries left).
    FAILED - RETRYING: Wait for all control plane pods to become ready (54 retries left).
    FAILED - RETRYING: Wait for all control plane pods to become ready (53 retries left).
    FAILED - RETRYING: Wait for all control plane pods to become ready (52 retries left).
    FAILED - RETRYING: Wait for all control plane pods to become ready (51 retries left).

这个原因是没有生etcd-ca的证书`/etc/origin/master/master.etcd-ca.crt`

root@openshift-master ~]# docker ps -a | grep api 
    5c20e93ac530        ebcfed580e6b                             "/bin/bash -c '#!/..."   2 minutes ago       Exited (255) 2 minutes ago                       k8s_api_master-api-openshift-master_kube-system_9ca23c5815da8ed1d3dca61d87e1f6ab_77
    7f55a8778021        docker.io/openshift/origin-pod:v3.10.0   "/usr/bin/pod"           5 hours ago         Up 5 hours                                       k8s_POD_master-api-openshift-master_kube-system_9ca23c5815da8ed1d3dca61d87e1f6ab_0
    [root@openshift-master ~]# docker logs 5c20e93ac530
    ...
    Invalid MasterConfig /etc/origin/master/master-config.yaml
      etcdClientInfo.ca: Invalid value: "/etc/origin/master/master.etcd-ca.crt": could not read file: stat /etc/origin/master/master.etcd-ca.crt: no such file or directory

相关的issue  
[https://github.com/openshift/openshift-ansible/issues/8376][https_github.com_openshift_openshift-ansible_issues_8376]  
[https://bugzilla.redhat.com/show\_bug.cgi?id=1638699][https_bugzilla.redhat.com_show_bug.cgi_id_1638699]  
[https://github.com/openshift/openshift-ansible/issues/10368][https_github.com_openshift_openshift-ansible_issues_10368]

配置 ansible 库存文件

[root@openshift-master ~]# cat /etc/ansible/hosts 
    # Create an OSEv3 group that contains the masters, nodes, and etcd groups
    [OSEv3:children]
    masters
    nodes
    etcd
    
    # Set variables common for all OSEv3 hosts
    [OSEv3:vars]
    ansible_ssh_user=root
    #ansible_become=true
    openshift_deployment_type=origin
    # uncomment the following to enable htpasswd authentication; defaults to AllowAllPasswordIdentityProvider
    openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
    
    
    openshift_ca_cert_expire_days=3650
    openshift_node_cert_expire_days=3650
    openshift_master_cert_expire_days=3650
    etcd_ca_default_days=3650
    
    #This variable overrides the default subdomain to use for exposed routes
    openshift_hosted_manage_registry=false
    openshift_disable_check=memory_availability,disk_availability,docker_image_availability
    openshift_enable_service_catalog=false
    template_service_broker_install=false
    ansible_service_broker_install=false
    
    # host group for masters
    [masters]
    openshift-master.example.com
    
    # host group for etcd
    [etcd]
    openshift-master.example.com
    
    # host group for nodes, includes region info
    [nodes]
    openshift-master.example.com openshift_node_group_name='node-config-master-infra'
    openshift-node-64.example.com openshift_node_group_name='node-config-compute'
    openshift-node-65.example.com openshift_node_group_name='node-config-compute'
    openshift-node-66.example.com openshift_node_group_name='node-config-compute'

执行准备工作命令

ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml

安装openshift

ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml

卸载openshift

ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/adhoc/uninstall.yml

--------------------

添加本地主hosts

➜  # cat /etc/hosts
    10.39.47.63 openshift-master

--------------------

创建用户，参考[设置openshift用户登录][openshift]

[root@openshift-master ~]# htpasswd -b /etc/origin/master/htpasswd dev dev
    Adding password for user dev
    [root@openshift-master ~]# cat /etc/origin/master/htpasswd
    dev:$apr1$kKt0XW0C$SjIX59pIkhk7jYB59uXA..
    [root@openshift-master ~]# oc login -u system:admin
    Logged into "https://openshift-master:8443" as "system:admin" using existing credentials.
    
    You have access to the following projects and can switch between them with 'oc project <projectname>':
    
      * default
        kube-public
        kube-system
        management-infra
        openshift
        openshift-infra
        openshift-logging
        openshift-node
        openshift-sdn
        openshift-web-console
    
    Using project "default".
    [root@openshift-master ~]# oc login
    Authentication required for https://openshift-master:8443 (openshift)
    Username: dev
    Password: 
    Login successful.
    
    You don't have any projects. You can try to create a new project, by running
    
        oc new-project <projectname>
    
    [root@openshift-master ~]# oc get pods -n kube-system 
    Error from server (Forbidden): pods is forbidden: User "dev" cannot list pods in the namespace "kube-system": User "dev" cannot list pods in project "kube-system"
    [root@openshift-master ~]# oc logout
    Logged "dev" out on "https://openshift-master:8443"
    [root@openshift-master ~]# oc get pods -n kube-system 
    Error from server (Forbidden): pods is forbidden: User "system:anonymous" cannot list pods in the namespace "kube-system": User "system:anonymous" cannot list pods in project "kube-system"
    [root@openshift-master ~]# oc get nodes
    Error from server (Forbidden): nodes is forbidden: User "system:anonymous" cannot list nodes at the cluster scope: User "system:anonymous" cannot list all nodes in the cluster
    
    [root@openshift-master ~]# oc login -u system:admin
    Logged into "https://openshift-master:8443" as "system:admin" using existing credentials.
    
    You have access to the following projects and can switch between them with 'oc project <projectname>':
    
      * default
        kube-public
        kube-system
        management-infra
        openshift
        openshift-infra
        openshift-logging
        openshift-node
        openshift-sdn
        openshift-web-console
    
    Using project "default".
    [root@openshift-master ~]# oc get nodes
    NAME                STATUS    ROLES          AGE       VERSION
    openshift-master    Ready     infra,master   2h        v1.10.0+b81c8f8
    openshift-node-64   Ready     compute        29m       v1.10.0+b81c8f8
    openshift-node-65   Ready     compute        29m       v1.10.0+b81c8f8
    openshift-node-66   Ready     compute        29m       v1.10.0+b81c8f8

--------------------

访问以下地址确保 OpenShift Origin 可以正确访问：  
访问 [https://openshift-master:8443][https_openshift-master_8443] 用户名和密码就是上一步创建的:dev/dev

![open][]

![10][]  
参考  
[example\_inventories][example_inventories]  
[index][]

[Openshift集群部署.md][Openshift_.md]

[设置openshift用户登录][openshift]

[Ansible_ Centos7 _ansible _cryptography]: https://blog.csdn.net/wu2700222/article/details/80773075
[https_github.com_openshift_openshift-ansible_issues_8376]: https://github.com/openshift/openshift-ansible/issues/8376
[https_bugzilla.redhat.com_show_bug.cgi_id_1638699]: https://bugzilla.redhat.com/show_bug.cgi?id=1638699
[https_github.com_openshift_openshift-ansible_issues_10368]: https://github.com/openshift/openshift-ansible/issues/10368
[openshift]: https://blog.csdn.net/j_ychen/article/details/79900588
[https_openshift-master_8443]: https://openshift-master:8443
[open]: /images/20220422/8728e5f09914437791a16aa753c05ac5.png
[open01]: /images/20220422/acda04dbede84484bcc2bbdca56790c3.png
[10]: /images/20220422/cf1de369d21d491ab778df2445172230.png
[example_inventories]: https://docs.okd.io/latest/install/example_inventories.html
[index]: https://docs.okd.io/latest/install/index.html
[Openshift_.md]: http://blog.leanote.com/post/liukang/Openshift%E8%B0%83%E7%A0%94.md