Centos7下配置DRBD搭建Active/Stanby iSCSi Cluster

布满荆棘的人生 2022-05-21 11:36 212阅读 0赞

# 操作环境 #

Centos 7

DRBDADM\_API\_VERSION=2  
DRBD\_KERNEL\_VERSION=9.0.14  
DRBDADM\_VERSION\_CODE=0x090301

DRBDADM\_VERSION=9.3.1

targetcli version 2.1.fb46

Corosync Cluster Engine, version '2.4.3'

Pacemaker 1.1.18-11.el7\_5.3

crm 3.0.0

网络拓扑图

![70][]

将在drbd-node1以及drbd-node3上，配置drbd/corosync/pacemaker/targetcli/crm服务。

# 操作步骤 #

## 配置DRBD ##

DRBD配置步骤参考[Centos6下drbd9安装与基本配置][Centos6_drbd9]进行配置，注意的是这里的两台vm都配置了双网卡，后面Cluster使用，drbd硬盘资源配置文件如下：

[root@drbd-node1 ~]# vi /etc/drbd.d/scsivol.res 
    resource scsivol {
            on drbd-node1 {
                    device /dev/drbd0;
                    disk   /dev/vdb;
                    address 10.10.200.228:7789;
                    meta-disk internal;
            }
    
            on drbd-node3 {
                    device /dev/drbd0;
                    disk   /dev/vdb;
                    address 10.10.200.226:7789;
                    meta-disk internal;
            }
    }

## 安装Corosync/Pacemaker/Crm ##

1.配置yum源文件如下：

[root@drbd-node1 yum.repos.d]# vi crm.repo 
    [network_ha-clustering_Stable]
    name=Stable High Availability/Clustering packages (CentOS_CentOS-7)
    type=rpm-md
    baseurl=http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/
    gpgcheck=1
    gpgkey=http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/repodata/repomd.xml.key
    enabled=1

[root@drbd-node1 yum.repos.d]# vi drbd.repo 
    [base]
    name=CentOS-$releasever - Base
    mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$ba[...]
    #baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
    exclude=pacemaker* corosync* cluster* drbd* resource-agents
    
    #released updates
    [updates]
    name=CentOS-$releasever - Updates
    mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$ba[...]
    #baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
    exclude=pacemaker* corosync* cluster* drbd* resource-agents

2.安装corosync/pacemaker/crmsh/targetcli

[root@drbd-node1 yum.repos.d]# yum -y install corosync pacemaker crmsh targetcli

在安装过程中，可能会出现有些依赖包因为网络原因无法安装的问题，需要自己去找rpm手动安装。

## 配置Corosync ##

1.配置corosync配置文件如下

[root@drbd-node1 yum.repos.d]# vi /etc/corosync/corosync.conf
    totem {
     version: 2
     secauth: off
     cluster_name: cluster
     transport: udpu
     rrp_mode: passive
    }
    
    nodelist {
     node {
       ring0_addr: 192.168.0.228
       ring1_addr: 10.10.200.228
       nodeid: 1
     }
     node {
       ring0_addr: 192.168.0.226
       ring1_addr: 10.10.200.226
       nodeid: 2
     }
    }
    
    quorum {
     provider: corosync_votequorum
     two_node: 1
    }
    
    logging {
     to_syslog: yes
    }

分别在2个节点上启动corosync/pacemaker

[root@drbd-node1 ~]# systemctl enable corosync
    [root@drbd-node1 ~]# systemctl enable pacemaker
    [root@drbd-node1 ~]# systemctl start corosync
    [root@drbd-node1 ~]# systemctl start pacemaker

查看corosync状态

[root@drbd-node1 ~]# systemctl status corosync
    ?corosync.service - Corosync Cluster Engine
       Loaded: loaded (/usr/lib/systemd/system/corosync.service; enabled; vendor preset: disabled)
       Active: active (running) since Fri 2018-07-06 22:29:14 CST; 2min 22s ago
         Docs: man:corosync
               man:corosync.conf
               man:corosync_overview
      Process: 1114 ExecStart=/usr/share/corosync/corosync start (code=exited, status=0/SUCCESS)
     Main PID: 1279 (corosync)
       CGroup: /system.slice/corosync.service
               忖1279 corosync
    
    Jul 06 22:29:14 drbd-node1 corosync[1279]:  [TOTEM ] adding new UDPU member {10.10.200.228}
    Jul 06 22:29:14 drbd-node1 corosync[1279]:  [TOTEM ] adding new UDPU member {10.10.200.226}
    Jul 06 22:29:14 drbd-node1 corosync[1279]:  [TOTEM ] A new membership (192.168.0.228:3392) was formed. Members joined: 1
    Jul 06 22:29:14 drbd-node1 corosync[1279]:  [TOTEM ] A new membership (192.168.0.226:3396) was formed. Members joined: 2
    Jul 06 22:29:14 drbd-node1 corosync[1279]:  [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2
    Jul 06 22:29:14 drbd-node1 corosync[1279]:  [QUORUM] This node is within the primary component and will provide service.
    Jul 06 22:29:14 drbd-node1 corosync[1279]:  [QUORUM] Members[2]: 2 1
    Jul 06 22:29:14 drbd-node1 corosync[1279]:  [MAIN  ] Completed service synchronization, ready to provide service.
    Jul 06 22:29:14 drbd-node1 corosync[1114]: Starting Corosync Cluster Engine (corosync): [  OK  ]
    Jul 06 22:29:14 drbd-node1 systemd[1]: Started Corosync Cluster Engine.

查看pacemaker状态

[root@drbd-node1 ~]# systemctl status pacemaker
    ?pacemaker.service - Pacemaker High Availability Cluster Manager
       Loaded: loaded (/usr/lib/systemd/system/pacemaker.service; enabled; vendor preset: disabled)
       Active: active (running) since Fri 2018-07-06 22:29:14 CST; 2min 29s ago
         Docs: man:pacemakerd
               http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Pacemaker_Explained/index.html
     Main PID: 1304 (pacemakerd)
       CGroup: /system.slice/pacemaker.service
               忖1304 /usr/sbin/pacemakerd -f
               忖1349 /usr/libexec/pacemaker/cib
               忖1351 /usr/libexec/pacemaker/stonithd
               忖1352 /usr/libexec/pacemaker/lrmd
               忖1353 /usr/libexec/pacemaker/attrd
               忖1354 /usr/libexec/pacemaker/pengine
               忖1355 /usr/libexec/pacemaker/crmd
    
    Jul 06 22:29:21 drbd-node1 crmd[1355]:   notice: Result of start operation for p_drbd_r0 on drbd-node1: 0 (ok)
    Jul 06 22:29:22 drbd-node1 crmd[1355]:   notice: Result of notify operation for p_drbd_r0 on drbd-node1: 0 (ok)
    Jul 06 22:29:22 drbd-node1 crmd[1355]:   notice: Result of notify operation for p_drbd_r0 on drbd-node1: 0 (ok)
    Jul 06 22:29:22 drbd-node1 crmd[1355]:   notice: Result of promote operation for p_drbd_r0 on drbd-node1: 0 (ok)
    Jul 06 22:29:22 drbd-node1 crmd[1355]:   notice: Result of notify operation for p_drbd_r0 on drbd-node1: 0 (ok)
    Jul 06 22:29:22 drbd-node1 crmd[1355]:   notice: Result of start operation for p_iscsi_portblock_on_drbd0 on drbd-node1: 0 (ok)
    Jul 06 22:29:27 drbd-node1 crmd[1355]:   notice: Result of start operation for p_iscsi_ip0 on drbd-node1: 0 (ok)
    Jul 06 22:29:29 drbd-node1 crmd[1355]:   notice: Result of start operation for p_iscsi_target_drbd0 on drbd-node1: 0 (ok)
    Jul 06 22:29:30 drbd-node1 crmd[1355]:   notice: Result of start operation for p_iscsi_lun_drbd0 on drbd-node1: 0 (ok)
    Jul 06 22:29:31 drbd-node1 crmd[1355]:   notice: Result of start operation for p_iscsi_portblock_off_drbd0 on drbd-node1: 0 (ok)

## CRM配置Cluster ##

1.添加以下配置到crm配置中

# crm configure property no-quorum-policy=ignore
    # crm configure property stonith-enabled=false

2.在crm中配置drbd

进入crm配置界面，drbd\_resource后面跟的是drbd资源的名称在drbd.d目录下配置scsivol.res中定义的。

[root@drbd-node1 ~]# crm
    crm(live)# configure
    crm(live)configure# primitive p_drbd_r0 ocf:linbit:drbd \

params drbd_resource="scsivol" \
      op start timeout=240 \
      op promote timeout=90 \
      op demote timeout=90 \
      op stop timeout=100 \
      op monitor interval="29" role="Master" \
      op monitor interval="31" role="Slave"

3.创建于drbd资源scsivol对应的Master/Slave资源

crm(live)configure# ms ms_drbd_r0 p_drbd_r0 \
      meta master-max=1 master-node-max=1 \
      notify=true clone-max=2 clone-node-max=1

4.创建cluster的虚拟ip

crm(live)configure# primitive p_iscsi_ip0 ocf:heartbeat:IPaddr2 \
      params ip="10.10.200.235" cidr_netmask="24" \
      op start timeout=20 \
      op stop timeout=20 \
      op monitor interval="10s"

5.配置iscsi target名称

crm(live)configure# primitive p_iscsi_target_drbd0 ocf:heartbeat:iSCSITarget \
      params iqn="iqn.2017-10.com.example1:edrbd0" \
      implementation=lio-t portals="10.10.200.235:3260" \
      op start timeout=20 \
      op stop timeout=20 \
      op monitor interval=20 timeout=40

6.配置LUN

crm(live)configure# primitive p_iscsi_lun_drbd0 ocf:heartbeat:iSCSILogicalUnit \
      params target_iqn="iqn.2017-10.com.example1:edrbd0" \
      implementation=lio-t lun=0 path="/dev/drbd0" \
      op start timeout=20 \
      op stop timeout=20 \
      op monitor interval=20 timeout=40

7.配置端口的blocking以及unblocking，这将防止在iSCSI目标成功进行故障转移之前iSCSI启动程序接收“拒绝连接”错误。

crm(live)configure# primitive p_iscsi_portblock_on_drbd0 ocf:heartbeat:portblock \
      params ip=10.10.200.235 portno=3260 protocol=tcp action=block \
      op start timeout=20 \
      op stop timeout=20 \
      op monitor timeout=20 interval=20

crm(live)configure# primitive p_iscsi_portblock_off_drbd0 ocf:heartbeat:portblock \
      params ip=10.10.200.235 portno=3260 protocol=tcp action=unblock \
      op start timeout=20 \
      op stop timeout=20 \
      op monitor timeout=20 interval=20

8.创建于iSCSI Target相关联的资源组

crm(live)configure# group g_iscsi_drbd0 \
      p_iscsi_portblock_on_drbd0 \
      p_iscsi_ip0 p_iscsi_target_drbd0 p_iscsi_lun_drbd0 \
      p_iscsi_portblock_off_drbd0

9.最后，我们要确认在资源组以及drbd资源运行在相同的节点上，该节点为Primary节点。

crm(live)configure# colocation cl_g_iscsi_drbd0-with-ms_drbd_r0 \
      inf: g_iscsi_drbd0:Started ms_drbd_r0:Master
    
    crm(live)configure# order o_ms_drbd_r0-before-g_iscsi_drbd0 \
      inf: ms_drbd_r0:promote g_iscsi_drbd0:start

10.提交配置

crm(live)configure# commit

11.查看crm配置状态

crm(live)# status
    Stack: corosync
    Current DC: drbd-node3 (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
    Last updated: Fri Jul  6 22:50:01 2018
    Last change: Fri Jul  6 21:31:38 2018 by root via cibadmin on drbd-node3
    
    2 nodes configured
    7 resources configured
    
    Online: [ drbd-node1 drbd-node3 ]
    
    Full list of resources:
    
     Master/Slave Set: ms_drbd_r0 [p_drbd_r0]
         Masters: [ drbd-node1 ]
         Slaves: [ drbd-node3 ]
     Resource Group: g_iscsi_drbd0
         p_iscsi_portblock_on_drbd0 (ocf::heartbeat:portblock):     Started drbd-node1
         p_iscsi_ip0        (ocf::heartbeat:IPaddr2):       Started drbd-node1
         p_iscsi_target_drbd0       (ocf::heartbeat:iSCSITarget):   Started drbd-node1
         p_iscsi_lun_drbd0  (ocf::heartbeat:iSCSILogicalUnit):      Started drbd-node1
         p_iscsi_portblock_off_drbd0        (ocf::heartbeat:portblock):     Started drbd-node1

# 测试iSCSI连接 #

[root@kvm-node ~]# iscsiadm -m discovery -t st -p 10.10.200.235
    10.10.200.235:3260,1 iqn.2017-10.com.example1:edrbd0
    [root@kvm-node ~]# iscsiadm -m node -T iqn.2017-10.com.example1:edrbd0 -l
    Logging in to [iface: default, target: iqn.2017-10.com.example1:edrbd0, portal: 10.10.200.235,3260] (multiple)
    Login to [iface: default, target: iqn.2017-10.com.example1:edrbd0, portal: 10.10.200.235,3260] successful.
    [root@kvm-node ~]# cat /proc/scsi/scsi 
    Attached devices:
    Host: scsi0 Channel: 02 Id: 00 Lun: 00
      Vendor: DELL     Model: PERC H700        Rev: 2.10
      Type:   Direct-Access                    ANSI  SCSI revision: 05
    Host: scsi0 Channel: 02 Id: 01 Lun: 00
      Vendor: DELL     Model: PERC H700        Rev: 2.10
      Type:   Direct-Access                    ANSI  SCSI revision: 05
    Host: scsi0 Channel: 02 Id: 02 Lun: 00
      Vendor: DELL     Model: PERC H700        Rev: 2.10
      Type:   Direct-Access                    ANSI  SCSI revision: 05
    Host: scsi0 Channel: 02 Id: 03 Lun: 00
      Vendor: DELL     Model: PERC H700        Rev: 2.10
      Type:   Direct-Access                    ANSI  SCSI revision: 05
    Host: scsi0 Channel: 02 Id: 04 Lun: 00
      Vendor: DELL     Model: PERC H700        Rev: 2.10
      Type:   Direct-Access                    ANSI  SCSI revision: 05
    Host: scsi0 Channel: 02 Id: 05 Lun: 00
      Vendor: DELL     Model: PERC H700        Rev: 2.10
      Type:   Direct-Access                    ANSI  SCSI revision: 05
    Host: scsi1 Channel: 00 Id: 00 Lun: 00
      Vendor: TEAC     Model: DVD-ROM DV-28SW  Rev: R.2B
      Type:   CD-ROM                           ANSI  SCSI revision: 05
    Host: scsi3 Channel: 00 Id: 00 Lun: 00
      Vendor: LIO-ORG  Model: p_iscsi_lun_drb  Rev: 4.0 
      Type:   Direct-Access                    ANSI  SCSI revision: 05

Host：scsi3，vendor：LIO-ORG为所配置的iSCSI设备。

# 测试故障切换 #

在进行故障切换前，要确保drbd状态为UptoDa/UpToDA，否则在切换时会不成功：

[root@drbd-node1 ~]# drbd-overview 
    NOTE: drbd-overview will be deprecated soon.
    Please consider using drbdtop.
    
     0:scsivol/0  Connected(2*) Primar/Second UpToDa/UpToDa

在iSCSI Cluster两个节点都正常工作的时候，状态如下，服务都在drbd-node1上

[root@drbd-node1 ~]# crm
    crm(live)# status
    Stack: corosync
    Current DC: drbd-node3 (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
    Last updated: Sat Jul  7 16:40:34 2018
    Last change: Fri Jul  6 21:31:38 2018 by root via cibadmin on drbd-node3
    
    2 nodes configured
    7 resources configured
    
    Online: [ drbd-node1 drbd-node3 ]
    
    Full list of resources:
    
     Master/Slave Set: ms_drbd_r0 [p_drbd_r0]
         Masters: [ drbd-node1 ]
         Slaves: [ drbd-node3 ]
     Resource Group: g_iscsi_drbd0
         p_iscsi_portblock_on_drbd0 (ocf::heartbeat:portblock):     Started drbd-node1
         p_iscsi_ip0        (ocf::heartbeat:IPaddr2):       Started drbd-node1
         p_iscsi_target_drbd0       (ocf::heartbeat:iSCSITarget):   Started drbd-node1
         p_iscsi_lun_drbd0  (ocf::heartbeat:iSCSILogicalUnit):      Started drbd-node1
         p_iscsi_portblock_off_drbd0        (ocf::heartbeat:portblock):     Started drbd-node1

此时模拟drbd-node1宕机，在另一节点上查看iSCSI Cluster状态，所有服务已经切换到drbd-node3上

[root@drbd-node3 ~]# crm
    crm(live)# status
    Stack: corosync
    Current DC: drbd-node3 (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
    Last updated: Sat Jul  7 16:51:39 2018
    Last change: Fri Jul  6 21:31:38 2018 by root via cibadmin on drbd-node3
    
    2 nodes configured
    7 resources configured
    
    Online: [ drbd-node3 ]
    OFFLINE: [ drbd-node1 ]
    
    Full list of resources:
    
     Master/Slave Set: ms_drbd_r0 [p_drbd_r0]
         Masters: [ drbd-node3 ]
         Stopped: [ drbd-node1 ]
     Resource Group: g_iscsi_drbd0
         p_iscsi_portblock_on_drbd0 (ocf::heartbeat:portblock):     Started drbd-node3
         p_iscsi_ip0        (ocf::heartbeat:IPaddr2):       Started drbd-node3
         p_iscsi_target_drbd0       (ocf::heartbeat:iSCSITarget):   Started drbd-node3
         p_iscsi_lun_drbd0  (ocf::heartbeat:iSCSILogicalUnit):      Started drbd-node3
         p_iscsi_portblock_off_drbd0        (ocf::heartbeat:portblock):     Started drbd-node3

试下iSCSI连接，继续上一节中《测试iSCSI连接》中的iSCSI Client，查看session为正常的

[root@kvm-node ~]# iscsiadm -m session
    tcp: [1] 10.10.200.235:3260,1 iqn.2017-10.com.example1:edrbd0 (non-flash)

查看SCSI设备，SCSI 3:0:0:0设备为iSCSI设备

[root@kvm-node ~]# lsscsi 
    [0:2:0:0]    disk    DELL     PERC H700        2.10  /dev/sda 
    [0:2:1:0]    disk    DELL     PERC H700        2.10  /dev/sdb 
    [0:2:2:0]    disk    DELL     PERC H700        2.10  /dev/sdc 
    [0:2:3:0]    disk    DELL     PERC H700        2.10  /dev/sdd 
    [0:2:4:0]    disk    DELL     PERC H700        2.10  /dev/sde 
    [0:2:5:0]    disk    DELL     PERC H700        2.10  /dev/sdf 
    [1:0:0:0]    cd/dvd  TEAC     DVD-ROM DV-28SW  R.2B  /dev/sr0 
    [3:0:0:0]    disk    LIO-ORG  p_iscsi_lun_drb  4.0   /dev/sdg

[70]: /images/20220521/a54720fdb7c14b92b5b9f16aff626fe6.png
[Centos6_drbd9]: https://blog.csdn.net/bobpen/article/details/80747404