Centos7下配置DRBD三节点模式（Cluster）

阳光穿透心脏的1/2处 2022-05-16 06:56 226阅读 0赞

# 操作环境 #

CentOS Linux release 7.4.1708 (Core)

DRBDADM\_BUILDTAG=GIT-hash:\\ ee126652638328b55dc6bff47d07d6161ab768db\\ build\\ by\\ root@drbd-node2\\,\\ 2018-07-30\\ 22:23:07  
DRBDADM\_API\_VERSION=2  
DRBD\_KERNEL\_VERSION\_CODE=0x09000e  
DRBD\_KERNEL\_VERSION=9.0.14  
DRBDADM\_VERSION\_CODE=0x090500  
DRBDADM\_VERSION=9.5.0

# 网络拓扑图 #

与[《Centos7下配置DRBD三节点模式（HA Cluster+Backup Node）》][Centos7_DRBD_HA Cluster_Backup Node]不同，现在的3节点模式下3个节点同属一个cluster，3个节点之间都用Protocol C进行数据传输。而在[《Centos7下配置DRBD三节点模式（HA Cluster+Backup Node）》][Centos7_DRBD_HA Cluster_Backup Node]中，HA Cluster种的2个节点使用Protocol C进行数据传输，HA Cluster与backup节点之间使用Protocol A进行数据传输。

![70][]

# 操作步骤 #

## 安装前准备 ##

参考前面的两篇Blog[《Centos6下drbd9安装与基本配置》][Centos6_drbd9]、[《Centos7下配置DRBD搭建Active/Stanby iSCSi Cluster》][Centos7_DRBD_Active_Stanby iSCSi Cluster]进行准备

## DRBD配置 ##

drbd配置文件编写如下，将该配置文件分别复制到3个节点上，根据drbd connection的计算公式C=N（N-1）/2，C代表连接数，N代表节点数，那个在3节点的情况下，连接数为3，所以在配置文件中有3条连接配置：

[root@drbd-node1 drbd.d]# vi /etc/drbd.d/scsivol.res
    resource scsivol {
            device  /dev/drbd1;
            disk    /dev/vdb1;
            meta-disk internal;
            on drbd-node1 {
                    address 10.10.200.228:7000;
                    node-id 0;
            }
            on drbd-node2 {
                    address 10.10.200.229:7001;
                    node-id 1;
            }
            on drbd-node3 {
                    address 10.10.200.226:7002;
                    node-id 2;
            }
    
    
            connection {
                    host drbd-node1   port 7010;
                    host drbd-node2   port 7001;
            }
            connection {
                    host drbd-node1   port 7020;
                    host drbd-node3   port 7002;
            }
            connection {
                    host drbd-node2   port 7012;
                    host drbd-node3   port 7021;
            }
    }

除了可以使用上面这个方式进行编写配置文件，还可以使用下述方法，通过connection-mesh就不用一条条的编写连接信息了。

如果服务器上配置了多个网口，可以指定网口用于drbd节点之间传输数据

resource scsivol {
            device  /dev/drbd1;
            disk    /dev/vdb1;
            meta-disk internal;
            on drbd-node1 {
                    address 10.10.200.228:7000;
                    node-id 0;
            }
            on drbd-node2 {
                    address 10.10.200.229:7001;
                    node-id 1;
            }
            on drbd-node3 {
                    address 10.10.200.226:7002;
                    node-id 2;
            }
    
    
            connection {
                    host drbd-node1 192.168.0.228  port 7010;
                    host drbd-node2 192.168.0.229  port 7001;
            }
            connection {
                    host drbd-node1 192.168.1.228  port 7020;
                    host drbd-node3 192.168.1.226  port 7002;
            }
            connection {
                    host drbd-node2 192.168.2.229  port 7012;
                    host drbd-node3 192.168.2.226  port 7021;
            }
    }

## DRBD Resource初始化 ##

分别在3个节点上执行以下命令：

[root@drbd-node2 ~]# drbdadm create-md scsivol
    md_offset 107374178304
    al_offset 107374145536
    bm_offset 107367591936
    
    Found xfs filesystem
         9765552 kB data area apparently used
       104851164 kB left usable by current configuration
    
    Even though it looks like this would place the new meta data into
    unused space, you still need to confirm, as this is only a guess.
    
    Do you want to proceed?
    [need to type 'yes' to confirm] yes
    
    initializing activity log
    initializing bitmap (6400 KB) to all zero
    Writing meta data...
    New drbd meta data block successfully created.
    success
    [root@drbd-node2 ~]# drbdadm up scsivol

在3个节点上分别初始化drbd resource后，选择drbd-node1作为主节点

[root@drbd-node1 drbd.d]# drbdadm primary scsivol --froce

## 查看DRBD状态 ##

[root@drbd-node1 drbd.d]# drbdadm status scsivol
    scsivol role:Primary
      disk:UpToDate
      drbd-node2 role:Secondary
        replication:SyncSource peer-disk:Inconsistent done:22.22
      drbd-node3 role:Secondary
        replication:SyncSource peer-disk:Inconsistent done:34.05

数据同步完成后，状态信息如下：

[root@drbd-node1 drbd.d]# drbdadm status
    scsivol role:Primary
      disk:UpToDate
      drbd-node2 role:Secondary
        peer-disk:UpToDate
      drbd-node3 role:Secondary
        peer-disk:UpToDate

# 失效切换测试 #

先在主节点上挂载drbd设备，创建文件系统，并吸入10G的数据

[root@drbd-node1 drbd.d]# mkfs.xfs /dev/drbd1 -f
    meta-data=/dev/drbd1             isize=512    agcount=4, agsize=6553198 blks
             =                       sectsz=512   attr=2, projid32bit=1
             =                       crc=1        finobt=0, sparse=0
    data     =                       bsize=4096   blocks=26212791, imaxpct=25
             =                       sunit=0      swidth=0 blks
    naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
    log      =internal log           bsize=4096   blocks=12799, version=2
             =                       sectsz=512   sunit=0 blks, lazy-count=1
    realtime =none                   extsz=4096   blocks=0, rtextents=0
    [root@drbd-node1 drbd.d]# mount /dev/drbd1 /mnt/
    [root@drbd-node1 drbd.d]# cd /mnt/
    [root@drbd-node1 mnt]# touch a
    [root@drbd-node1 mnt]# dd if=/dev/zero of=a bs=1M count=10240
    10240+0 records in
    10240+0 records out
    10737418240 bytes (11 GB) copied, 258.881 s, 41.5 MB/s

数据写入后，关闭drbd-node1，模拟主节点宕机。这是我们查看drbd-node2节点状态，drbd-node2以及drbd-node3都为secondary状态

[root@drbd-node2 /]# drbdadm status
    scsivol role:Secondary
      disk:UpToDate
      drbd-node1 connection:Connecting
      drbd-node3 role:Secondary
        peer-disk:UpToDate

在drbd-node2上，挂载drbd设备至/mnt目录，数据正常。

[root@drbd-node2 /]# mount /dev/drbd1 /mnt/
    [root@drbd-node2 /]# ls /mnt/
    a
    [root@drbd-node2 /]# df -h
    Filesystem               Size  Used Avail Use% Mounted on
    /dev/mapper/centos-root   41G  2.1G   39G   6% /
    devtmpfs                 486M     0  486M   0% /dev
    tmpfs                    1.0G   54M  971M   6% /dev/shm
    tmpfs                    497M  6.6M  490M   2% /run
    tmpfs                    497M     0  497M   0% /sys/fs/cgroup
    /dev/vda1               1014M  127M  888M  13% /boot
    tmpfs                    100M     0  100M   0% /run/user/0
    /dev/drbd1               100G  9.1G   91G  10% /mnt

此时，再来查看drbd cluster状态，在挂载drbd设备后，drbd-node2状态变更为primary状态。

[root@drbd-node2 /]# drbdadm status
    scsivol role:Primary
      disk:UpToDate
      drbd-node1 connection:Connecting
      drbd-node3 role:Secondary
        peer-disk:UpToDate

再次模拟drbd-node2节点宕机，在drbd-node3节点上查看drbd信息如下

[root@drbd-node3 ~]# drbdadm status
    scsivol role:Secondary
      disk:UpToDate
      drbd-node1 connection:Connecting
      drbd-node2 connection:Connecting

挂载drbd设备至/mnt目录，查看数据

[root@drbd-node3 ~]# mount /dev/drbd1 /mnt/
    [root@drbd-node3 ~]# ls /mnt/
    a
    [root@drbd-node3 ~]# df -h
    Filesystem               Size  Used Avail Use% Mounted on
    /dev/mapper/centos-root   44G  2.8G   42G   7% /
    devtmpfs                 3.9G     0  3.9G   0% /dev
    tmpfs                    3.9G     0  3.9G   0% /dev/shm
    tmpfs                    3.9G  8.4M  3.9G   1% /run
    tmpfs                    3.9G     0  3.9G   0% /sys/fs/cgroup
    /dev/vda1               1014M  145M  870M  15% /boot
    tmpfs                    783M     0  783M   0% /run/user/0
    /dev/drbd1               100G  9.1G   91G  10% /mnt

查看drbd状态，此时drbd-node3为主节点

[root@drbd-node3 ~]# drbdadm status
    scsivol role:Primary
      disk:UpToDate
      drbd-node1 connection:Connecting
      drbd-node2 connection:Connecting

[Centos7_DRBD_HA Cluster_Backup Node]: https://blog.csdn.net/bobpen/article/details/81286633
[70]: /images/20220516/1725a68828194fee89e7d7586b184a19.png
[Centos6_drbd9]: https://blog.csdn.net/bobpen/article/details/80747404
[Centos7_DRBD_Active_Stanby iSCSi Cluster]: https://blog.csdn.net/bobpen/article/details/80940099