Centos7下配置DRBD Cluster扩展节点

一时失言乱红尘 2022-05-15 02:04 209阅读 0赞

# 操作环境 #

CentOS Linux release 7.4.1708 (Core)

DRBDADM\_BUILDTAG=GIT-hash:\\ ee126652638328b55dc6bff47d07d6161ab768db\\ build\\ by\\ root@drbd-node2\\,\\ 2018-07-30\\ 22:23:07  
DRBDADM\_API\_VERSION=2  
DRBD\_KERNEL\_VERSION\_CODE=0x09000e  
DRBD\_KERNEL\_VERSION=9.0.14  
DRBDADM\_VERSION\_CODE=0x090500  
DRBDADM\_VERSION=9.5.0

## 网络拓扑图 ##

原DRBD Cluster中有4个节点，需要将其扩展为5个节点，新添加节点为drbd-node5

![70][]

# 操作步骤 #

## 原DRBD Cluster信息 ##

[root@drbd-node1 drbd.d]# drbdadm status
    scsivol role:Secondary
      disk:UpToDate
      drbd-node2 role:Secondary
        peer-disk:UpToDate
      drbd-node3 role:Secondary
        peer-disk:UpToDate
      drbd-node4 role:Secondary
        peer-disk:UpToDate

## 扩展DRBD Cluster ##

### 修改DRBD 配置文件 ###

原配置文件如下：

[root@drbd-node1 drbd.d]# vi scsivol.res
    resource scsivol {
            device  /dev/drbd1;
            disk    /dev/vdb1;
            meta-disk internal;
            on drbd-node1 {
                    address 10.10.200.228:7000;
                    node-id 0;
            }
            on drbd-node2 {
                    address 10.10.200.229:7001;
                    node-id 1;
            }
            on drbd-node3 {
                    address 10.10.200.226:7002;
                    node-id 2;
            }
            on drbd-node4 {
                    address 10.10.200.230:7003;
                    node-id 3;
            }
            connection-mesh {
                hosts drbd-node1 drbd-node2 drbd-node3 drbd-node4;
                     net {
                              use-rle no;
                    }
            }
    }

添加新节点drbd-node5信息后，而后将配置文件分发到DRBD Cluster中的所有节点，配置文件如下：

[root@drbd-node1 drbd.d]# vi scsivol.res
    resource scsivol {
            device  /dev/drbd1;
            disk    /dev/vdb1;
            meta-disk internal;
            on drbd-node1 {
                    address 10.10.200.228:7000;
                    node-id 0;
            }
            on drbd-node2 {
                    address 10.10.200.229:7001;
                    node-id 1;
            }
            on drbd-node3 {
                    address 10.10.200.226:7002;
                    node-id 2;
            }
            on drbd-node4 {
                    address 10.10.200.230:7003;
                    node-id 3;
            }
            on drbd-node5 {
                    address 10.10.200.231:7004;
                    node-id 4;
            }
            connection-mesh {
                hosts drbd-node1 drbd-node2 drbd-node3 drbd-node4 drbd-node5;
                     net {
                              use-rle no;
                    }
            }
    }

## 配置DRBD Cluster元数据 ##

在扩展drbd cluster节点之前，需要在元数据配置信息里面更改最大节点数，否则无法添加新节点，会提示错误信息如下：

drbd-node1 kernel: drbd scsivol/0 drbd1 drbd-node5: Not enough free bitmap slots

1.在所有节点上，导出drbd cluster元数据信息，其中scsivol为drbd resource名称，在drbd配置文件中已定义。

[root@drbd-node1 drbd.d]# drbdmeta --force $(drbdadm sh-minor scsivol) v09 $(drbdadm sh-ll-dev scsivol) internal dump-md > scsivol.metadata

元数据配置信息如下，需要修改max-peers，以及添加bitmap\[n\]

[root@drbd-node1 drbd.d]# vi scsivol.metadata 
    # DRBD meta data dump
    # 2018-08-23 19:24:06 +0800 [1535023446]
    # drbd-node1> drbdmeta --force 1 v09 /dev/vdb1 internal dump-md
    #
    
    version "v09";
    
    This_is_an_unclean_meta_data_dump._Don't_trust_the_bitmap.
    # You should "apply-al" first, if you plan to restore this.
    
    max-peers 3;
    # md_size_sect 3912
    # md_offset 21474832384
    # al_offset 21474799616
    # bm_offset 21472833536
    ......
    # al-extents 1237;
    la-size-sect 20969528;
    bm-byte-per-bit 4096;
    device-uuid 0xBE4D681B6892E302;
    la-peer-max-bio-size 1048576;
    al-stripes 1;
    al-stripe-size-4k 8;
    # bm-bytes 983040;
    bitmap[0] {
       # at 0kB
        40960 times 0x0000000000000000;
    }
    bitmap[1] {
       # at 0kB
        40960 times 0x0000000000000000;
    }
    bitmap[2] {
       # at 0kB
        4096 times 0xFFFFFFFFFFFFFFFF;
       # at 3145728kB
        36864 times 0x0000000000000000;
    }
    # bits-set 262144;

修改后的元数据配置信息如下：

[root@drbd-node1 drbd.d]# vi scsivol.metadata 
    # DRBD meta data dump
    # 2018-08-23 19:24:06 +0800 [1535023446]
    # drbd-node1> drbdmeta --force 1 v09 /dev/vdb1 internal dump-md
    #
    
    version "v09";
    
    This_is_an_unclean_meta_data_dump._Don't_trust_the_bitmap.
    # You should "apply-al" first, if you plan to restore this.
    
    max-peers 4;
    # md_size_sect 3912
    # md_offset 21474832384
    # al_offset 21474799616
    # bm_offset 21472833536
    ......
    # bm-bytes 983040;
    bitmap[0] {
       # at 0kB
        40960 times 0x0000000000000000;
    }
    bitmap[1] {
       # at 0kB
        40960 times 0x0000000000000000;
    }
    bitmap[2] {
       # at 0kB
        4096 times 0xFFFFFFFFFFFFFFFF;
       # at 3145728kB
        36864 times 0x0000000000000000;
    }
    bitmap[3] {
       # at 0kB
        40960 times 0x0000000000000000;
    }
    # bits-set 262144;

2.导入元数据

在导入元数据前，需要先将scsivol down掉

[root@drbd-node1 drbd.d]# drbdadm down scsivol

在所有节点上，再次导入元数据

[root@drbd-node1 drbd.d]# drbdmeta --force $(drbdadm sh-minor scsivol) v09 $(drbdadm sh-ll-dev scsivol) internal restore-md scsivol.metadata 
    
    Valid meta-data in place, overwrite?
    *** confirmation forced via --force option ***
    reinitializing
    Successfully restored meta data

## 初始化新节点 ##

在新节点drbd-node5上，初始化drbd resource

[root@drbd-node5 drbd.d]# drbdadm create-md scsivol
    You want me to create a v09 style flexible-size internal meta data block.
    There appears to be a v09 flexible-size internal meta data block
    already in place on /dev/vdb1 at byte offset 21475233792
    
    Do you really want to overwrite the existing meta-data?
    [need to type 'yes' to confirm] yes
    
    md_offset 21475233792
    al_offset 21475201024
    bm_offset 21472575488
    
    Found some data
    
     ==> This might destroy existing data! <==
    
    Do you want to proceed?
    [need to type 'yes' to confirm] yes
    
    initializing activity log
    initializing bitmap (2564 KB) to all zero
    ioctl(/dev/vdb1, BLKZEROOUT, [21472575488, 2625536]) failed: Inappropriate ioctl for device
    Using slow(er) fallback.
    100%
    Writing meta data...
    New drbd meta data block successfully created.

启动drbd resource

[root@drbd-node5 ~]# drbdadm up scsivol

等待一段时间的同步后，查看drbd cluster状态

[root@drbd-node1 ~]# drbdadm status
    scsivol role:Secondary
      disk:UpToDate
      drbd-node2 role:Secondary
        peer-disk:UpToDate
      drbd-node3 role:Secondary
        peer-disk:UpToDate
      drbd-node4 role:Secondary
        peer-disk:UpToDate
      drbd-node5 role:Secondary
        peer-disk:UpToDate

发现所有的节点都是secondary状态，重新设置drbd-node1为主节点

[root@drbd-node1 ~]# drbdadm primary scsivol
    [root@drbd-node1 ~]# drbdadm status
    scsivol role:Primary
      disk:UpToDate
      drbd-node2 role:Secondary
        peer-disk:UpToDate
      drbd-node3 role:Secondary
        peer-disk:UpToDate
      drbd-node4 role:Secondary
        peer-disk:UpToDate
      drbd-node5 role:Secondary
        peer-disk:UpToDate

# 总结 #

drbd节点扩展这块做的不够好，在配置过程中，时长会碰到奇怪的问题，相同的步骤不是每次都能成功。

[70]: /images/20220515/aafbd0fa438c4c30a1aef53a54b8b2c0.png