mongodb 分片集群安装搭建测试

偏执的太偏执、 2023-02-14 10:45 8阅读 0赞

关于什么是mongodb分片集群，这些这里就不介绍，网上有很多的说明。

由于是测试资源有限，于是就把所有的节点都搭建在一台服务器上面了，实际的生产环境最好config/mongos/shard都分开安装到不同的机器上，这里搭建了12个节点的分片集群；将端口区分开，创建3个mongs节点，3节点的config-server副本集，3个2节点的shard副本集。使用版本：4.0.18

<table> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:103.7pt;"> 主机 </td> 
 <td style="vertical-align:top;width:103.7pt;"> 端口 </td> 
 <td style="vertical-align:top;width:103.7pt;"> 角色 </td> 
 <td style="vertical-align:top;width:103.7pt;"> 副本集/分片类型 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:103.7pt;"> 172.100.1.35 </td> 
 <td style="vertical-align:top;width:103.7pt;"> 27017,37017,47017 </td> 
 <td style="vertical-align:top;width:103.7pt;"> mongos </td> 
 <td style="vertical-align:top;width:103.7pt;"> — </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:103.7pt;"> 172.100.1.35 </td> 
 <td style="vertical-align:top;width:103.7pt;"> 27019,37019,47019 </td> 
 <td style="vertical-align:top;width:103.7pt;"> config server </td> 
 <td style="vertical-align:top;width:103.7pt;"> cs0/configsvr </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:103.7pt;"> 172.100.1.35 </td> 
 <td style="vertical-align:top;width:103.7pt;"> 27018,37018,47018 </td> 
 <td style="vertical-align:top;width:103.7pt;"> shardA </td> 
 <td style="vertical-align:top;width:103.7pt;"> shardA/shardsvr </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:103.7pt;"> 172.100.1.35 </td> 
 <td style="vertical-align:top;width:103.7pt;"> 27016,37016,47016 </td> 
 <td style="vertical-align:top;width:103.7pt;"> shardB </td> 
 <td style="vertical-align:top;width:103.7pt;"> shardB/shardsvr </td> 
 </tr> 
 </tbody> 
</table>

话不多说直接开搞:

下面就是此次测试环境需要的mongodb节点

[root@cpe-172-100-1-35 jsunicom]# ll
    total 0
    drwxr-xr-x  6 root  root  203 Jun  2 16:10 mongodb_config_27019
    drwxr-xr-x  6 root  root  203 Jun  2 11:28 mongodb_config_37019
    drwxr-xr-x  6 root  root  203 Jun  2 11:28 mongodb_config_47019
    drwxr-xr-x  6 root  root  198 Jun  2 14:31 mongodb_mongos1
    drwxr-xr-x  6 root  root  198 Jun  2 14:31 mongodb_mongos2
    drwxr-xr-x  6 root  root  198 Jun  2 14:30 mongodb_mongos3
    drwxr-xr-x  6 root  root  199 Jun  2 14:38 mongodb_shardA_1
    drwxr-xr-x  6 root  root  199 Jun  2 14:38 mongodb_shardA_2
    drwxr-xr-x  6 root  root  199 Jun  2 14:02 mongodb_shardA_3
    drwxr-xr-x  6 root  root  199 Jun  2 14:22 mongodb_shardB_1
    drwxr-xr-x  6 root  root  199 Jun  2 14:22 mongodb_shardB_2
    drwxr-xr-x  6 root  root  199 Jun  2 14:22 mongodb_shardB_3

**首先对configsvr进行配置**

分别在 mongodb\_config\_27019，mongodb\_config\_37019，mongodb\_config\_47019下创建对应的配置文件，这里我的配置文件分别为:mogondb\_config\_27019.conf，mogondb\_config\_37019.conf，mogondb\_config\_47019.conf

编辑配置文件，这里就以添加mongodb\_config\_27019几点为例，添加以下内容:

[root@cpe-172-100-1-35 mongodb_config_27019]# cat mogondb_config_27019.conf
    systemLog:
       traceAllExceptions: false
       path: /webapp/jsunicom/mongodb_config_27019/logs/mongod.log
       logAppend: true
       destination: file
       timeStampFormat: ctime
    processManagement:
       fork: true
       pidFilePath: /webapp/jsunicom/mongodb_config_27019/tmp/mongod.pid
    net:
       port: 27019
       bindIp: 172.100.1.35
       maxIncomingConnections: 20000
    storage:
       dbPath: /webapp/jsunicom/mongodb_config_27019/data
       directoryPerDB: true
       wiredTiger:
          engineConfig:
             cacheSizeGB: 1
             directoryForIndexes: true
    operationProfiling:
       slowOpThresholdMs: 1000
    replication:
       oplogSizeMB: 2048
       replSetName: cs0
    sharding:
       clusterRole: configsvr

注意：其余2个节点除了端口号以及路径和上面的不一样，其余完全一致。

配置好之后，分别启动这3个config节点

[root@cpe-172-100-1-35 ~]#  /webapp/jsunicom/mongodb_config_27019/bin/mongod -f /webapp/jsunicom/mongodb_config_27019/mogondb_config_27019.conf
    
    [root@cpe-172-100-1-35 ~]#  /webapp/jsunicom/mongodb_config_37019/bin/mongod -f /webapp/jsunicom/mongodb_config_37019/mogondb_config_37019.conf
    
    [root@cpe-172-100-1-35 ~]#  /webapp/jsunicom/mongodb_config_47019/bin/mongod -f /webapp/jsunicom/mongodb_config_47019/mogondb_config_47019.conf

启动之后，随便登录上面3个config节点中的一个进行初始化

[root@cpe-172-100-1-35 ~]# /webapp/jsunicom/mongodb_config_27019/bin/mongo --host 172.100.1.35 --port 27019
    
    执行以下命令
    use admin
    
    rs.initiate(
      {
        _id: "cs0",
        configsvr: true,
        members: [
          { _id : 0, host : "172.100.1.35:27019" },
          { _id : 1, host : "172.100.1.35:37019" },
          { _id : 2, host : "172.100.1.35:47019" }
        ]
      }
    )
    
    执行成功之后，一定要用以下命令来查看初始化是否正常
    rs.status()
    
    其中health：1 代表初始化成功

到此config配置结束

**下面对shardA,shardB配置：**

**shardA配置:**

分别在mongodb\_shardA\_1，mongodb\_shardA\_2，mongodb\_shardA\_3下创建对应的配置文件

我这里分别为：mongodb\_shardA\_1.conf，mongodb\_shardA\_2.conf ，mongodb\_shardA\_3.conf

编辑配置文件，这里还是以mongodb\_shardA\_1.conf为例，添加如下内容：

[root@cpe-172-100-1-35 mongodb_shardA_1]# cat mongodb_shardA_1.conf 
    systemLog:
       traceAllExceptions: false
       path: /webapp/jsunicom/mongodb_shardA_1/logs/mongod.log
       logAppend: true
       destination: file
       timeStampFormat: ctime
    processManagement:
       fork: true
       pidFilePath: /webapp/jsunicom/mongodb_shardA_1/tmp/mongod.pid
    net:
       port: 27018
       bindIp: 172.100.1.35
       maxIncomingConnections: 20000
    storage:
       dbPath: /webapp/jsunicom/mongodb_shardA_1/data
       directoryPerDB: true
       wiredTiger:
          engineConfig:
             cacheSizeGB: 1
             directoryForIndexes: true
    operationProfiling:
       slowOpThresholdMs: 1000
    replication:
       oplogSizeMB: 3072
       replSetName: shardA
    sharding:
       clusterRole: shardsvr
    setParameter:
       connPoolMaxShardedInUseConnsPerHost: 100
       shardedConnPoolIdleTimeoutMinutes: 10
       connPoolMaxInUseConnsPerHost: 100
       globalConnPoolIdleTimeoutMinutes: 10
       maxIndexBuildMemoryUsageMegabytes: 2048

备注：其余2个节点配置除了端口号以及路径不一致以外，其余完全和上面一致

**shardB配置:**

其实shardB配置和shardA完全一样，只要注意端口以及路径就行了

同样分别在mongodb\_shardB\_1，mongodb\_shardB\_2，mongodb\_shardB\_3下创建对应的配置文件

我这里分别为：mongodb\_shardB\_1.conf，mongodb\_shardB\_2.conf ，mongodb\_shardB\_3.conf

编辑配置文件，这里还是以mongodb\_shardB\_1.conf为例，添加如下内容：

[root@cpe-172-100-1-35 mongodb_shardB_1]# cat mongodb_shardB_1.conf 
    systemLog:
       traceAllExceptions: false
       path: /webapp/jsunicom/mongodb_shardB_1/logs/mongod.log
       logAppend: true
       destination: file
       timeStampFormat: ctime
    processManagement:
       fork: true
       pidFilePath: /webapp/jsunicom/mongodb_shardB_1/tmp/mongod.pid
    net:
       port: 27016
       bindIp: 172.100.1.35
       maxIncomingConnections: 20000
    storage:
       dbPath: /webapp/jsunicom/mongodb_shardB_1/data
       directoryPerDB: true
       wiredTiger:
          engineConfig:
             cacheSizeGB: 1
             directoryForIndexes: true
    operationProfiling:
       slowOpThresholdMs: 1000
    replication:
       oplogSizeMB: 3072
       replSetName: shardB
    sharding:
       clusterRole: shardsvr
    setParameter:
       connPoolMaxShardedInUseConnsPerHost: 100
       shardedConnPoolIdleTimeoutMinutes: 10
       connPoolMaxInUseConnsPerHost: 100
       globalConnPoolIdleTimeoutMinutes: 10
       maxIndexBuildMemoryUsageMegabytes: 2048

备注：其余2个节点配置除了端口号以及路径不一致以外，其余完全和上面一致

配置好之后，那就把上面的shardA3个节点以及shardB三个节点分别起起来。

[root@cpe-172-100-1-35 ~]# /webapp/jsunicom/mongodb_shardA_1/bin/mongod -f /webapp/jsunicom/mongodb_shardA_1/mongodb_shardA_1.conf
    
    
    [root@cpe-172-100-1-35 ~]# /webapp/jsunicom/mongodb_shardA_2/bin/mongod -f /webapp/jsunicom/mongodb_shardA_2/mongodb_shardA_2.conf
    
    
    [root@cpe-172-100-1-35 ~]# /webapp/jsunicom/mongodb_shardA_3/bin/mongod -f /webapp/jsunicom/mongodb_shardA_3/mongodb_shardA_3.conf
    
    
    [root@cpe-172-100-1-35 ~]# /webapp/jsunicom/mongodb_shardB_1/bin/mongod -f /webapp/jsunicom/mongodb_shardB_1/mongodb_shardB_1.conf
    
    
    [root@cpe-172-100-1-35 ~]# /webapp/jsunicom/mongodb_shardB_2/bin/mongod -f /webapp/jsunicom/mongodb_shardB_2/mongodb_shardB_2.conf
    
    
    [root@cpe-172-100-1-35 ~]# /webapp/jsunicom/mongodb_shardB_3/bin/mongod -f /webapp/jsunicom/mongodb_shardB_3/mongodb_shardB_3.conf

然后分别初始化话replicat 配置

shardA:

随便登录shardA3个节点中的其中一个节点即可：

[root@cpe-172-100-1-35 ~]# /webapp/jsunicom/mongodb_config_27019/bin/mongo --host 172.100.1.35 --port 27018
    
    登录之后执行以下命令
    use admin
    
    rs.initiate(
      {
        _id : "shardA",
        members: [
          { _id : 0, host : "172.100.1.35:27018" },
          { _id : 1, host : "172.100.1.35:37018" },
          { _id : 2, host : "172.100.1.35:47018",arbiterOnly: true }
        ]
      }
    )
    
    同样也要用下面命令去查看初始化是否成功
    rs.status()
    
    
    其中health：1 代表初始化成功

shardB:

随便登录shardB3个节点中的其中一个节点即可：

[root@cpe-172-100-1-35 ~]# /webapp/jsunicom/mongodb_config_27019/bin/mongo --host 172.100.1.35 --port 27016
    
    登录成功以后执行以下命令进行初始化
    use admin
    
    rs.initiate(
      {
        _id : "shardB",
        members: [
          { _id : 0, host : "172.100.1.35:27016" },
          { _id : 1, host : "172.100.1.35:37016" },
          { _id : 2, host : "172.100.1.35:47016",arbiterOnly: true }
        ]
      }
    )
    
    同样需要使用rs.status()命令查看是否执行成功
    
    其中health ：1 代表初始化成功

至此shardA，shardB分片配置到此结束

**最后配置mongos**

这里mongos也做了3个节点，

分别在mongodb\_mongos1，mongodb\_mongos2，mongodb\_mongos3下配置mongodb\_mongos1.conf，mongodb\_mongos2.conf，mongodb\_mongos3.conf

编辑配置文件，这里同样以mongodb\_mongos1.conf为例，添加如下内容:

[root@cpe-172-100-1-35 mongodb_mongos1]# cat mongodb_mongos1.conf 
    systemLog:
       traceAllExceptions: false
       path: /webapp/jsunicom/mongodb_mongos1/logs/mongos.log
       logAppend: true
       destination: file
       timeStampFormat: ctime
    processManagement:
       fork: true
       pidFilePath: /webapp/jsunicom/mongodb_mongos1/tmp/mongos.pid
    net:
       port: 27017
       bindIp: 172.100.1.35
       maxIncomingConnections: 20000
    operationProfiling:
       slowOpThresholdMs: 1000
    replication:
       localPingThresholdMs: 300
    sharding:
       configDB: cs0/172.100.1.35:27019,172.100.1.35:37019,172.100.1.35:47019
    setParameter:
       connPoolMaxShardedInUseConnsPerHost: 100
       shardedConnPoolIdleTimeoutMinutes: 10
       connPoolMaxInUseConnsPerHost: 100
       globalConnPoolIdleTimeoutMinutes: 10

备注：其余2个节点配置除了端口号以及路径和上面不一样之外，其余完全一致

上面配置文件配置好之后，分别启动这3个节点：

[root@cpe-172-100-1-35 ~]# /webapp/jsunicom/mongodb_mongos1/bin/mongos -f /webapp/jsunicom/mongodb_mongos1/mongodb_mongos1.conf
    
    [root@cpe-172-100-1-35 ~]# /webapp/jsunicom/mongodb_mongos2/bin/mongos -f /webapp/jsunicom/mongodb_mongos2/mongodb_mongos2.conf
    
    [root@cpe-172-100-1-35 ~]# /webapp/jsunicom/mongodb_mongos3/bin/mongos -f /webapp/jsunicom/mongodb_mongos3/mongodb_mongos3.conf

随便登录上面3个节点中的其中一个，添加replicat 到sharding中

[root@cpe-172-100-1-35 ~]# /webapp/jsunicom/mongodb_config_27019/bin/mongo --host 172.100.1.35 --port 27017
    
    登录之后执行如下命令:
    use admin
    
    sh.addShard("shardA/172.100.1.35:27018,172.100.1.35:37018,172.100.1.35:47018")
    
    sh.addShard("shardA/172.100.1.35:27018,172.100.1.35:37016,172.100.1.35:47016")
    
    
    执行完之后使用rs.status()命令查看状态
    mongos> sh.status()
    --- Sharding Status --- 
      sharding version: {
            "_id" : 1,
            "minCompatibleVersion" : 5,
            "currentVersion" : 6,
            "clusterId" : ObjectId("5ed5e536856e004008317884")
      }
      shards:
            {  "_id" : "shardA",  "host" : "shardA/172.100.1.35:27018,172.100.1.35:37018",  "state" : 1 }
            {  "_id" : "shardB",  "host" : "shardB/172.100.1.35:27016,172.100.1.35:37016",  "state" : 1 }
      active mongoses:
            "4.0.18" : 3
      autosplit:
            Currently enabled: yes
      balancer:
            Currently enabled:  yes
            Currently running:  no
            Failed balancer rounds in last 5 attempts:  0
            Migration Results for the last 24 hours: 
                    No recent migrations
      databases:
            {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                    config.system.sessions
                            shard key: { "_id" : 1 }
                            unique: false
                            balancing: true
                            chunks:
                                    shardB  1
                            { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : shardB Timestamp(1, 0)

到此所有的配置都结束了，下面来进行测试以下:

下面所有的操作都是在mongos其中一个节点上执行的

启动sharding库:  
sh.enableSharding("yuhuashi")

创建分片集合：  
sh.shardCollection("yuhuashi.user",\{"name" : 1\}) \#ranged 分片  
sh.shardCollection("yuhuashi.user", \{ "task\_id" : "hashed" \} ) \#hash分片

我这里使用了hashed分片

mongos> sh.shardCollection("yuhuashi.user", { "task_id" : "hashed" } )
    {
            "collectionsharded" : "yuhuashi.user",
            "collectionUUID" : UUID("43d3cce7-0225-4dcc-8b7e-aecc7b3932b6"),
            "ok" : 1,
            "operationTime" : Timestamp(1591081873, 27),
            "$clusterTime" : {
                    "clusterTime" : Timestamp(1591081873, 27),
                    "signature" : {
                            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                            "keyId" : NumberLong(0)
                    }
            }
    }

可以使用sh.status()来查看状态

mongos> sh.status()
    --- Sharding Status --- 
      sharding version: {
            "_id" : 1,
            "minCompatibleVersion" : 5,
            "currentVersion" : 6,
            "clusterId" : ObjectId("5ed5e536856e004008317884")
      }
      shards:
            {  "_id" : "shardA",  "host" : "shardA/172.100.1.35:27018,172.100.1.35:37018",  "state" : 1 }
            {  "_id" : "shardB",  "host" : "shardB/172.100.1.35:27016,172.100.1.35:37016",  "state" : 1 }
      active mongoses:
            "4.0.18" : 3
      autosplit:
            Currently enabled: yes
      balancer:
            Currently enabled:  yes
            Currently running:  no
            Failed balancer rounds in last 5 attempts:  0
            Migration Results for the last 24 hours: 
                    No recent migrations
      databases:
            {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                    config.system.sessions
                            shard key: { "_id" : 1 }
                            unique: false
                            balancing: true
                            chunks:
                                    shardB  1
                            { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : shardB Timestamp(1, 0) 
            {  "_id" : "yuhuashi",  "primary" : "shardA",  "partitioned" : true,  "version" : {  "uuid" : UUID("fb60172b-49e9-4d3a-8052-1e8780b0ff52"),  "lastMod" : 1 } }
                    yuhuashi.user
                            shard key: { "task_id" : "hashed" }
                            unique: false
                            balancing: true
                            chunks:
                                    shardA  2
                                    shardB  2
                            { "task_id" : { "$minKey" : 1 } } -->> { "task_id" : NumberLong("-4611686018427387902") } on : shardA Timestamp(1, 0) 
                            { "task_id" : NumberLong("-4611686018427387902") } -->> { "task_id" : NumberLong(0) } on : shardA Timestamp(1, 1) 
                            { "task_id" : NumberLong(0) } -->> { "task_id" : NumberLong("4611686018427387902") } on : shardB Timestamp(1, 2) 
                            { "task_id" : NumberLong("4611686018427387902") } -->> { "task_id" : { "$maxKey" : 1 } } on : shardB Timestamp(1, 3)

下面来测试插入测试数据看以下

mongos> use yuhuashi
 switched to db yuhuashi
 mongos> show tables;
 user
 mongos> db.user.find().count()
 0
 mongos> for(i=1;i<=10000;i++){db.user.insert({"task_id":i,"name":"shiyu"+i,"age":i})}
 WriteResult({ "nInserted" : 1 })

分别登录shardA和shardB分片查看

shardA:PRIMARY> use yuhuashi
    switched to db yuhuashi
    shardA:PRIMARY> db.user.find().count()
    7521
    shardA:PRIMARY> 
    
    
    shardB:PRIMARY> use yuhuashi
    switched to db yuhuashi
    shardB:PRIMARY> db.user.find().count()
    2479
    shardB:PRIMARY>

GAME OVER!