MongoDB copydatabase引起数据库hang的原因

浅浅的花香味﹌ 2023-10-09 12:09 50阅读 0赞

开发使用copydatabase导数，找过来说MongoDB hang住了，查询被卡住。  
连上去执行show dbs;果然卡住了，没反应

/apps/svr/mongodb32/bin/mongo --port 27017
    MongoDB shell version: 3.2.22
    connecting to: 127.0.0.1:27017/test
    replset_devops:PRIMARY> show dbs;
    ^C
    do you want to kill the current op(s) on the server? (y/n): y
    2019-09-25T14:19:50.560+0800 I -        [main] Assertion: 13111:field not found, expected type 2
    
    ^C

莫非是MongoDB真挂了？看一下后台日志，发现以下内容：

2019-09-25T14:23:03.421+0800 I -        [conn36993]   Index Build: 177088600/283854248 62%
    2019-09-25T14:23:10.267+0800 I -        [conn36993]   Index Build: 177182700/283854248 62%

在建索引，而且进度百分号还在不停增加，那么说明MongoDB还在运行。  
再连上实例，执行db.currentOp()，看到copydatabase操作的相关信息：

{
    			"desc" : "conn36993",
    			"threadId" : "47404132460288",
    			"connectionId" : 36993,
    			"client" : "xxx.xxx.xxx.xxx:43523",
    			"active" : true,
    			"opid" : 3291065,
    			"secs_running" : 12971,
    			"microsecs_running" : NumberLong("12971335587"),
    			"op" : "command",
    			"ns" : "admin.$cmd",
    			"query" : {
    				"copydb" : 1,
    				"fromhost" : "xxx.xxx.xxx.xxx:27017",
    				"fromdb" : "eagle",
    				"todb" : "eagle"
    			},
    			"msg" : "Index Build Index Build: 186481526/283854248 65%",
    			"progress" : {
    				"done" : 186481526,
    				"total" : 283854248
    			},
    			"numYields" : 3143142,
    			"locks" : {
    				"Global" : "w",
    				"Database" : "W"
    			},
    			"waitingForLock" : false,
    			"lockStats" : {
    				"Global" : {
    					"acquireCount" : {
    						"r" : NumberLong(405483933),
    						"w" : NumberLong(402323773),
    						"W" : NumberLong(3160160)
    					},
    					"acquireWaitCount" : {
    						"W" : NumberLong(3063133)
    					},
    					"timeAcquiringMicros" : {
    						"W" : NumberLong(263444032)
    					}
    				},
    				"Database" : {
    					"acquireCount" : {
    						"w" : NumberLong(402323737),
    						"W" : NumberLong(36)
    					}
    				},
    				"Metadata" : {
    					"acquireCount" : {
    						"w" : NumberLong(402323737)
    					}
    				},
    				"oplog" : {
    					"acquireCount" : {
    						"w" : NumberLong(402323737)
    					}
    				}
    			}
    		},

可以看到copydatabase已经到了创建索引的阶段，且通过连接描述conn36993可以看出后台日志不停刷出的创建索引信息就是copydatabase操作的。  
也可以看到执行show dbs在等待锁：

{
    			"desc" : "conn37885",
    			"threadId" : "47404129302272",
    			"connectionId" : 37885,
    			"client" : "xxx.xxx.xxx.xxx:65352",
    			"active" : true,
    			"opid" : 4264970,
    			"secs_running" : 1180,
    			"microsecs_running" : NumberLong(1180474813),
    			"op" : "command",
    			"ns" : "admin.$cmd",
    			"query" : {
    				"listDatabases" : 1
    			},
    			"numYields" : 0,
    			"locks" : {
    				"Global" : "r",
    				"Database" : "r"
    			},
    			"waitingForLock" : true,
    			"lockStats" : {
    				"Global" : {
    					"acquireCount" : {
    						"r" : NumberLong(2)
    					}
    				},
    				"Database" : {
    					"acquireCount" : {
    						"r" : NumberLong(1)
    					},
    					"acquireWaitCount" : {
    						"r" : NumberLong(1)
    					},
    					"timeAcquiringMicros" : {
    						"r" : NumberLong(1180186301)
    					}
    				}
    			}
    		},

那么为什么连show dbs都会被卡住呢？  
这跟copydatabase有关，copydatabase会使用前台的方式创建索引，而MongoDB使用前台方式创建索引，是需要全局锁的，在创建索引过程中，其余一切操作都会被阻塞。以下是官方说明：  
MongoDB performs foreground builds of indexes on databases copied via copydb. Foreground index builds lock the database and prevent all other operations on that database until the foreground build completes. There may also be a performance impact on other databases while the indexes build.  
By default, creating an index blocks all other operations on a database. When building an index on a collection, the database that holds the collection is unavailable for read or write operations until the index build completes. Any operation that requires a read or write lock on all databases (e.g. listDatabases) will wait for the foreground index build to complete.