[hbase案例]从冷备份的hdfs数据中恢复到原来的hbase表

系统管理员 2022-05-28 21:26 422阅读 0赞

【hadoop版本：2.7.1 ；hbase版本：1.1.2】

之前我已经在关闭整个hbase集群时，通过hadoop distcp对本集群的hbase指定表数据做了个全量冷备份，那么这次要从已有的全量备份文件中恢复hbase数据，需要三步：

**Step 1.** 使用distcp命令将所备份的文件拷贝回原来所在HBase目录就可以

**Step 2.** 确保权限正确，例如有关的HBase目录和文件的所有者是hbase用户

**Step 3.** 开启hbase集群后，修复元数据、分配表数据到有关的region server

\# 之前冷备份的数据存放在本地hdfs集群的/backup/bk\_20180122目录下

\[hbase@goodMaster-1 ~\]$ hdfs dfs -ls /backup/bk\_20180122

Found 1 items

drwxr-xr-x - hdfs hdfs 0 2018-01-22 21:14 /backup/bk\_20180122/data

\[hbase@goodMaster-1 ~\]$ hdfs dfs -ls /backup/bk\_20180122/data/hbase\_data/all\_info

Found 11 items

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk\_20180122/data/hbase\_data/all\_info/chenshan1\_test

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk\_20180122/data/hbase\_data/all\_info/export\_import\_table\_test

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk\_20180122/data/hbase\_data/all\_info/urls\_30

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk\_20180122/data/hbase\_data/all\_info/urls\_7

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk\_20180122/data/hbase\_data/all\_info/test\_1

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk\_20180122/data/hbase\_data/all\_info/test\_2

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk\_20180122/data/hbase\_data/all\_info/monthly\_statistics

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk\_20180122/data/hbase\_data/all\_info/high\_class\_users

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk\_20180122/data/hbase\_data/all\_info/week\_jobs\_data

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk\_20180122/data/hbase\_data/all\_info/ios\_users

drwxr-xr-x - hdfs hdfs 0 2018-01-22 22:12 /backup/bk\_20180122/data/hbase\_data/all\_info/users\_info

\[hbase@goodMaster-1 ~\]$ hdfs dfs -ls /backup/bk\_20180122/data/hbase\_data/all\_info/users\_info | wc -l

\[hbase@goodMaster-1 ~\]$ hdfs dfs -du -h /backup/bk\_20180122/data/hbase\_data/all\_info/ | grep --color users\_info

9.8 G /backup/bk\_20180122/data/hbase\_data/all\_info/users\_info

**Step 1.**

\[hbase@goodMaster-1 ~\]$ hadoop distcp hdfs://hdfs-ha/backup/bk\_20180122/data/hbase\_data/all\_info/users\_info hdfs://hdfs-ha/apps/hbase/data/hbase\_data/all\_info > ~/distcprecover.log 2>&1 &

\[1\] 6928

**Step 2.**

\# 检查权限，必须保证该表对应hdfs目录下的文件owner都是hbase用户。若不是则hadoop fs -chown -R hbase:hdfs /apps/hbase

\[hbase@goodMaster-1 ~\]$ hdfs dfs -ls /apps/hbase | awk '\{print $3,$4\}' | grep hbase | head -n 2

hbase hdfs

\[hbase@goodMaster-1 ~\]$ hdfs dfs -ls /apps/hbase | awk '\{print $3\}' | grep hdfs \# 说明owner都是hbase用户而不是hdfs用户

\[hbase@goodMaster-1 ~\]$ less ~/fix\_log

**Step 3.**

\# 确认权限没问题就在Active Hmaster上修复元数据，还要把表数据重新分配到相关区域(会自动按照hbase的相关机制进行分配，具体如何分配的请上网搜)

\[hbase@goodMaster-1 ~\]$ hbase hbck -fixMeta -fixAssignments "all\_info:users\_info" > ~/fix\_log 2>&1 &

\[1\] 11747

\[hbase@kgoodMaster-1 ~\]$ jobs

\[1\]+ **Running** hbase hbck -fixMeta -fixAssignments "all\_info:users\_info" > ~/fix\_log 2>&1 &

\[hbase@goodMaster-1 ~\]$ jobs

\[1\]+ **Done** hbase hbck -fixMeta -fixAssignments "all\_info:users\_info" > ~/fix\_log 2>&1

\[hbase@goodMaster-1 ~\]$ less ~/fix\_log \# 当出现Deployed on \[your\_region\_server\] 相关字样时，说明该表已成功部署到region server

\# 去hbase shell验证一下

hbase(main):003:0> scan "all\_info:users\_info" , LIMIT => 3

ROW COLUMN+CELL

0 column=cf:confidence, timestamp=1502878819524, value=1.0

0 column=cf:label, timestamp=1502878819524, value=\\xE4\\xB8\\xAD\\xE5\\x9B\\xBD,\\xE6\\xB1\\x9F\\xE8\\x8B\\x8F,\\xE6\\xB7\\xAE\\xE5\\xAE\\x89,,\\xE7\\x94\\xB5\\xE4\\xBF\\xA1

00000000000000000000000000000000 column=cf:confidence, timestamp=10000, value=0.5

00000000000000000000000000000000 column=cf:label, timestamp=10000, value=\\xE4\\xB8\\xAD\\xE5\\x9B\\xBD,\\xE5\\xB9\\xBF\\xE4\\xB8\\x9C,\\xE4\\xBD\\x9B\\xE5\\xB1\\xB1,,\\xE7\\x94\\xB5\\xE4\\xBF\\xA1

00000051992B3C6B48DF65CFCD00F570 column=cf:confidence, timestamp=1502895476079, value=1.0

00000051992B3C6B48DF65CFCD00F570 column=cf:label, timestamp=1502895476079, value=\\xE4\\xB8\\xAD\\xE5\\x9B\\xBD,\\xE5\\xB9\\xBF\\xE4\\xB8\\x9C,\\xE7\\x8F\\xA0\\xE6\\xB5\\xB7,,\\xE7\\x94\\xB5\\xE4\\xBF\\xA1

3 row(s) in 0.1810 seconds

hbase(main):009:0> count "all\_info:users\_info", INTERVAL => 5000000

Current count: 5000000, row: 12CB6A9E9CCD42674CAEBF68CB59B644

Current count: 10000000, row: 2597BB29A9B94ED48D86287575917903

Current count: 15000000, row: 38615F1F568A52017E65EDA1639CFF55

Current count: 20000000, row: 4B2C059D1DA69C5BA780E431F7A88F22

Current count: 25000000, row: 5DF4FAD9697292B6DED164C14D6D19A3

Current count: 30000000, row: 70BBDC41E4C5757FB8D26618F22698F2

Current count: 35000000, row: 8385DC4053D07A1910B815B2D445F6E1

Current count: 40000000, row: 964C61B9B42B41FEDCE8F5073638B483

Current count: 45000000, row: A9126076D2B29BDF5BE335A4B884FA69

Current count: 50000000, row: BBDBF49A21E278D46DB774490C3EABCE