HBase 集群问题集永久RIT(Region-In-Transition)问题

我会带着你远行 2022-01-10 11:05 796阅读 0赞

HBase 永久RIT(Region-In-Transition)问题：异常关机导致HBase表损坏和丢失，大量Regions 处于Offline状态，无法上线。

**问题1：启动HBase时，HBase Regionserver Web UI，一直停留在The RegionServer is initializing! 界面**

Initializing Master file system (since 10mins, 16sec ago) 
    The RegionServer is initializing!

**问题1：故障排查及解决思路**

查看HBase日志

2018-02-27 17:59:43,114 INFO  [prd-bldb-hdp-name01:60000.activeMasterManager] util.FSUtils: Waiting for dfs to exit safe mode...
    2018-02-27 17:59:53,116 INFO  [prd-bldb-hdp-name01:60000.activeMasterManager] util.FSUtils: Waiting for dfs to exit safe mode...
    2018-02-27 18:00:03,119 INFO  [prd-bldb-hdp-name01:60000.activeMasterManager] util.FSUtils: Waiting for dfs to exit safe mode...
    2018-02-27 18:00:13,121 INFO  [prd-bldb-hdp-name01:60000.activeMasterManager] util.FSUtils: Waiting for dfs to exit safe mode...
    2018-02-27 18:00:23,123 INFO  [prd-bldb-hdp-name01:60000.activeMasterManager] util.FSUtils: Waiting for dfs to exit safe mode...
    2018-02-27 18:00:33,125 INFO  [prd-bldb-hdp-name01:60000.activeMasterManager] util.FSUtils: Waiting for dfs to exit safe mode...
    2018-02-27 18:00:43,128 INFO  [prd-bldb-hdp-name01:60000.activeMasterManager] util.FSUtils: Waiting for dfs to exit safe mode...
    2018-02-27 18:00:53,130 INFO  [prd-bldb-hdp-name01:60000.activeMasterManager] util.FSUtils: Waiting for dfs to exit safe mode...
    2018-02-27 18:01:03,132 INFO  [prd-bldb-hdp-name01:60000.activeMasterManager] util.FSUtils: Waiting for dfs to exit safe mode...
    2018-02-27 18:01:13,135 INFO  [prd-bldb-hdp-name01:60000.activeMasterManager] util.FSUtils: Waiting for dfs to exit safe mode...
    2018-02-27 18:01:23,137 INFO  [prd-bldb-hdp-name01:60000.activeMasterManager] util.FSUtils: Waiting for dfs to exit safe mode...
    2018-02-27 18:01:33,139 INFO  [prd-bldb-hdp-name01:60000.activeMasterManager] util.FSUtils: Waiting for dfs to exit safe mode...
    2018-02-27 18:01:43,141 INFO  [prd-bldb-hdp-name01:60000.activeMasterManager] util.FSUtils: Waiting for dfs to exit safe mode...
    2018-02-27 18:01:53,144 INFO  [prd-bldb-hdp-name01:60000.activeMasterManager] util.FSUtils: Waiting for dfs to exit safe mode...
    2018-02-27 18:02:03,146 INFO  [prd-bldb-hdp-name01:60000.activeMasterManager] util.FSUtils: Waiting for dfs to exit safe mode...
    2018-02-27 18:02:13,148 INFO  [prd-bldb-hdp-name01:60000.activeMasterManager] util.FSUtils: Waiting for dfs to exit safe mode...
    2018-02-27 18:02:23,151 INFO  [prd-bldb-hdp-name01:60000.activeMasterManager] util.FSUtils: Waiting for dfs to exit safe mode...

# 查看hdfs safe mode
    hadoop dfsadmin -safemode get
    
    DEPRECATED: Use of this script to execute hdfs command is deprecated.
    Instead use the hdfs command for it.
    
    Safe mode is ON in namenode02/172.31.132.72:9000
    Safe mode is ON in namenode01/172.31.132.71:9000

# 退出hdfs safe mode
    hadoop dfsadmin -safemode leave
    
    DEPRECATED: Use of this script to execute hdfs command is deprecated.
    Instead use the hdfs command for it.
    
    Safe mode is OFF in namenode02/172.31.132.72:9000
    Safe mode is OFF in namenode01/172.31.132.71:9000

**问题2：Hadoop Namenode Web UI 界面的报错提示，有missing block。**

![781248-20180228182613538-1463247617.png][]

**问题3：HBase  Master Web UI界面的提示，有大量Offline Regions 和RIT(Region-In-Transition)**

![781248-20180228174514739-570493816.png][]

![781248-20180228174528597-1150255389.png][]

**问题2/3：故障排查及解决思路**

# 查看dfs 状态报告
    hadoop dfsadmin -report

![781248-20180228182240992-1843676102.png][]

# 查看损坏文件、当前hdfs的副本数
    hdfs fsck /
    或者
    hadoop fsck -locations

Under replicated blocks　　　　副本数少于指定副本数的block数量  
Blocks with corrupt replicas　　 存在损坏副本的block的数据  
Missing blocks　　　　　　　　丢失block数量

**观察发现，662-53=609 ，正好和问题2中的RIT数量，610 吻合！可以初步判断，HBase Offline Regions 和RIT(Region-In-Transition)问题，是由于Hadoop里存在未修复的block造成。**

之前多次通过hbase hbck 试图修复hbase表无效，原来还是因为hadoop的副本没有完全恢复好。

对以上截图中红框部分的理解是：hdfs上有**662**个block存在副本缺失问题（Under replicated blocks），有**609 **个** **block只剩1-2个副本，有**53**个block副本全部丢失（Missing blocks）。

# **核心修复步骤1：** #

# 更改已经上传文件的副本数，修复Missing blocks
    hadoop fs -setrep -R 3 /

通过该命令，对于存在副本缺失问题（Under replicated blocks）的**662**个block，可以从剩下的1-2个副本，重新生成3个副本，从而找回了丢失的副本。

![781248-20180228183504853-939721043.png][]

Under replicated blocks: 53  
Blocks with corrupt replicas: 0  
Missing blocks: 53  
Missing blocks (with replication factor 1): 0  
Pending deletion blocks: 0

从这里可以看到，现在还有53个缺失的blocks，这53个缺失的block，一个副本都没有。存在1-2个副本的block，已经全部修复！

# **核心修复步骤2：** #

# 删除损坏文件
    hdfs fsck -delete

通过多次运行该命令，对于副本全部丢失（Missing blocks）或损坏的53个block，可以从namenode节点删除元信息和损坏文件。

Under replicated blocks: 33
    Blocks with corrupt replicas: 0
    Missing blocks: 33
    Missing blocks (with replication factor 1): 0
    Pending deletion blocks: 244

此时再查看HBase Master 的Web UI，Offline Regions 和RIT(Region-In-Transition) 已经大大降低。

![781248-20180228190110284-1169539668.png][]

![781248-20180228185855821-1640834993.png][]

过了没多久，发现datanode03的regionserver挂了

![781248-20180228190340961-108319268.png][]

此时通过重启HBase和Hadoop集群，即可消除全部问题

# 在namenode01执行，关闭HBase
    stop-hbase.sh
    
    # 在namnode01执行，关闭Hadoop
    stop-all.sh
    
    # 在namenode01执行
    start-all.sh
    
    # 在namenode01和namenode02节点，分别执行
    start-hbase.sh

![781248-20180228190659132-932193566.png][]

![781248-20180228190020128-1590788243.png][]

### **其他问题：HBase相关故障排查修复思路** ###

HBase的 Regions in Transition 问题
    # 查看hbase中损坏的block
    hbase hbck
    
    # 修复hbase
    hbase hbck -repair
    
    
    The Load Balancer is not enabled which will eventually cause performance degradation in HBase as Regions will not be distributed across all RegionServers. The balancer is only expected to be disabled during rolling upgrade scenarios. 
    
    
    关闭balance,防止在停掉服务后，原先节点上的分片会迁移到其他节点上，到时候在移回来，浪费时间。 
    hbase(main):001:0> balance_switch true
    
    
    2018-02-27 21:14:54,236 INFO  [hbasefsck-pool1-t38] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => e540df791e7fcdc93c118b8055d1c74f, NAME => 'pos_flow_summary_20170713,,1503656523513.e540df791e7fcdc93c118b8055d1c74f.', STARTKEY => '', ENDKEY => ''}
    2018-02-27 21:14:54,236 INFO  [hbasefsck-pool1-t47] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => e59b1015c6fed189cdb9ba8493024563, NAME => 'pos_flow_summary_20180111,,1515768771542.e59b1015c6fed189cdb9ba8493024563.', STARTKEY => '', ENDKEY => ''}
    2018-02-27 21:14:54,241 INFO  [hbasefsck-pool1-t44] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => d22e214e72ff89e87b4df3eebd9603f9, NAME => 'pos_flow_summary_20180112,,1515855181051.d22e214e72ff89e87b4df3eebd9603f9.', STARTKEY => '', ENDKEY => ''}
    2018-02-27 21:14:54,244 INFO  [hbasefsck-pool1-t23] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => e8667191e988db9d65b52cfdb5e83a4d, NAME => 'pos_flow_summary_20170310,,1504229353726.e8667191e988db9d65b52cfdb5e83a4d.', STARTKEY => '', ENDKEY => ''}
    2018-02-27 21:14:54,245 INFO  [hbasefsck-pool1-t45] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => d05b759994d757b8fc857993e3351648, NAME => 'app_point,5000,1510910952310.d05b759994d757b8fc857993e3351648.', STARTKEY => '5000', ENDKEY => '5505|1dcfb8c9a44c4147acc823c2e463d536'}
    
    # 修复 .META表
    hbase hbck -fixMeta
    
    ERROR: Region { meta => pos_flow,2012|dd12dceee69c56f6776154d02e49f840,1518061965154.71eb7d463708010bc2a3f1e96deca135., hdfs => hdfs://namenode01:9000/hbase/data/default/pos_flow/71eb7d463708010bc2a3f1e96deca135, deployed => , replicaId => 0 } not deployed on any region server.
    ERROR: Region { meta => pos_flow_summary_20180115,,1516115199923.70df944adbd82c1422be8f7ee8c24f3e., hdfs => hdfs://namenode01:9000/hbase/data/default/pos_flow_summary_20180115/70df944adbd82c1422be8f7ee8c24f3e, deployed => , replicaId => 0 } not deployed on any region server.
    ERROR: Region { meta => pos_flow,5215|249f79b383f5c144cdd95cd1c29fdec3,1518380260884.67bfa42b4c45ec847c7eb27bbd7d86e5., hdfs => hdfs://namenode01:9000/hbase/data/default/pos_flow/67bfa42b4c45ec847c7eb27bbd7d86e5, deployed => , replicaId => 0 } not deployed on any region server.
    ERROR: Region { meta => pos_flow_summary_20170528,,1504142971183.679bcdecd0335c99d847374db34de31d., hdfs => hdfs://namenode01:9000/hbase/data/default/pos_flow_summary_20170528/679bcdecd0335c99d847374db34de31d, deployed => , replicaId => 0 } not deployed on any region server.
    ERROR: Region { meta => pos_flow,4744|dcf7bccc75f738986e5db100f1f54473,1518489513549.673e899d577f6111b5699b3374ba6adc., hdfs => hdfs://namenode01:9000/hbase/data/default/pos_flow/673e899d577f6111b5699b3374ba6adc, deployed => , replicaId => 0 } not deployed on any region server.
    ERROR: Region { meta => pos_flow,1321|ab83f75ef25bdd0d2ecc363fe1fe0106,1518466350793.66b9622950bba42339f011ac745b080b., hdfs => hdfs://namenode01:9000/hbase/data/default/pos_flow/66b9622950bba42339f011ac745b080b, deployed => , replicaId => 0 } not deployed on any region server.
    ERROR: Region { meta => pos_flow,9449|1ed33683e675c3e9ddbecf4d9bd42183,1518041132081.66b11e69bc62f356b3f81f351b8a6c68., hdfs => hdfs://namenode01:9000/hbase/data/default/pos_flow/66b11e69bc62f356b3f81f351b8a6c68, deployed => , replicaId => 0 } not deployed on any region server.
    ERROR: Region { meta => access_log,1000,1517363823393.65c41f802af180f41af848f1fed8e725., hdfs => hdfs://namenode01:9000/hbase/data/default/access_log/65c41f802af180f41af848f1fed8e725, deployed => , replicaId => 0 } not deployed on any region server.
    
    
    Table pos_flow_summary_20180222 is okay.
        Number of regions: 0
        Deployed on: 
    Table pos_flow_summary_20180223 is okay.
        Number of regions: 0
        Deployed on: 
    Table pos_flow_summary_20180224 is okay.
        Number of regions: 1
        Deployed on:  prd-bldb-hdp-data02,60020,1519734905071
    Table pos_flow_summary_20180225 is okay.
        Number of regions: 1
        Deployed on:  prd-bldb-hdp-data02,60020,1519734905071
    Table pos_flow_summary_20180226 is okay.
        Number of regions: 1
        Deployed on:  prd-bldb-hdp-data02,60020,1519734905071
    Table hbase:namespace is okay.
        Number of regions: 1
        Deployed on:  prd-bldb-hdp-data02,60020,1519734905071
    Table gb_app_active is inconsistent.
        Number of regions: 7
        Deployed on:  prd-bldb-hdp-data01,60020,1519734905393 prd-bldb-hdp-data02,60020,1519734905071 prd-bldb-hdp-data03,60020,1519734905043
    Table app_point is inconsistent.
        Number of regions: 3
        Deployed on:  prd-bldb-hdp-data01,60020,1519734905393 prd-bldb-hdp-data03,60020,1519734905043
    inconsistencies detected.
    Status: INCONSISTENT
    2018-02-27 21:40:59,644 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
    2018-02-27 21:40:59,644 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x161d70981710083
    2018-02-27 21:40:59,646 INFO  [main] zookeeper.ZooKeeper: Session: 0x161d70981710083 closed
    2018-02-27 21:40:59,646 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
    
    
    # 当出现漏洞  
    hbase hbck -fixHdfsHoles 
    
    # 缺少regioninfo  
    hbase hbck -fixHdfsOrphans
    
    # hbase region 引用文件出错
    # Found lingering reference file hdfs：  
    hbase hbck -fixReferenceFiles 
    
    # 修复assignments问题
    hbase hbck -fixAssignments
    
    2018-02-28 14:07:57,814 INFO  [hbasefsck-pool1-t40] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 0836651ac3e23c331ca049e2f333e19f, NAME => 'pos_flow,9139|62b0a7a92cb8c4d25cea82991856334e,1518205798951.0836651ac3e23c331ca049e2f333e19f.', STARTKEY => '9139|62b0a7a92cb8c4d25cea82991856334e', ENDKEY => '9159|441da161eba8d989493f9d2ca2a3e4a2'}
    2018-02-28 14:07:57,814 INFO  [hbasefsck-pool1-t11] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 0e4dbc902294c799db7029df118c61a4, NAME => 'app_point,9000,1510910950800.0e4dbc902294c799db7029df118c61a4.', STARTKEY => '9000', ENDKEY => '9509|05a93'}
    2018-02-28 14:07:57,817 INFO  [hbasefsck-pool1-t10] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 03f6d27f36e4f73e8030cfa6454dfadf, NAME => 'pos_flow_summary_20170913,,1505387404710.03f6d27f36e4f73e8030cfa6454dfadf.', STARTKEY => '', ENDKEY => ''}
    2018-02-28 14:07:57,817 INFO  [hbasefsck-pool1-t35] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 0634fab1259b036a5fbd024fd8da4ba7, NAME => 'pos_flow_summary_20171213,,1513262830799.0634fab1259b036a5fbd024fd8da4ba7.', STARTKEY => '', ENDKEY => ''}
    2018-02-28 14:07:57,818 INFO  [hbasefsck-pool1-t29] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 0666757ecbb89b60a52613cba2dab2f0, NAME => 'pos_flow,4344|8226741aea1eb243789f87abd6e44318,1518152887266.0666757ecbb89b60a52613cba2dab2f0.', STARTKEY => '4344|8226741aea1eb243789f87abd6e44318', ENDKEY => '4404|5fe6f71832f527f173696d3570556461'}
    2018-02-28 14:07:57,819 INFO  [hbasefsck-pool1-t42] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 037d3dff1101418ea3c3868dc9855ecc, NAME => 'pos_flow,7037|c0124cbc08feb233e745aae0d896195a,1518489580825.037d3dff1101418ea3c3868dc9855ecc.', STARTKEY => '7037|c0124cbc08feb233e745aae0d896195a', ENDKEY => '7057|9fae9bddd296a534155c02297532cd28'}
    2018-02-28 14:07:57,820 INFO  [hbasefsck-pool1-t12] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 01ff32a4f85c2de9e3e16c9b6156afa2, NAME => 'pos_flow_summary_20170902,,1504437004424.01ff32a4f85c2de9e3e16c9b6156afa2.', STARTKEY => '', ENDKEY => ''}
    2018-02-28 14:07:57,823 INFO  [hbasefsck-pool1-t41] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 0761d39042e2ec7002dbf291ce23e209, NAME => 'pos_flow_summary_20170725,,1503624161916.0761d39042e2ec7002dbf291ce23e209.', STARTKEY => '', ENDKEY => ''}
      
    
    hbase master stop
    hbase master start
    service hbase-master restart
    
    
    2018-02-28 14:41:39,547 INFO  [main] hdfs.DFSClient: No node available for BP-1225127698-172.31.132.71-1516782893469:blk_1073741999_1175 file=/hbase/data/default/pos_flow_summary_20170304/.tabledesc/.tableinfo.0000000001
    2018-02-28 14:41:39,547 INFO  [main] hdfs.DFSClient: Could not obtain BP-1225127698-172.31.132.71-1516782893469:blk_1073741999_1175 from any node: java.io.IOException: No live nodes contain current block No live nodes contain current block Block locations: Dead nodes: . Will get new block locations from namenode and retry...
    2018-02-28 14:41:39,547 WARN  [main] hdfs.DFSClient: DFS chooseDataNode: got # 1 IOException, will wait for 251.87213750182957 msec.
    2018-02-28 14:41:39,799 INFO  [main] hdfs.DFSClient: No node available for BP-1225127698-172.31.132.71-1516782893469:blk_1073741999_1175 file=/hbase/data/default/pos_flow_summary_20170304/.tabledesc/.tableinfo.0000000001
    2018-02-28 14:41:39,799 INFO  [main] hdfs.DFSClient: Could not obtain BP-1225127698-172.31.132.71-1516782893469:blk_1073741999_1175 from any node: java.io.IOException: No live nodes contain current block No live nodes contain current block Block locations: Dead nodes: . Will get new block locations from namenode and retry...
    2018-02-28 14:41:39,799 WARN  [main] hdfs.DFSClient: DFS chooseDataNode: got # 2 IOException, will wait for 5083.97300871329 msec.
    2018-02-28 14:41:44,883 INFO  [main] hdfs.DFSClient: No node available for BP-1225127698-172.31.132.71-1516782893469:blk_1073741999_1175 file=/hbase/data/default/pos_flow_summary_20170304/.tabledesc/.tableinfo.0000000001
    2018-02-28 14:41:44,883 INFO  [main] hdfs.DFSClient: Could not obtain BP-1225127698-172.31.132.71-1516782893469:blk_1073741999_1175 from any node: java.io.IOException: No live nodes contain current block No live nodes contain current block Block locations: Dead nodes: . Will get new block locations from namenode and retry...
    2018-02-28 14:41:44,883 WARN  [main] hdfs.DFSClient: DFS chooseDataNode: got # 3 IOException, will wait for 9836.488682902267 msec.

转：

[https://www.cnblogs.com/cenliang/p/8485011.html][https_www.cnblogs.com_cenliang_p_8485011.html]

[781248-20180228182613538-1463247617.png]: /images/20220110/36778e2b9e484ae79807c17f3d1ae528.png
[781248-20180228174514739-570493816.png]: /images/20220110/f4d3375235ac44e7b214744bc9c75f66.png
[781248-20180228174528597-1150255389.png]: /images/20220110/d92e6eafa7a44f27a654ca73f84c8d2b.png
[781248-20180228182240992-1843676102.png]: /images/20220110/a8d7a6c7160d4a71abf53f7af42295ca.png
[781248-20180228183504853-939721043.png]: /images/20220110/06bb1d4c7e214e19bcd74e24c04ca6f9.png
[781248-20180228190110284-1169539668.png]: /images/20220110/135a0759f06a4678884d8cc3964995c9.png
[781248-20180228185855821-1640834993.png]: /images/20220110/7db0b44e412142e590cba8ef34a420d9.png
[781248-20180228190340961-108319268.png]: /images/20220110/e24bb09ca52b48a9a156ecbdafed78ed.png
[781248-20180228190659132-932193566.png]: /images/20220110/006d2734db764750b2ac0c68f5822714.png
[781248-20180228190020128-1590788243.png]: /images/20220110/1c783e1b8ca04f1787dcca29d842e140.png
[https_www.cnblogs.com_cenliang_p_8485011.html]: https://www.cnblogs.com/cenliang/p/8485011.html