kylin2.3作业将结果写入hbase时报错TableNotFoundException

心已赠人 2022-05-25 06:51 217阅读 0赞

执行kylin作业报错，这个作业是要把运行结果写入到hbase的表里的，但是再写入hbase过程中报错hbase中没有表 'kylin\_metadata'。  
错误日志摘要——  
2018-05-07 20:28:46,137 WARN \[main\] util.HeapMemorySizeUtil:55 : hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size  
2018-05-07 20:28:46,395 DEBUG \[main\] hbase.HBaseConnection:307 : HTable 'kylin\_metadata' already exists  
Exception in thread "main" java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin\_metadata@hbase  
at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:96)  
at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:108)  
at org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:94)  
at org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41)  
Caused by: java.lang.reflect.InvocationTargetException  
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)  
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)  
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)  
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)  
at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:90)  
... 3 more  
Caused by: org.apache.hadoop.hbase.TableNotFoundException: kylin\_metadata  
at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:578)  
at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:551)  
at org.apache.kylin.storage.hbase.HBaseConnection.createHTableIfNeeded(HBaseConnection.java:308)  
at org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:111)  
at org.apache.kylin.storage.hbase.HBaseResourceStore.<init>(HBaseResourceStore.java:92)  
... 8 more  
  
大概说的是default名称空间下的表不存在：TableNotFoundException: kylin\_metadata 这个kylin\_metadata的作用：存放了所有的cube元数据，包括cube描述和实例、项目信息、倒排索引描述、作业、表和字典。Kylin使用HBase而不是正常的文件系统来存放这些元数据。如果检查你的配置文件（kylin.properties）会发现这一行：  
\#\# The metadata store in hbase  
kylin.metadata.url=kylin\_metadata@hbase  
意思是说元数据将保存在hbase中的 kylin\_metadata 表里，可以在hbase shell里扫描来查看里面的数据  
  
进到hbase shell查看default下面只有test\_yhl这张表，而没有kylin\_metadata，于是尝试新建一张表kylin\_metadata  
hbase(main):005:0> list  
TABLE  
nice\_users:users\_like  
nice\_users:china\_users  
nice\_users:china\_users0426  
nice\_users:weekly\_stat  
test\_yhl  
5 row(s) in 0.0120 seconds  
=> \["nice\_users:users\_like", "nice\_users:china\_users", "nice\_users:china\_users0426", "nice\_users:weekly\_stat", "test\_yhl"\]  
  
【尝试解决1】建表时报错  
hbase(main):003:0> create 'default:kylin\_metadata', "f1"  
ERROR: Table already exists: default:kylin\_metadata!  
  
【尝试解决2】使用hbck检查该表  
\[hbase@kmr-a20125dd-gn-a05044c6-master-1-001 root\]$ hbase hbck -details "default:kylin\_metadata" > ~/kylin\_metadata\_DETAILS.log 2>&1  
检查结果kylin\_metadata\_DETAILS.log摘要：  
Number of regions in transition: 11  
KYLIN\_JVKSFWWFT7,,1524046155657.8f50d43fff533da39e6b6bce8e2536b7. state=FAILED\_CLOSE, ts=Mon May 07 17:02:14 CST 2018 (13106s ago), server=null  
kylin\_metadata,,1524037727389.e770d1e6528b6154f29a5314e6c8733d. state=FAILED\_CLOSE, ts=Mon May 07 17:02:14 CST 2018 (13106s ago), server=null  
KYLIN\_8FCVO3VMZ5,,1525312564354.367e62a2489d180428f6c23f0092bdf6. state=FAILED\_CLOSE, ts=Mon May 07 17:02:14 CST 2018 (13106s ago), server=null  
KYLIN\_Z806Q81TI0,,1524723913265.05c0ffe0e9f64b117e4ab40171782cae. state=FAILED\_CLOSE, ts=Mon May 07 17:02:13 CST 2018 (13107s ago), server=null  
KYLIN\_LJW3WT8H3O,,1525663588151.1a17bb4d1f47ef6f146655b9a4a5bbed. state=FAILED\_CLOSE, ts=Mon May 07 17:02:13 CST 2018 (13107s ago), server=null  
KYLIN\_9YN4WTKNI0,,1525656643173.40353cc56028c2bf85fad6a01626ddb3. state=FAILED\_CLOSE, ts=Mon May 07 17:02:14 CST 2018 (13106s ago), server=null  
KYLIN\_1JTYCRWRQT,,1525667723449.da8073440b414fc140cb7f5fa0341ae5. state=FAILED\_CLOSE, ts=Mon May 07 17:02:14 CST 2018 (13106s ago), server=null  
KYLIN\_VYVN08HI3O,,1524038585742.b0eaa0d6f000ebaa2090703d3e53ed89. state=FAILED\_CLOSE, ts=Mon May 07 17:02:13 CST 2018 (13107s ago), server=null  
KYLIN\_80TYFBXYHA,,1525660830618.2aafa56d517997e0aee45a516ac990f1. state=FAILED\_CLOSE, ts=Mon May 07 17:02:14 CST 2018 (13106s ago), server=null  
KYLIN\_875PS7V2PN,,1524715341451.c96d67c4643b819513312fc83a7549a4. state=FAILED\_CLOSE, ts=Mon May 07 17:02:13 CST 2018 (13107s ago), server=null  
KYLIN\_8SV2D5XCMB,,1525657404680.fabe589ab2df00f73c4ba370ceda0a15. state=FAILED\_CLOSE, ts=Mon May 07 17:02:14 CST 2018 (13106s ago), server=null  
2018-05-07 20:40:41,600 WARN \[hbasefsck-pool1-t5\] util.HBaseFsck: No HDFS region dir found: \{ meta => kylin\_metadata,,1524037727389.e770d1e6528b6154f29a5314e6c8733d., hdfs => null, deployed => , replicaId => 0 \} meta=\{ENCODED => e770d1e6528b6154f29a5314e6c8733d, NAME => 'kylin\_metadata,,1524037727389.e770d1e6528b6154f29a5314e6c8733d.', STARTKEY => '', ENDKEY => ''\}  
2018-05-07 20:40:41,842 INFO \[main-EventThread\] zookeeper.ClientCnxn: EventThread shut down  
ERROR: Region \{ meta => kylin\_metadata,,1524037727389.e770d1e6528b6154f29a5314e6c8733d., hdfs => null, deployed => , replicaId => 0 \} found in META, but not in HDFS or deployed on any region server.  
  
kylin\_metadata\_DETAILS.log 里面的Number of regions in transition: 11 这些还在分配过程(RIT)中的区域，其实已经作废了的，我不久前就手动删除了，不想要他们了，但是没删干净。  
在web UI也查看到Regions in Transition栏目下的表 kylin\_metadata 赫然在列，而且状态state=FAILED\_CLOSE，RIT time(ms) 持续时间已经8924398 即持续了2.47个小时！  
  
【解决】清除掉zk节点中处于RIT状态的表  
\[zk: localhost:2181(CONNECTED) 0\] ls /hbase-unsecure/region-in-transition  
\[b0eaa0d6f000ebaa2090703d3e53ed89, 40353cc56028c2bf85fad6a01626ddb3, 2aafa56d517997e0aee45a516ac990f1, 367e62a2489d180428f6c23f0092bdf6, e770d1e6528b6154f29a5314e6c8733d, 1a17bb4d1f47ef6f146655b9a4a5bbed, 8f50d43fff533da39e6b6bce8e2536b7, 05c0ffe0e9f64b117e4ab40171782cae, c96d67c4643b819513312fc83a7549a4, fabe589ab2df00f73c4ba370ceda0a15, da8073440b414fc140cb7f5fa0341ae5\]  
\[zk: localhost:2181(CONNECTED) 1\] rmr /hbase-unsecure/region-in-transition/b0eaa0d6f000ebaa2090703d3e53ed89  
\[zk: localhost:2181(CONNECTED) 2\] rmr /hbase-unsecure/region-in-transition/40353cc56028c2bf85fad6a01626ddb3  
\[zk: localhost:2181(CONNECTED) 3\] rmr /hbase-unsecure/region-in-transition/2aafa56d517997e0aee45a516ac990f1  
\[zk: localhost:2181(CONNECTED) 4\] rmr /hbase-unsecure/region-in-transition/367e62a2489d180428f6c23f0092bdf6  
\[zk: localhost:2181(CONNECTED) 5\] rmr /hbase-unsecure/region-in-transition/e770d1e6528b6154f29a5314e6c8733d  
\[zk: localhost:2181(CONNECTED) 6\] rmr /hbase-unsecure/region-in-transition/1a17bb4d1f47ef6f146655b9a4a5bbed  
\[zk: localhost:2181(CONNECTED) 7\] rmr /hbase-unsecure/region-in-transition/8f50d43fff533da39e6b6bce8e2536b7  
\[zk: localhost:2181(CONNECTED) 8\] rmr /hbase-unsecure/region-in-transition/05c0ffe0e9f64b117e4ab40171782cae  
\[zk: localhost:2181(CONNECTED) 9\] rmr /hbase-unsecure/region-in-transition/c96d67c4643b819513312fc83a7549a4  
\[zk: localhost:2181(CONNECTED) 10\] rmr /hbase-unsecure/region-in-transition/fabe589ab2df00f73c4ba370ceda0a15  
\[zk: localhost:2181(CONNECTED) 11\] rmr /hbase-unsecure/region-in-transition/da8073440b414fc140cb7f5fa0341ae5  
\[zk: localhost:2181(CONNECTED) 12\] ls /hbase-unsecure/region-in-transition  
\[\]  
  
再重启两个hmaster，结果：仍然在hbase的web UI中看到 regions in transition的那些表，那些表都是我之前删掉了的。怀疑是元数据还在。  
进入hbase shell中检查刚才被删掉的表是否还在 habse:meta 表(元数据表)中有记录，结果：果然在元数据表中还在。  
那么解决办法是，把 hbase:meta 表中包含那些处于RIT状态表的记录逐条删除即可。  
hbase(main):015:0> scan "hbase:meta", \{STARTROW => 'KYLIN\_JVKSFWWFT7', LIMIT => 3\}  
ROW COLUMN+CELL  
KYLIN\_JVKSFWWFT7,,1524046155657.8f50d43fff533da39e6b6bce8e2 column=info:regioninfo, timestamp=1524046156037, value=\{ENCODED => 8f50d43fff533da39e6b6bce8e2536b7, NAME => 'KYLIN\_JVKSFWWFT7,,1524046155657.8f50d43fff533da39e6b6bce8e2536b7.  
536b7.', STARTKEY => '', ENDKEY => ''\}  
KYLIN\_JVKSFWWFT7,,1524046155657.8f50d43fff533da39e6b6bce8e2 column=info:seqnumDuringOpen, timestamp=1525663825507, value=\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x1E  
536b7.  
KYLIN\_JVKSFWWFT7,,1524046155657.8f50d43fff533da39e6b6bce8e2 column=info:server, timestamp=1525663825507, value=kmr-a20125dd-gn-a05044c6-core-1-008.ksc.com:16020  
536b7.  
KYLIN\_JVKSFWWFT7,,1524046155657.8f50d43fff533da39e6b6bce8e2 column=info:serverstartcode, timestamp=1525663825507, value=1525662768456  
536b7.  
KYLIN\_LJW3WT8H3O,,1525663588151.1a17bb4d1f47ef6f146655b9a4a column=info:regioninfo, timestamp=1525663588877, value=\{ENCODED => 1a17bb4d1f47ef6f146655b9a4a5bbed, NAME => 'KYLIN\_LJW3WT8H3O,,1525663588151.1a17bb4d1f47ef6f146655b9a4a5bbed.  
5bbed.', STARTKEY => '', ENDKEY => ''\}  
KYLIN\_LJW3WT8H3O,,1525663588151.1a17bb4d1f47ef6f146655b9a4a column=info:seqnumDuringOpen, timestamp=1525663827974, value=\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x04  
5bbed.  
KYLIN\_LJW3WT8H3O,,1525663588151.1a17bb4d1f47ef6f146655b9a4a column=info:server, timestamp=1525663827974, value=kmr-a20125dd-gn-a05044c6-core-1-008.ksc.com:16020  
5bbed.  
KYLIN\_LJW3WT8H3O,,1525663588151.1a17bb4d1f47ef6f146655b9a4a column=info:serverstartcode, timestamp=1525663827974, value=1525662768456  
5bbed.  
KYLIN\_VYVN08HI3O,,1524038585742.b0eaa0d6f000ebaa2090703d3e5 column=info:regioninfo, timestamp=1524038586204, value=\{ENCODED => b0eaa0d6f000ebaa2090703d3e53ed89, NAME => 'KYLIN\_VYVN08HI3O,,1524038585742.b0eaa0d6f000ebaa2090703d3e53ed89.  
3ed89.', STARTKEY => '', ENDKEY => ''\}  
KYLIN\_VYVN08HI3O,,1524038585742.b0eaa0d6f000ebaa2090703d3e5 column=info:seqnumDuringOpen, timestamp=1525663680965, value=\\x00\\x00\\x00\\x00\\x00\\x00\\x00$  
3ed89.  
KYLIN\_VYVN08HI3O,,1524038585742.b0eaa0d6f000ebaa2090703d3e5 column=info:server, timestamp=1525663680965, value=kmr-a20125dd-gn-a05044c6-core-1-008.ksc.com:16020  
3ed89.  
KYLIN\_VYVN08HI3O,,1524038585742.b0eaa0d6f000ebaa2090703d3e5 column=info:serverstartcode, timestamp=1525663680965, value=1525662768456  
3ed89.  
3 row(s) in 0.0200 seconds  
  
hbase(main):024:0> delete 'hbase:meta','KYLIN\_JVKSFWWFT7,,1524046155657.8f50d43fff533da39e6b6bce8e2536b7.','info:regioninfo'  
hbase(main):032:0> delete 'hbase:meta','KYLIN\_JVKSFWWFT7,,1524046155657.8f50d43fff533da39e6b6bce8e2536b7.','info:seqnumDuringOpen'  
hbase(main):034:0> delete 'hbase:meta','KYLIN\_JVKSFWWFT7,,1524046155657.8f50d43fff533da39e6b6bce8e2536b7.','info:server'  
hbase(main):035:0> delete 'hbase:meta','KYLIN\_JVKSFWWFT7,,1524046155657.8f50d43fff533da39e6b6bce8e2536b7.','info:serverstartcode'  
hbase(main):040:0> delete 'hbase:meta','KYLIN\_LJW3WT8H3O,,1525663588151.1a17bb4d1f47ef6f146655b9a4a5bbed.','info:regioninfo'  
hbase(main):041:0> delete 'hbase:meta','KYLIN\_LJW3WT8H3O,,1525663588151.1a17bb4d1f47ef6f146655b9a4a5bbed.','info:seqnumDuringOpen'  
hbase(main):042:0> delete 'hbase:meta','KYLIN\_LJW3WT8H3O,,1525663588151.1a17bb4d1f47ef6f146655b9a4a5bbed.','info:server'  
hbase(main):043:0> delete 'hbase:meta','KYLIN\_LJW3WT8H3O,,1525663588151.1a17bb4d1f47ef6f146655b9a4a5bbed.','info:serverstartcode'  
hbase(main):048:0> delete 'hbase:meta','KYLIN\_VYVN08HI3O,,1524038585742.b0eaa0d6f000ebaa2090703d3e53ed89.','info:regioninfo'  
hbase(main):049:0> delete 'hbase:meta','KYLIN\_VYVN08HI3O,,1524038585742.b0eaa0d6f000ebaa2090703d3e53ed89.','info:seqnumDuringOpen'  
hbase(main):050:0> delete 'hbase:meta','KYLIN\_VYVN08HI3O,,1524038585742.b0eaa0d6f000ebaa2090703d3e53ed89.','info:server'  
hbase(main):051:0> delete 'hbase:meta','KYLIN\_VYVN08HI3O,,1524038585742.b0eaa0d6f000ebaa2090703d3e53ed89.','info:serverstartcode'  
  
\# 再次到zk相关节点，删除掉 region-in-transition 下面的所有节点  
\[zk: localhost:2181(CONNECTED) 0\] ls /hbase-unsecure/region-in-transition  
\[b0eaa0d6f000ebaa2090703d3e53ed89, 40353cc56028c2bf85fad6a01626ddb3, 2aafa56d517997e0aee45a516ac990f1, 367e62a2489d180428f6c23f0092bdf6, e770d1e6528b6154f29a5314e6c8733d, 1a17bb4d1f47ef6f146655b9a4a5bbed, 8f50d43fff533da39e6b6bce8e2536b7, 05c0ffe0e9f64b117e4ab40171782cae, c96d67c4643b819513312fc83a7549a4, fabe589ab2df00f73c4ba370ceda0a15, da8073440b414fc140cb7f5fa0341ae5\]  
\[zk: localhost:2181(CONNECTED) 1\] rmr /hbase-unsecure/region-in-transition/b0eaa0d6f000ebaa2090703d3e53ed89  
\[zk: localhost:2181(CONNECTED) 2\] rmr /hbase-unsecure/region-in-transition/40353cc56028c2bf85fad6a01626ddb3  
\[zk: localhost:2181(CONNECTED) 3\] rmr /hbase-unsecure/region-in-transition/2aafa56d517997e0aee45a516ac990f1  
\[zk: localhost:2181(CONNECTED) 4\] rmr /hbase-unsecure/region-in-transition/367e62a2489d180428f6c23f0092bdf6  
\[zk: localhost:2181(CONNECTED) 5\] rmr /hbase-unsecure/region-in-transition/e770d1e6528b6154f29a5314e6c8733d  
\[zk: localhost:2181(CONNECTED) 6\] rmr /hbase-unsecure/region-in-transition/1a17bb4d1f47ef6f146655b9a4a5bbed  
\[zk: localhost:2181(CONNECTED) 7\] rmr /hbase-unsecure/region-in-transition/8f50d43fff533da39e6b6bce8e2536b7  
\[zk: localhost:2181(CONNECTED) 8\] rmr /hbase-unsecure/region-in-transition/05c0ffe0e9f64b117e4ab40171782cae  
\[zk: localhost:2181(CONNECTED) 9\] rmr /hbase-unsecure/region-in-transition/c96d67c4643b819513312fc83a7549a4  
\[zk: localhost:2181(CONNECTED) 10\] rmr /hbase-unsecure/region-in-transition/fabe589ab2df00f73c4ba370ceda0a15  
\[zk: localhost:2181(CONNECTED) 11\] rmr /hbase-unsecure/region-in-transition/da8073440b414fc140cb7f5fa0341ae5  
\[zk: localhost:2181(CONNECTED) 12\] ls /hbase-unsecure/region-in-transition  
\[\]  
  
再重启两个 HMaster，结果：在hbase的web UI中不再有regions in transition. 至此，刚才已删除的表在元数据表hbase:meta中就算清理干净了。