hive sql 卡在提交JOB

Love The Way You Lie 2022-11-07 14:31 134阅读 0赞

备注:  
Hive 版本 2.1.1

### 文章目录 ###

*  一.问题描述
 *  二.问题解决
 *   *  2.1 首先想到的是重启大法
     *  2.1 查看正在执行的job
 *  三.验证
 *  参考

# 一.问题描述 #

之前开始归档，然后job卡住，我直接其它CTRL + C退出，后面运行mr job都卡在提交job的环节

hive> select * from emp where empno = 7369;
    Query ID = root_20201224123252_93210ca3-928f-49e7-9cf4-d442d2a6ca3e
    Total jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    20/12/24 12:32:54 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    Starting Job = job_1608780340033_0003, Tracking URL = http://hp3:8088/proxy/application_1608780340033_0003/
    Kill Command = /opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hadoop/bin/hadoop job  -kill job_1608780340033_0003

# 二.问题解决 #

## 2.1 首先想到的是重启大法 ##

因为维护hadoop的经验暂缺，所以想到的是重启的方法。  
首先尝试重启hive，发现问题依旧。  
后面重启整个CDH集群，发现问题依旧。  
最后尝试了重启CDH集群所在的操作系统，发现问题依旧。

## 2.1 查看正在执行的job ##

使用 hadoop job -list 和 yarn application -list 查看，发现没有job在执行。

[root@hp1 ~]# hadoop job -list
    WARNING: Use of this script to execute job is deprecated.
    WARNING: Attempting to execute replacement "mapred job" instead.
    
    20/12/24 13:38:49 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    20/12/24 13:38:49 INFO conf.Configuration: resource-types.xml not found
    20/12/24 13:38:49 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
    Total jobs:4
                      JobId              JobName         State           StartTime      UserName           Queue      Priority       UsedContainers  RsvdContainers  UsedMem         RsvdMem         NeededMem    AM info
     job_1606698967173_0358 Archiving ods_fact_s          PREP       1608631165868          root    root.users.root    DEFAULT                    0               0       0M              0M                0M http://hp3:8088/proxy/application_1606698967173_0358/
     job_1606698967173_0359 Archiving ods_fact_s          PREP       1608686680739          root    root.users.root    DEFAULT                    0               0       0M              0M                0M http://hp3:8088/proxy/application_1606698967173_0359/
     job_1606698967173_0360 obtain_user_short_me          PREP       1608711418647          root    root.users.root    DEFAULT                    0               0       0M              0M                0M http://hp3:8088/proxy/application_1606698967173_0360/
     job_1608716823585_0001 obtain_user_short_me          PREP       1608717111853          root    root.users.root    DEFAULT                    0               0       0M              0M                0M http://hp3:8088/proxy/application_1608716823585_0001/
    [root@hp1 ~]# 
    [root@hp1 ~]# 
    [root@hp1 ~]# yarn application -list
    WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
    20/12/24 13:45:34 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):8
                    Application-Id      Application-Name        Application-Type          User           Queue                   State             Final-State             Progress                        Tracking-URL
    application_1606698967173_0358  Archiving ods_fact_sa...sale_date=2010-04-12               MAPREDUCE          root      root.users.root           ACCEPTED               UNDEFINED                   0%                                 N/A
    application_1606698967173_0359  Archiving ods_fact_sa...sale_date=2010-04-12               MAPREDUCE          root      root.users.root           ACCEPTED               UNDEFINED                   0%                                 N/A
    application_1606698967173_0357  Hive on Spark (hiveSessionId = 6644365a-1e85-450f-9408-0dc223239cde)                   SPARK     anonymous      root.users.anonymous              ACCEPTED               UNDEFINED               0%                                 N/A
    application_1606698967173_0360  obtain_user_short_message.jar              MAPREDUCE          root      root.users.root           ACCEPTED               UNDEFINED                   0%                                 N/A
    application_1608780340033_0001  Hive on Spark (hiveSessionId = ed520b3f-4779-4614-86cb-ea94d43bef85)                   SPARK          root      root.users.root           ACCEPTED               UNDEFINED              0%                                  N/A
    application_1608716823585_0003  Hive on Spark (hiveSessionId = 4562d25b-0456-4bc0-86bd-78a95c6a489c)                   SPARK          root      root.users.root           ACCEPTED               UNDEFINED              0%                                  N/A
    application_1608716823585_0001  obtain_user_short_message.jar              MAPREDUCE          root      root.users.root           ACCEPTED               UNDEFINED                   0%                                 N/A
    application_1608779426561_0001  Hive on Spark (hiveSessionId = e2f00acc-cc4f-4ab3-ba16-6e56bae1a3db)                   SPARK          root      root.users.root           ACCEPTED               UNDEFINED              0%                                  N/A
    [root@hp1 ~]#

**通过kill命令杀掉卡住的job**  
语法:

-- kill mr job
    hadoop job -kill job_1606698967173_0358  
    -- kill yarn application
    yarn application -kill application_1606698967173_0357

测试记录:

[root@hp1 ~]# hadoop job -kill job_1606698967173_0358  
    WARNING: Use of this script to execute job is deprecated.
    WARNING: Attempting to execute replacement "mapred job" instead.
    
    20/12/24 13:47:41 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    20/12/24 13:47:42 INFO impl.YarnClientImpl: Killed application application_1606698967173_0358
    Killed job job_1606698967173_0358
    [root@hp1 ~]# 
    [root@hp1 ~]# hadoop job -kill job_1606698967173_0359
    WARNING: Use of this script to execute job is deprecated.
    WARNING: Attempting to execute replacement "mapred job" instead.
    
    20/12/24 13:47:56 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    20/12/24 13:47:56 INFO impl.YarnClientImpl: Killed application application_1606698967173_0359
    Killed job job_1606698967173_0359
    [root@hp1 ~]# 
    [root@hp1 ~]# hadoop job -kill job_1606698967173_0360
    WARNING: Use of this script to execute job is deprecated.
    WARNING: Attempting to execute replacement "mapred job" instead.
    
    20/12/24 13:48:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    20/12/24 13:48:22 INFO impl.YarnClientImpl: Killed application application_1606698967173_0360
    Killed job job_1606698967173_0360
    [root@hp1 ~]# 
    [root@hp1 ~]# hadoop job -kill job_1606698967173_0361
    WARNING: Use of this script to execute job is deprecated.
    WARNING: Attempting to execute replacement "mapred job" instead.
    
    20/12/24 13:48:26 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    20/12/24 13:48:26 INFO mapred.ClientServiceDelegate: Could not get Job info from RM for job job_1606698967173_0361. Redirecting to job history server.
    20/12/24 13:48:26 INFO tools.CLI: Could not obtain job info after 1 attempt(s). Sleeping for 2 seconds and retrying.
    20/12/24 13:48:28 INFO mapred.ClientServiceDelegate: Could not get Job info from RM for job job_1606698967173_0361. Redirecting to job history server.
    20/12/24 13:48:28 INFO tools.CLI: Could not obtain job info after 2 attempt(s). Sleeping for 2 seconds and retrying.
    20/12/24 13:48:30 INFO mapred.ClientServiceDelegate: Could not get Job info from RM for job job_1606698967173_0361. Redirecting to job history server.
    20/12/24 13:48:30 INFO tools.CLI: Could not obtain job info after 3 attempt(s). Sleeping for 2 seconds and retrying.
    20/12/24 13:48:32 INFO mapred.ClientServiceDelegate: Could not get Job info from RM for job job_1606698967173_0361. Redirecting to job history server.
    Could not find job job_1606698967173_0361
    [root@hp1 ~]# 
    [root@hp1 ~]# hadoop job -kill job_1608716823585_0001
    WARNING: Use of this script to execute job is deprecated.
    WARNING: Attempting to execute replacement "mapred job" instead.
    
    20/12/24 13:49:27 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    20/12/24 13:49:27 INFO impl.YarnClientImpl: Killed application application_1608716823585_0001
    Killed job job_1608716823585_0001
    [root@hp1 ~]# 
    [root@hp1 ~]# hadoop job -list
    WARNING: Use of this script to execute job is deprecated.
    WARNING: Attempting to execute replacement "mapred job" instead.
    
    20/12/24 13:49:37 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    20/12/24 13:49:38 INFO conf.Configuration: resource-types.xml not found
    20/12/24 13:49:38 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
    Total jobs:0
                      JobId              JobName         State           StartTime      UserName           Queue      Priority       UsedContainers  RsvdContainers  UsedMem         RsvdMem         NeededMem    AM info
    [root@hp1 ~]# 
    [root@hp1 ~]# 
    
    
    
    
    [root@hp1 ~]# yarn application -kill application_1606698967173_0357
    WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
    20/12/24 13:51:26 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    Killing application application_1606698967173_0357
    20/12/24 13:51:26 INFO impl.YarnClientImpl: Killed application application_1606698967173_0357
    [root@hp1 ~]# 
    [root@hp1 ~]# yarn application -kill application_1608780340033_0001
    WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
    20/12/24 13:51:46 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    Application application_1608780340033_0001 has already finished 
    [root@hp1 ~]# 
    [root@hp1 ~]# yarn application -kill application_1608716823585_0003
    WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
    20/12/24 13:51:53 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    Application application_1608716823585_0003 has already finished 
    [root@hp1 ~]# 
    [root@hp1 ~]# yarn application -kill application_1608779426561_0001
    WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
    20/12/24 13:52:00 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    Application application_1608779426561_0001 has already finished 
    [root@hp1 ~]# 
    [root@hp1 ~]# yarn application -list 
    WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
    20/12/24 13:52:06 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):0
                    Application-Id      Application-Name        Application-Type          User           Queue                   State             Final-State             Progress                        Tracking-URL
    [root@hp1 ~]#

# 三.验证 #

可以看到现在运行mr job已经没有问题了

[root@hp1 ~]# yarn application -kill application_1606698967173_0357
    WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
    20/12/24 13:51:26 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    Killing application application_1606698967173_0357
    20/12/24 13:51:26 INFO impl.YarnClientImpl: Killed application application_1606698967173_0357
    [root@hp1 ~]# 
    [root@hp1 ~]# yarn application -kill application_1608780340033_0001
    WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
    20/12/24 13:51:46 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    Application application_1608780340033_0001 has already finished 
    [root@hp1 ~]# 
    [root@hp1 ~]# yarn application -kill application_1608716823585_0003
    WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
    20/12/24 13:51:53 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    Application application_1608716823585_0003 has already finished 
    [root@hp1 ~]# 
    [root@hp1 ~]# yarn application -kill application_1608779426561_0001
    WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
    20/12/24 13:52:00 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    Application application_1608779426561_0001 has already finished 
    [root@hp1 ~]# 
    [root@hp1 ~]# yarn application -list 
    WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
    20/12/24 13:52:06 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
    Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):0
                    Application-Id      Application-Name        Application-Type          User           Queue                   State             Final-State             Progress                        Tracking-URL
    [root@hp1 ~]#

# 参考 #

1.https://blog.csdn.net/weixin\_43930715/article/details/109765454