Spark提交Jar任务警告: Initial job has not accepted any resources;

约定不等于承诺〃 2023-06-24 04:49 4阅读 0赞

## 错误信息描述 ##

**在Spark提交任务时, 会一直出现下面警告**

**会一直出现警告信息**

WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMzcxNTU2_size_16_color_FFFFFF_t_70]

# 提交任务的命令
     ./spark-submit --master spark://node1:7077 --class  ah.szxy.scalaspark.sql.DataSetAndDataFrame.CreateDataFrameFromHive /root/test/MySpark-1.0-SNAPSHOT-jar-with-dependencies.jar

**300秒之后还会导致超时问题**

Exception in thread "main" java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]
    	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
    	at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
    	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:201)
    	at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.doExecuteBroadcast(BroadcastExchangeExec.scala:136)
    	at org.apache.spark.sql.execution.InputAdapter.doExecuteBroadcast(WholeStageCodegenExec.scala:367)
    	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeBroadcast$1.apply(SparkPlan.scala:144)
    	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeBroadcast$1.apply(SparkPlan.scala:140)
    	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
    	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
    	at org.apache.spark.sql.execution.SparkPlan.executeBroadcast(SparkPlan.scala:140)
    	at org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.prepareBroadcast(BroadcastHashJoinExec.scala:135)
    	at org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.codegenInner(BroadcastHashJoinExec.scala:232)
    	at org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.doConsume(BroadcastHashJoinExec.scala:102)
    	at org.apache.spark.sql.execution.CodegenSupport$class.consume(WholeStageCodegenExec.scala:181)
    	at org.apache.spark.sql.execution.FilterExec.consume(basicPhysicalOperators.scala:85)
    	at org.apache.spark.sql.execution.FilterExec.doConsume(basicPhysicalOperators.scala:206)
    	at org.apache.spark.sql.execution.CodegenSupport$class.consume(WholeStageCodegenExec.scala:181)
    	at org.apache.spark.sql.execution.InputAdapter.consume(WholeStageCodegenExec.scala:354)
    	at org.apache.spark.sql.execution.InputAdapter.doProduce(WholeStageCodegenExec.scala:383)
    	at org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:88)
    	at org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:83)
    	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
    	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
    	at org.apache.spark.sql.execution.CodegenSupport$class.produce(WholeStageCodegenExec.scala:83)
    	at org.apache.spark.sql.execution.InputAdapter.produce(WholeStageCodegenExec.scala:354)
    	at org.apache.spark.sql.execution.FilterExec.doProduce(basicPhysicalOperators.scala:125)
    	at org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:88)
    	at org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:83)
    	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
    	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
    	at org.apache.spark.sql.execution.CodegenSupport$class.produce(WholeStageCodegenExec.scala:83)
    	at org.apache.spark.sql.execution.FilterExec.produce(basicPhysicalOperators.scala:85)
    	at org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.doProduce(BroadcastHashJoinExec.scala:97)
    	at org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:88)
    	at org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:83)
    	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
    	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
    	at org.apache.spark.sql.execution.CodegenSupport$class.produce(WholeStageCodegenExec.scala:83)
    	at org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.produce(BroadcastHashJoinExec.scala:39)
    	at org.apache.spark.sql.execution.ProjectExec.doProduce(basicPhysicalOperators.scala:45)
    	at org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:88)
    	at org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:83)
    	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
    	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
    	at org.apache.spark.sql.execution.CodegenSupport$class.produce(WholeStageCodegenExec.scala:83)
    	at org.apache.spark.sql.execution.ProjectExec.produce(basicPhysicalOperators.scala:35)
    	at org.apache.spark.sql.execution.BaseLimitExec$class.doProduce(limit.scala:70)
    	at org.apache.spark.sql.execution.LocalLimitExec.doProduce(limit.scala:97)
    	at org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:88)
    	at org.apache.spark.sql.execution.CodegenSupport$$anonfun$produce$1.apply(WholeStageCodegenExec.scala:83)
    	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
    	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
    	at org.apache.spark.sql.execution.CodegenSupport$class.produce(WholeStageCodegenExec.scala:83)
    	at org.apache.spark.sql.execution.LocalLimitExec.produce(limit.scala:97)
    	at org.apache.spark.sql.execution.WholeStageCodegenExec.doCodeGen(WholeStageCodegenExec.scala:524)
    	at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:576)
    	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
    	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
    	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
    	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
    	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
    	at org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:247)
    	at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:337)
    	at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)
    	at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:3273)
    	at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2484)
    	at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2484)
    	at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3254)
    	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
    	at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3253)
    	at org.apache.spark.sql.Dataset.head(Dataset.scala:2484)
    	at org.apache.spark.sql.Dataset.take(Dataset.scala:2698)
    	at org.apache.spark.sql.Dataset.showString(Dataset.scala:254)
    	at org.apache.spark.sql.Dataset.show(Dataset.scala:723)
    	at org.apache.spark.sql.Dataset.show(Dataset.scala:682)
    	at com.bjsxt.scalaspark.sql.DataSetAndDataFrame.CreateDataFrameFromHive$.main(CreateDataFrameFromHive.scala:24)
    	at com.bjsxt.scalaspark.sql.DataSetAndDataFrame.CreateDataFrameFromHive.main(CreateDataFrameFromHive.scala)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:498)
    	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
    	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
    	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
    	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
    	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

## 错误解决思路 ##

1.  根据提示我们查看Spark Master的WebUI  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMzcxNTU2_size_16_color_FFFFFF_t_70 1]
2.  可以看到当前正在运行的程序有两个, **第一个是我们提交的Spark任务, 但是这个任务没有分到相应的核数, 且任务状态是等待状态 ( 原因是申请不到资源 ); 而资源都被第二个任务(Spark-Shell ,处于Running状态 )所占**  
     因此我们可以通过WebUI 将该任务 Kill即可 
3.  由图我们可以看到阻塞的任务有重新运行了  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMzcxNTU2_size_16_color_FFFFFF_t_70 2]  
    注意: 我们还可以在提交任务时指定**执行核的参数以及内存参数**也能解决该问题, 总思路就是能够让当前应用程序能够申请并使用资源

## 总结 ##

1.  **提交任务的时候也会用到Spark shell, 因此应该将别的窗口的Spark Shell关闭**
2.  **Spark Shell 进入,Spark的bin目录下** `./spark-shell --master spark://node1:7077`
3.  **Spark Master WebUI端口为 8080**
4.  Spark用到的端口\\
    
        50070  	Hdfs WebUI
        7077 	   Spark Master通信端口
        8080       Spark Master WebUI
        8081 	   Spark Worker WebUI
        18080	   Spark HistoryServer WebUI
        4040 	   Spark Driver WebUI 端口  任务调度
        9083	   hive源数据服务端口(node3)

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMzcxNTU2_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191220162739975.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMzcxNTU2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMzcxNTU2_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191220162917171.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMzcxNTU2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMzcxNTU2_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191220163503915.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMzcxNTU2,size_16,color_FFFFFF,t_70