如何优雅地终止正在运行的Spark Streaming程序

你的名字 2022-09-27 06:47 299阅读 0赞

>  转载原文地址：可以点击：https://www.iteblog.com/archives/1890.html进入原文阅读。  
>

###     ###

一直运行的Spark Streaming程序如何关闭呢？是直接使用kill命令强制关闭吗？这种手段是可以达到关闭的目的，但是带来的后果就是可能会导致数据的丢失，因为这时候如果程序正在处理接收到的数据，但是由于接收到kill命令，那它只能停止整个程序，而那些正在处理或者还没有处理的数据可能就会被丢失。那我们咋办？这里有两种方法。

文章目录

*  1 等作业运行完再关闭
 *  2 通过Spark内置机制关闭
    
     *  2.1 Spark 1.4版本之前
     *  2.2 Spark 1.4版本之后

## 等作业运行完再关闭 ##

我们都知道，Spark Streaming每隔batchDuration的时间会把源源不断的流数据分割成一批有限数据集，然后计算这些数据，我们可以从Spark提供的监控页面看到当前batch是否执行完成，当作业执行完，我们就可以手动执行kill命令来强制关闭这个Streaming作业。这种方式的缺点就是得盯着监控页面，然后决定关不关闭，很不灵活。

如果想及时了解Spark、Hadoop或者Hbase相关的文章，欢迎关注微信公共帐号：iteblog\_hadoop

## 通过Spark内置机制关闭 ##

其实Spark内置就为我们提供了一种优雅的方法来关闭长期运行的Streaming作业，我们来看看 `StreamingContext`类中定义的一个 `stop` 方法：

官方文档对其解释是：Stop the execution of the streams, with option of ensuring all received data has been processed. 控制所有接收的数据是否被处理的参数就是 `stopGracefully`，如果我们将它设置为`true`，Spark则会等待所有接收的数据被处理完成，然后再关闭计算引擎，这样就可以避免数据的丢失。现在的问题是我们在哪里调用这个stop方法？

### Spark 1.4版本之前 ###

在Spark 1.4版本之前，我们需要手动调用这个 `stop` 方法，一种比较合适的方式是通过 `Runtime.getRuntime().addShutdownHook` 来添加一个钩子，其会在JVM关闭的之前执行传递给他的函数，如下：

如果你使用的是Scala，我们还可以通过以下的方法实现类似的功能：

通过上面的办法，我们客户确保程序退出之前会执行上面的函数，从而保证Streaming程序关闭的时候不丢失数据。

### Spark 1.4版本之后 ###

上面方式可以达到我们的需求，但是在每个程序里面都添加这样的重复代码也未免太过麻烦了！值得高兴的是，从Apache Spark 1.4版本开始，Spark内置提供了`spark.streaming.stopGracefullyOnShutdown`参数来决定是否需要以Gracefully方式来关闭Streaming程序（详情请参见SPARK-7776）。Spark会在启动 `StreamingContext` 的时候注册这个钩子，如下：

<table style="color:rgb(62,62,62); margin:0px 0px 10px; padding:0px; border-collapse:collapse; width:736px; max-width:100%; font-family:'Hiragino Sans GB','Microsoft YaHei',Arial,sans-serif; font-size:16px; line-height:25.6px; word-wrap:break-word!important"> 
 <tbody style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.1em!important; outline:0px!important; overflow:visible!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-size:1em!important; background:none!important"> 
  <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.1em!important; outline:0px!important; overflow:visible!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-size:1em!important; background:none!important"> 
   <td style="margin:0px; padding:0px!important; word-wrap:break-word!important; word-break:break-all; border:0px!important; max-width:100%; bottom:auto!important; float:none!important; left:auto!important; line-height:1.1em!important; outline:0px!important; overflow:visible!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-size:1em!important; background:none!important"> 
    <div style="margin:0px; padding-top:0px; padding-right:1em!important; padding-bottom:0px; padding-left:0em!important; max-width:100%; word-wrap:break-word!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.6!important; outline:0px!important; overflow:visible!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-size:1em!important"> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">shutdownHookRef&nbsp;</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(137,89,168)!important">=</code>&nbsp; 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">ShutdownHookManager.addShutdownHook(</code> 
    </div> 
    <div style="margin:0px; padding-top:0px; padding-right:1em!important; padding-bottom:0px; padding-left:0em!important; max-width:100%; word-wrap:break-word!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.6!important; outline:0px!important; overflow:visible!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-size:1em!important"> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(51,51,51)">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">StreamingContext.SHUTDOWN</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(137,89,168)!important">_</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">HOOK</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(137,89,168)!important">_</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">PRIORITY)(stopOnShutdown)</code> 
    </div> 
    <div style="margin:0px; padding-top:0px; padding-right:1em!important; padding-bottom:0px; padding-left:0em!important; max-width:100%; word-wrap:break-word!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.6!important; outline:0px!important; overflow:visible!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-size:1em!important">
      &nbsp; 
    </div> 
    <div style="margin:0px; padding-top:0px; padding-right:1em!important; padding-bottom:0px; padding-left:0em!important; max-width:100%; word-wrap:break-word!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.6!important; outline:0px!important; overflow:visible!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-size:1em!important"> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(137,89,168)!important">private</code>&nbsp; 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(137,89,168)!important">def</code>&nbsp; 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">stopOnShutdown()</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(137,89,168)!important">:</code>&nbsp; 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">Unit&nbsp;</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(137,89,168)!important">=</code>&nbsp; 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">{ </code> 
    </div> 
    <div style="margin:0px; padding-top:0px; padding-right:1em!important; padding-bottom:0px; padding-left:0em!important; max-width:100%; word-wrap:break-word!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.6!important; outline:0px!important; overflow:visible!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-size:1em!important"> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(51,51,51)">&nbsp;&nbsp;&nbsp;&nbsp;</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(137,89,168)!important">val</code>&nbsp; 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">stopGracefully&nbsp;</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(137,89,168)!important">=</code>&nbsp; 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">conf.getBoolean(</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(113,140,0)!important">"spark.streaming.stopGracefullyOnShutdown"</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">,&nbsp;</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(137,89,168)!important">false</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">)</code> 
    </div> 
    <div style="margin:0px; padding-top:0px; padding-right:1em!important; padding-bottom:0px; padding-left:0em!important; max-width:100%; word-wrap:break-word!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.6!important; outline:0px!important; overflow:visible!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-size:1em!important"> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(51,51,51)">&nbsp;&nbsp;&nbsp;&nbsp;</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">logInfo(s</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(113,140,0)!important">"Invoking stop(stopGracefully=$stopGracefully) from shutdown hook"</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">)</code> 
    </div> 
    <div style="margin:0px; padding-top:0px; padding-right:1em!important; padding-bottom:0px; padding-left:0em!important; max-width:100%; word-wrap:break-word!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.6!important; outline:0px!important; overflow:visible!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-size:1em!important"> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(51,51,51)">&nbsp;&nbsp;&nbsp;&nbsp;</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(142,144,140)!important">// Do not stop SparkContext, let its own shutdown hook stop it</code> 
    </div> 
    <div style="margin:0px; padding-top:0px; padding-right:1em!important; padding-bottom:0px; padding-left:0em!important; max-width:100%; word-wrap:break-word!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.6!important; outline:0px!important; overflow:visible!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-size:1em!important"> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(51,51,51)">&nbsp;&nbsp;&nbsp;&nbsp;</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">stop(stopSparkContext&nbsp;</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(137,89,168)!important">=</code>&nbsp; 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(137,89,168)!important">false</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">, stopGracefully&nbsp;</code> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(137,89,168)!important">=</code>&nbsp; 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">stopGracefully)</code> 
    </div> 
    <div style="margin:0px; padding-top:0px; padding-right:1em!important; padding-bottom:0px; padding-left:0em!important; max-width:100%; word-wrap:break-word!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.6!important; outline:0px!important; overflow:visible!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-size:1em!important"> 
     <code style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important; color:rgb(77,77,76)!important">}</code> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

从上面的代码可以看出，我们可以根据自己的需求来设置 `spark.streaming.stopGracefullyOnShutdown` 的值，而不需要在每个Streaming程序里面手动调用`StreamingContext`的stop方法，确实方便多了。不过虽然这个参数在Spark 1.4开始引入，但是却是在Spark 1.6才开始才有文档正式介绍（可以参见https://github.com/apache/spark/pull/8898和http://spark.apache.org/docs/1.6.0/configuration.html）