Kubernetes与大数据之二：编译并运行基于Scalar的Spark程序WordCount

喜欢ヅ旅行 2022-05-09 05:36 180阅读 0赞

## **一、前言** ##

通过SBT编译scala程序然后在Kubernetes使用Spark运行WordCount任务。

转载自https://blog.csdn.net/cloudvtech

## **二、安装环境和编译** ##

2.1 安装SBT

> curl [https://bintray.com/sbt/rpm/rpm][https_bintray.com_sbt_rpm_rpm] > bintray-sbt-rpm.repo
> 
> mv bintray-sbt-rpm.repo /etc/yum.repos.d/
> 
> yum install -y sbt

2.2 编辑文件

> mkdir -p spark-example-project/src/main/scala/
> 
> cd spark-example-project
> 
> ls /opt/spark/spark-2.1.1-bin-hadoop2.7/jars/scala-library-2.11.8.jar
> 
> ls /opt/spark/spark-2.1.1-bin-hadoop2.7/jars/spark-core\_2.11-2.1.1.jar

simple.sbt

name := "Simple Project"
    version := "1.0"
    scalaVersion := "2.11.8"
    libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.1"

src/main/scala/SparkPi.scala

package org.apache.spark.examples
    import scala.math.random
    import org.apache.spark._
    /** Computes an approximation to pi */
    object SparkPi {
      def main(args: Array[String]) {
        val conf = new SparkConf().setAppName("Spark Pi")
        val spark = new SparkContext(conf)
        val slices = if (args.length > 0) args(0).toInt else 2
        val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow
        val count = spark.parallelize(1 until n, slices).map { i =>
          val x = random * 2 - 1
          val y = random * 2 - 1
          if (x*x + y*y < 1) 1 else 0
        }.reduce(_ + _)
        println("Pi is roughly " + 4.0 * count / n)
        spark.stop()
      }
    }

2.3 编译

sbt clean
    sbt package
    [info] Loading project definition from /root/spark-example-project/project
    [info] Loading settings for project spark-example-project from simple.sbt ...
    [info] Set current project to Simple Project (in build file:/root/spark-example-project/)
    [info] Updating ...
    [info] Done updating.
    [warn] There may be incompatibilities among your library dependencies.
    [warn] Run 'evicted' to see detailed eviction warnings
    [info] Compiling 2 Scala sources to /root/spark-example-project/target/scala-2.11/classes ...
    [info] Done compiling.
    [warn] Multiple main classes detected.  Run 'show discoveredMainClasses' to see the list
    [info] Packaging /root/spark-example-project/target/scala-2.11/simple-project_2.11-1.0.jar ...
    [info] Done packaging.
    [success] Total time: 27 s, completed Sep 10, 2018 1:48:22 PM

转载自https://blog.csdn.net/cloudvtech

## **三、打包和运行** ##

3.1 构建docker image

cp spark-example-project/target/scala-2.11/simple-project_2.11-1.0.jar /root/spark-2.3.0-bin-hadoop2.7/examples/jars/sparkpi_2.11-1.0.jar
    cd /root/spark-2.3.0-bin-hadoop2.7
    docker build -t  192.168.56.10:5000/spark:2.3.0.4 -f kubernetes/dockerfiles/spark/Dockerfile  .
    docker push 192.168.56.10:5000/spark:2.3.0.4

3.2 运行task

/root/spark/bin/spark-submit  \
       --master k8s://https://192.168.56.10:6443  \
       --deploy-mode cluster  \
       --name spark-pi   \
       --class org.apache.spark.examples.SparkPi \
       --conf spark.executor.instances=3  \
       --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
       --conf spark.kubernetes.container.image=192.168.56.10:5000/spark:2.3.0.4  \
              local:///opt/spark/examples/jars/sparkpi_2.11-1.0.jar

![70][]

![70 1][]

转载自https://blog.csdn.net/cloudvtech

[https_bintray.com_sbt_rpm_rpm]: https://bintray.com/sbt/rpm/rpm
[70]: /images/20220509/aab1d6c0b9c3434ea0b7c69beb27dd2f.png
[70 1]: /images/20220509/a72a4f95bd4e49919315d14a5d7d77ff.png