Spark的三种提交方式（基于IDEA）

「爱情、让人受尽委屈。」 2023-06-16 13:53 24阅读 0赞

# 0、创建了一个Maven工程 #

*  将一些暂时无关的文件和文件夹删除掉，主要有 main\\java, main\\resources 和 test 这三个；
 *  将Scala的框架添加到这个项目中，方法是在左侧栏中的项目名称上右键菜单中点击Add Framework Support…，然后在打开的对话框左侧边栏中，勾选Scala前面的复选框，然后点击确定即可（前提是上文中所述步骤都已正确走通，否则你很有可能看不到Scala这个选项的）；
 *  在main文件夹中建立一个名为 scala 的文件夹，并右键点击 scala 文件夹，选择 Make Directory as，然后选择Sources Root ，这里主要意思是将 scala 文件夹标记为一个源文件的根目录，然后在其内的所有代码中的 package ，其路径就从这个根目录下开始算起。
 *  在已经标记好为源文件根目录的 scala 文件夹 上，右键选择 New，然后选择 Scala Class，随后设置好程序的名称，并且记得将其设置为一个 Object(类似于Java中含有静态成员的静态类)

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
    <project xmlns="http://maven.apache.org/POM/4.0.0"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
        <modelVersion>4.0.0</modelVersion>
    
        <groupId>com.sparklearn</groupId>
        <artifactId>sparklearn</artifactId>
        <version>1.0-SNAPSHOT</version>
    
        <properties>
            <spark.version>2.2.1</spark.version>
            <scala.version>2.11</scala.version>
            <hadoop.version>2.6.5</hadoop.version>
        </properties>
    
        <dependencies>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-core_${scala.version}</artifactId>
                <version>${spark.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-sql_${scala.version}</artifactId>
                <version>${spark.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-hive_${scala.version}</artifactId>
                <version>${spark.version}</version>
            </dependency>
            
                
                
                
            
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-client</artifactId>
                <version>2.6.0</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-mllib_${scala.version}</artifactId>
                <version>${spark.version}</version>
            </dependency>
            <dependency>
                <groupId>mysql</groupId>
                <artifactId>mysql-connector-java</artifactId>
                <version>5.1.39</version>
            </dependency>
            <dependency>
                <groupId>junit</groupId>
                <artifactId>junit</artifactId>
                <version>4.12</version>
            </dependency>
        </dependencies>
        <build>
            <plugins>
    
                <plugin>
                    <groupId>org.scala-tools</groupId>
                    <artifactId>maven-scala-plugin</artifactId>
                    <version>2.15.2</version>
                    <executions>
                        <execution>
                            <goals>
                                <goal>compile</goal>
                                <goal>testCompile</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
    
                <plugin>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>2.3</version>
                    <configuration>
                        <source>1.8</source>
                        <target>1.8</target>
                    </configuration>
                </plugin>
    
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>2.19</version>
                    <configuration>
                        <skip>true</skip>
                    </configuration>
                </plugin>
    
            </plugins>
        </build>
    
    </project>

# 1、Local模式 #

*  初始构建项目阶段，使用Local模式本地运行

1）从本地（windows）读取文件，输出到本地

object UserItemList {
      def main(args: Array[String]): Unit = {
        val conf = new SparkConf().setMaster("local").setAppName("useritemlist")
        val sc = new SparkContext(conf)
    
        //window下的输入输出路径
        val datardd = sc.textFile("C:\\Users\\root\\spark_test\\useritem\\input\\user_item.data")
        val output = "C:\\Users\\root\\spark_test\\useritem\\output"
    
        val resultrdd = datardd.filter{ x =>
          val ui = x.split("\t")
          ui(2).toDouble >3
        }.map{ x =>
          val ui = x.split("\t")
          (ui(0),(ui(1),ui(2)))
        }.groupByKey()
          .map{ x=>
            val userid = x._1
            val itemlist = x._2
            val tem_arr = itemlist.toArray.sortWith(_._2 > _._2)
            var len = tem_arr.length
            if (len >3){
              len = 3
            }
    
            var stb = new StringBuilder
            for (i <- 0 until len){
              stb ++= tem_arr(i)._1
              stb.append(":")
              stb ++= tem_arr(i)._2
              stb.append(" ")
            }
            userid + "\t" + stb
    
          }
    
        resultrdd.saveAsTextFile(output)
    
      }
    
    }

2）从HDFS上读取数据，结果写到HDFS

import org.apache.spark.{SparkConf, SparkContext}
    
    object Test {
    
      def main(args: Array[String]): Unit = {
        val conf = new SparkConf()
            conf.setMaster("local")
            conf.setAppName("useritemlist")
        val sc = new SparkContext(conf)
    
        val datardd = sc.textFile("hdfs://node1:8020/Test/spark_test/useritem/input/user_item.data")
        val output = "hdfs://node1:8020/Test/spark_test/useritem/output"
    
        val resultrdd = datardd.filter{ x =>
          val ui = x.split("\t")
          ui(2).toDouble >3
        }.map{ x =>
          val ui = x.split("\t")
          (ui(0),(ui(1),ui(2)))
        }.groupByKey()
          .map{ x=>
            val userid = x._1
            val itemlist = x._2
            val tem_arr = itemlist.toArray.sortWith(_._2 > _._2)
            var len = tem_arr.length
            if (len >3){
              len = 3
            }
    
            var stb = new StringBuilder
            for (i <- 0 until len){
              stb ++= tem_arr(i)._1
              stb.append(":")
              stb ++= tem_arr(i)._2
              stb.append(" ")
            }
            userid + "\t" + stb
    
          }
        resultrdd.saveAsTextFile(output)
      }
    }

# 2、standalone模式（分client与cluster两种方式） #

确保spark集群启动

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ltMDEyMTM_size_16_color_FFFFFF_t_70][]

打包程序上传，注意程序中**setMaster要注释掉。**

\--deploy-mode client，在提交的节点会有个main进程，来运行Driver program。

\--deploy-mode cluster，则Driver program直接运行在worker中。

## 1）client模式 ##

/usr/local/src/spark-2.2.1-bin-hadoop2.6/bin/spark-submit  \
            --master spark://node1:7077 \
            --deploy-mode client \
            --class Test /root/Test/spark_test/sparklearn.jar

## 2）cluster 模式 ##

注意：standalone 模式的 cluster 要**把jar 文件传到hdfs上面去**，因为driver在集群中的任意一节点执行。

/usr/local/src/spark-2.2.1-bin-hadoop2.6/bin/spark-submit  \
             --master spark://node1:7077 \
             --deploy-mode cluster \
             --class Test hdfs://node1:8020/Test/spark_test/useritem/sparklearn.jar

![20191127144830857.png][]

# 3、Yarn模式（分client与cluster两种方式） #

## 1）client模式 ##

/usr/local/src/spark-2.2.1-bin-hadoop2.6/bin/spark-submit  \
            --master yarn \
            --deploy-mode client \
            --class Test /root/Test/spark_test/sparklearn.jar

## 2）cluster 模式 ##

/usr/local/src/spark-2.2.1-bin-hadoop2.6/bin/spark-submit  \
            --master yarn \
            --deploy-mode cluster \
            --class Test /root/Test/spark_test/sparklearn.jar

![20191127145556543.png][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ltMDEyMTM_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191127125803652.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ltMDEyMTM=,size_16,color_FFFFFF,t_70
[20191127144830857.png]: https://img-blog.csdnimg.cn/20191127144830857.png
[20191127145556543.png]: https://img-blog.csdnimg.cn/20191127145556543.png