Scala IDE 搭建Spark 2开发环境和运行例子

àì夳堔傛蜴生んèń 2022-07-16 09:25 217阅读 0赞

在widow上用Scala IDE  创建Spark 2.0 的开发环境

1、创建 maven Project

![Center][]

2、 修改pom.xml

花了很多时间在这里修改pom.xml,  可以参考如maven repository和Github的pom.xml

最后我的pom.xml如下：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">  
  <modelVersion>4.0.0</modelVersion>  
  <groupId>com.big</groupId>  
  <artifactId>mydata</artifactId>  
  <version>0.0.1-SNAPSHOT</version>  
   <dependencies>  
    <dependency>  
    <groupId>org.scala-lang</groupId>  
    <artifactId>scala-library</artifactId>  
    <version>2.10.6</version>  
  </dependency>  
  <dependency>  
    <groupId>org.apache.spark</groupId>  
    <artifactId>spark-core\_2.10</artifactId>  
    <version>2.0.0</version>  
   </dependency>  
  </dependencies>  
<pluginRepositories>  
<pluginRepository>  
    <id>scala-tools.org</id>  
    <name>Scala-tools Maven2 Repository</name>  
    <url>http://scala-tools.org/repo-releases</url>  
  </pluginRepository>  
</pluginRepositories>  
<build>  
    <sourceDirectory>src/main/scala</sourceDirectory>  
    <testSourceDirectory>src/test/scala</testSourceDirectory>  
    <plugins>  
      <plugin>  
        <groupId>org.scala-tools</groupId>  
        <artifactId>maven-scala-plugin</artifactId>  
        <executions>  
          <execution>  
            <goals>  
              <goal>compile</goal>  
              <goal>testCompile</goal>  
            </goals>  
          </execution>  
        </executions>  
      </plugin>  
       
    </plugins>  
    <defaultGoal>compile</defaultGoal>  
  </build>   
    
</project>

3. 修改配置

选中项目右键 configure -> add  scala nature..

右键property  在对话框调整Scala 编译器等信息

4. 写spark例子

在/src/main/scala  下,创建SimpleApp .scala

import org.apache.spark.SparkContext  
import org.apache.spark.SparkContext.\_  
import org.apache.spark.SparkConf  
  
object SimpleApp \{  
  def main(args: Array\[String\]) \{  
    val logFile = "C:/spark/README.md" // Should be some file on your system  
    val conf = new SparkConf().setAppName("Simple Application").setMaster("local")  
    val sc = new SparkContext(conf)  
    val logData = sc.textFile(logFile, 2).cache()  
    val numAs = logData.filter(line => line.contains("a")).count()  
    val numBs = logData.filter(line => line.contains("b")).count()  
    println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))  
  \}  
\}

5.  运行例子

选中pom.xml 右键 Run AS ->Maven  build

如果build成功，可以选中 SimpleApp .scala  右键运行Scala application

就可以在控制台看到Log, 说明成功。

............................

16/10/13 16:32:20 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 43 ms on localhost (2/2)  
16/10/13 16:32:20 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool   
16/10/13 16:32:20 INFO DAGScheduler: Job 0 finished: count at SimpleApp .scala:15, took 0.414282 s  
16/10/13 16:32:20 INFO SparkContext: Starting job: count at SimpleApp .scala:16  
16/10/13 16:32:20 INFO DAGScheduler: Got job 1 (count at SimpleApp .scala:16) with 2 output partitions  
16/10/13 16:32:20 INFO DAGScheduler: Final stage: ResultStage 1 (count at SimpleApp .scala:16)  
16/10/13 16:32:20 INFO DAGScheduler: Parents of final stage: List()  
16/10/13 16:32:20 INFO DAGScheduler: Missing parents: List()  
16/10/13 16:32:20 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD\[3\] at filter at SimpleApp .scala:16), which has no missing parents  
16/10/13 16:32:20 INFO MemoryStore: Block broadcast\_2 stored as values in memory (estimated size 3.2 KB, free 897.5 MB)  
16/10/13 16:32:20 INFO MemoryStore: Block broadcast\_2\_piece0 stored as bytes in memory (estimated size 1965.0 B, free 897.5 MB)  
16/10/13 16:32:20 INFO BlockManagerInfo: Added broadcast\_2\_piece0 in memory on 10.56.182.2:52894 (size: 1965.0 B, free: 897.6 MB)  
16/10/13 16:32:20 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1012  
16/10/13 16:32:20 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 1 (MapPartitionsRDD\[3\] at filter at SimpleApp .scala:16)  
16/10/13 16:32:20 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks  
16/10/13 16:32:20 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, localhost, partition 0, PROCESS\_LOCAL, 5278 bytes)  
16/10/13 16:32:20 INFO Executor: Running task 0.0 in stage 1.0 (TID 2)  
16/10/13 16:32:20 INFO BlockManager: Found block rdd\_1\_0 locally  
16/10/13 16:32:20 INFO Executor: Finished task 0.0 in stage 1.0 (TID 2). 954 bytes result sent to driver  
16/10/13 16:32:20 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, localhost, partition 1, PROCESS\_LOCAL, 5278 bytes)  
16/10/13 16:32:20 INFO Executor: Running task 1.0 in stage 1.0 (TID 3)  
16/10/13 16:32:20 INFO BlockManager: Found block rdd\_1\_1 locally  
16/10/13 16:32:20 INFO Executor: Finished task 1.0 in stage 1.0 (TID 3). 954 bytes result sent to driver  
16/10/13 16:32:20 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 32 ms on localhost (1/2)  
16/10/13 16:32:20 INFO DAGScheduler: ResultStage 1 (count at SimpleApp .scala:16) finished in 0.038 s  
16/10/13 16:32:20 INFO DAGScheduler: Job 1 finished: count at SimpleApp .scala:16, took 0.069170 s  
16/10/13 16:32:20 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 21 ms on localhost (2/2)  
16/10/13 16:32:20 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool   
Lines with a: 61, Lines with b: 27  
16/10/13 16:32:20 INFO SparkContext: Invoking stop() from shutdown hook  
16/10/13 16:32:20 INFO SparkUI: Stopped Spark web UI at http://10.56.182.2:4040  
16/10/13 16:32:20 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!  
16/10/13 16:32:20 INFO MemoryStore: MemoryStore cleared  
16/10/13 16:32:20 INFO BlockManager: BlockManager stopped  
16/10/13 16:32:20 INFO BlockManagerMaster: BlockManagerMaster stopped  
16/10/13 16:32:20 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!  
16/10/13 16:32:20 INFO SparkContext: Successfully stopped SparkContext  
16/10/13 16:32:20 INFO ShutdownHookManager: Shutdown hook called

[Center]: /images/20220716/f6684b2416764577aa7f0b14b607b3b5.png