[Hadoop] Spark2.3.1编译报错“[error] javac: invalid source release: 1.8”解决

你的名字 2022-05-14 09:59 128阅读 0赞

编译环境：CentOS7.3 + Maven-3.3.9 +  JDK1.8.0\_181 + Scala-2.11.12

Hadoop版本：hadoop-2.6.0-cdh5.13.1

Spark源码版本：spark-2.3.1

Spark编译报错：\[error\] javac: invalid source release: 1.8

[hadoop@hadoop000 spark-2.3.1]$ ./dev/make-distribution.sh --name 2.6.0-cdh5.13.1 --tgz -Dhadoop.version=2.6.0-cdh5.13.1 -Dhadoop2.6 -Phive -Phive-thriftserver -Pyarn
    +++ dirname ./dev/make-distribution.sh
    ++ cd ./dev/..
    ++ pwd
    + SPARK_HOME=/home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1
    + DISTDIR=/home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/dist
    + MAKE_TGZ=false
    + MAKE_PIP=false
    + MAKE_R=false
    + NAME=none
    + MVN=/home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/build/mvn
    + ((  8  ))
    + case $1 in
    + NAME=2.6.0-cdh5.13.1
    + shift
    + shift
    + ((  6  ))
    + case $1 in
    + MAKE_TGZ=true
    + shift
    + ((  5  ))
    + case $1 in
    + break
    + '[' -z /home/hadoop/app/jdk1.8.0_181 ']'
    + '[' -z /home/hadoop/app/jdk1.8.0_181 ']'
    ++ command -v git
    + '[' /usr/bin/git ']'
    ++ git rev-parse --short HEAD
    ++ :
    + GITREV=
    + '[' '!' -z '' ']'
    + unset GITREV
    ++ command -v /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/build/mvn
    + '[' '!' /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/build/mvn ']'
    + VERSION=2.3.1
    + SCALA_VERSION=2.11
    + SPARK_HADOOP_VERSION=2.6.0-cdh5.13.1
    + SPARK_HIVE=1
    + '[' 2.6.0-cdh5.13.1 == none ']'
    + echo 'Spark version is 2.3.1'
    Spark version is 2.3.1
    + '[' true == true ']'
    + echo 'Making spark-2.3.1-bin-2.6.0-cdh5.13.1.tgz'
    Making spark-2.3.1-bin-2.6.0-cdh5.13.1.tgz
    + cd /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1
    + export 'MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=512m'
    + MAVEN_OPTS='-Xmx2g -XX:ReservedCodeCacheSize=512m'
    + BUILD_COMMAND=("$MVN" -T 1C clean package -DskipTests $@)
    + echo -e '\nBuilding with...'
    
    Building with...
    + echo -e '$ /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/build/mvn' -T 1C clean package -DskipTests -Dhadoop.version=2.6.0-cdh5.13.1 -Dhadoop2.6 -Phive -Phive-thriftserver '-Pyarn\n'
    $ /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/build/mvn -T 1C clean package -DskipTests -Dhadoop.version=2.6.0-cdh5.13.1 -Dhadoop2.6 -Phive -Phive-thriftserver -Pyarn
    
    + /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/build/mvn -T 1C clean package -DskipTests -Dhadoop.version=2.6.0-cdh5.13.1 -Dhadoop2.6 -Phive -Phive-thriftserver -Pyarn
    Using `mvn` from path: /home/hadoop/app/apache-maven-3.3.9/bin/mvn
    [INFO] Scanning for projects...
    [INFO] ------------------------------------------------------------------------
    [INFO] Reactor Build Order:
    [INFO] 
    [INFO] Spark Project Parent POM
    [INFO] Spark Project Tags
    [INFO] Spark Project Sketch
    [INFO] Spark Project Local DB
    [INFO] Spark Project Networking
    [INFO] Spark Project Shuffle Streaming Service
    [INFO] Spark Project Unsafe
    [INFO] Spark Project Launcher
    [INFO] Spark Project Core
    [INFO] Spark Project ML Local Library
    [INFO] Spark Project GraphX
    [INFO] Spark Project Streaming
    [INFO] Spark Project Catalyst
    [INFO] Spark Project SQL
    [INFO] Spark Project ML Library
    [INFO] Spark Project Tools
    [INFO] Spark Project Hive
    [INFO] Spark Project REPL
    [INFO] Spark Project YARN Shuffle Service
    [INFO] Spark Project YARN
    [INFO] Spark Project Hive Thrift Server
    [INFO] Spark Project Assembly
    [INFO] Spark Integration for Kafka 0.10
    [INFO] Kafka 0.10 Source for Structured Streaming
    [INFO] Spark Project Examples
    [INFO] Spark Integration for Kafka 0.10 Assembly
    [INFO] 
    [INFO] Using the MultiThreadedBuilder implementation with a thread count of 8
    [INFO]                                                                         
    [INFO] ------------------------------------------------------------------------
    [INFO] Building Spark Project Parent POM 2.3.1
    [INFO] ------------------------------------------------------------------------
    [INFO] 
    [INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ spark-parent_2.11 ---
    [INFO] Deleting /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/target
    [INFO] 
    [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-versions) @ spark-parent_2.11 ---
    [INFO] 
    [INFO] --- scala-maven-plugin:3.2.2:add-source (eclipse-add-source) @ spark-parent_2.11 ---
    [INFO] Add Source directory: /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/src/main/scala
    [INFO] Add Test Source directory: /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/src/test/scala
    [INFO] 
    [INFO] --- maven-dependency-plugin:3.0.2:build-classpath (default-cli) @ spark-parent_2.11 ---
    [INFO] Dependencies classpath:
    /home/hadoop/maven_repo/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar
    [INFO] 
    [INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) @ spark-parent_2.11 ---
    [INFO] 
    [INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-parent_2.11 ---
    [INFO] No sources to compile
    [INFO] 
    [INFO] --- maven-antrun-plugin:1.8:run (create-tmp-dir) @ spark-parent_2.11 ---
    [INFO] Executing tasks
    
    main:
        [mkdir] Created dir: /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/target/tmp
    [INFO] Executed tasks
    [INFO] 
    [INFO] --- maven-dependency-plugin:3.0.2:build-classpath (generate-test-classpath) @ spark-parent_2.11 ---
    [INFO] Dependencies classpath:
    /home/hadoop/maven_repo/org/scala-lang/modules/scala-xml_2.11/1.0.5/scala-xml_2.11-1.0.5.jar:/home/hadoop/maven_repo/org/scalatest/scalatest_2.11/3.0.3/scalatest_2.11-3.0.3.jar:/home/hadoop/maven_repo/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/home/hadoop/maven_repo/org/scala-sbt/test-interface/1.0/test-interface-1.0.jar:/home/hadoop/maven_repo/com/novocode/junit-interface/0.11/junit-interface-0.11.jar:/home/hadoop/maven_repo/org/scala-lang/scala-library/2.11.8/scala-library-2.11.8.jar:/home/hadoop/maven_repo/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:/home/hadoop/maven_repo/org/scalactic/scalactic_2.11/3.0.3/scalactic_2.11-3.0.3.jar:/home/hadoop/maven_repo/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.4/scala-parser-combinators_2.11-1.0.4.jar:/home/hadoop/maven_repo/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar:/home/hadoop/maven_repo/junit/junit/4.12/junit-4.12.jar
    [INFO] 
    [INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) @ spark-parent_2.11 ---
    [INFO] No sources to compile
    [INFO] 
    [INFO] --- maven-surefire-plugin:2.20.1:test (test) @ spark-parent_2.11 ---
    [INFO] Tests are skipped.
    [INFO] 
    [INFO] --- scalatest-maven-plugin:1.0:test (test) @ spark-parent_2.11 ---
    [INFO] Tests are skipped.
    [INFO] 
    [INFO] --- maven-jar-plugin:3.0.2:test-jar (prepare-test-jar) @ spark-parent_2.11 ---
    [INFO] Building jar: /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/target/spark-parent_2.11-2.3.1-tests.jar
    [INFO] 
    [INFO] --- maven-site-plugin:3.5.1:attach-descriptor (attach-descriptor) @ spark-parent_2.11 ---
    [INFO] 
    [INFO] --- maven-shade-plugin:3.1.0:shade (default) @ spark-parent_2.11 ---
    [INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
    [INFO] Replacing original artifact with shaded artifact.
    [INFO] 
    [INFO] --- maven-source-plugin:3.0.1:jar-no-fork (create-source-jar) @ spark-parent_2.11 ---
    [INFO] 
    [INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork (create-source-jar) @ spark-parent_2.11 ---
    [INFO]                                                                         
    [INFO] ------------------------------------------------------------------------
    [INFO] Building Spark Project Tags 2.3.1
    [INFO] ------------------------------------------------------------------------
    [INFO]                                                                         
    [INFO] ------------------------------------------------------------------------
    [INFO] Building Spark Project Tools 2.3.1
    [INFO] ------------------------------------------------------------------------
    [INFO] 
    [INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ spark-tags_2.11 ---
    [INFO] Deleting /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/common/tags/target
    [INFO] 
    [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-versions) @ spark-tags_2.11 ---
    [INFO] 
    [INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ spark-tools_2.11 ---
    [INFO] Deleting /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/tools/target
    [INFO] 
    [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-versions) @ spark-tools_2.11 ---
    [INFO] 
    [INFO] --- scala-maven-plugin:3.2.2:add-source (eclipse-add-source) @ spark-tags_2.11 ---
    [INFO] Add Source directory: /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/common/tags/src/main/scala
    [INFO] Add Test Source directory: /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/common/tags/src/test/scala
    [INFO] 
    [INFO] --- maven-dependency-plugin:3.0.2:build-classpath (default-cli) @ spark-tags_2.11 ---
    [INFO] Dependencies classpath:
    /home/hadoop/maven_repo/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/home/hadoop/maven_repo/org/scala-lang/scala-library/2.11.8/scala-library-2.11.8.jar
    [INFO] 
    [INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) @ spark-tags_2.11 ---
    [INFO] 
    [INFO] --- scala-maven-plugin:3.2.2:add-source (eclipse-add-source) @ spark-tools_2.11 ---
    [INFO] Add Source directory: /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/tools/src/main/scala
    [INFO] Add Test Source directory: /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/tools/src/test/scala
    [INFO] 
    [INFO] --- maven-dependency-plugin:3.0.2:build-classpath (default-cli) @ spark-tools_2.11 ---
    [INFO] Dependencies classpath:
    /home/hadoop/maven_repo/org/ow2/asm/asm-tree/5.1/asm-tree-5.1.jar:/home/hadoop/maven_repo/org/ow2/asm/asm-util/5.1/asm-util-5.1.jar:/home/hadoop/maven_repo/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/home/hadoop/maven_repo/org/ow2/asm/asm-commons/5.1/asm-commons-5.1.jar:/home/hadoop/maven_repo/org/scala-lang/scala-library/2.11.8/scala-library-2.11.8.jar:/home/hadoop/maven_repo/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.4/scala-parser-combinators_2.11-1.0.4.jar:/home/hadoop/maven_repo/org/scala-lang/scala-compiler/2.11.8/scala-compiler-2.11.8.jar:/home/hadoop/maven_repo/org/clapper/grizzled-scala_2.11/4.2.0/grizzled-scala_2.11-4.2.0.jar:/home/hadoop/maven_repo/org/scala-lang/modules/scala-xml_2.11/1.0.4/scala-xml_2.11-1.0.4.jar:/home/hadoop/maven_repo/org/ow2/asm/asm/5.1/asm-5.1.jar:/home/hadoop/maven_repo/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar:/home/hadoop/maven_repo/org/clapper/classutil_2.11/1.1.2/classutil_2.11-1.1.2.jar
    [INFO] 
    [INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) @ spark-tools_2.11 ---
    [INFO] 
    [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ spark-tags_2.11 ---
    [INFO] Using 'UTF-8' encoding to copy filtered resources.
    [INFO] skip non existing resourceDirectory /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/common/tags/src/main/resources
    [INFO] Copying 3 resources
    [INFO] 
    [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ spark-tools_2.11 ---
    [INFO] Using 'UTF-8' encoding to copy filtered resources.
    [INFO] skip non existing resourceDirectory /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/tools/src/main/resources
    [INFO] Copying 3 resources
    [INFO] 
    [INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ spark-tools_2.11 ---
    [INFO] 
    [INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ spark-tags_2.11 ---
    [INFO] Not compiling main sources
    [INFO] Not compiling main sources
    [INFO] 
    [INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-tags_2.11 ---
    [INFO] 
    [INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-tools_2.11 ---
    [INFO] Using zinc server for incremental compilation
    [INFO] Using zinc server for incremental compilation
    [[warn] Pruning sources from previous analysis, due to incompatible CompileSetup.
    warn] Pruning sources from previous analysis, due to incompatible CompileSetup.
    [info] Compiling 1 Scala source to /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/tools/target/scala-2.11/classes...
    [info] Compiling 2 Scala sources and 6 Java sources to /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/common/tags/target/scala-2.11/classes...
    [error] javac: invalid source release: 1.8
    [error] Usage: javac <options> <source files>
    [error] use -help for a list of possible options
    [error] Compile failed at Sep 6, 2018 11:25:37 AM [0.326s]
    [INFO] ------------------------------------------------------------------------
    [INFO] Reactor Summary:
    [INFO] 
    [INFO] Spark Project Parent POM ........................... SUCCESS [  1.812 s]
    [INFO] Spark Project Tags ................................. FAILURE [  0.643 s]
    [INFO] Spark Project Sketch ............................... SKIPPED
    [INFO] Spark Project Local DB ............................. SKIPPED
    [INFO] Spark Project Networking ........................... SKIPPED
    [INFO] Spark Project Shuffle Streaming Service ............ SKIPPED
    [INFO] Spark Project Unsafe ............................... SKIPPED
    [INFO] Spark Project Launcher ............................. SKIPPED
    [INFO] Spark Project Core ................................. SKIPPED
    [INFO] Spark Project ML Local Library ..................... SKIPPED
    [INFO] Spark Project GraphX ............................... SKIPPED
    [INFO] Spark Project Streaming ............................ SKIPPED
    [INFO] Spark Project Catalyst ............................. SKIPPED
    [INFO] Spark Project SQL .................................. SKIPPED
    [INFO] Spark Project ML Library ........................... SKIPPED
    [INFO] Spark Project Tools ................................ SKIPPED
    [INFO] Spark Project Hive ................................. SKIPPED
    [INFO] Spark Project REPL ................................. SKIPPED
    [INFO] Spark Project YARN Shuffle Service ................. SKIPPED
    [INFO] Spark Project YARN ................................. SKIPPED
    [INFO] Spark Project Hive Thrift Server ................... SKIPPED
    [INFO] Spark Project Assembly ............................. SKIPPED
    [INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
    [INFO] Kafka 0.10 Source for Structured Streaming ......... SKIPPED
    [INFO] Spark Project Examples ............................. SKIPPED
    [INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD FAILURE
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 2.892 s (Wall Clock)
    [INFO] Finished at: 2018-09-06T11:25:37+08:00
    [INFO] Final Memory: 51M/636M
    [INFO] ------------------------------------------------------------------------
    [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-tags_2.11: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. CompileFailed -> [Help 1]
    [ERROR] 
    [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
    [ERROR] Re-run Maven using the -X switch to enable full debug logging.
    [ERROR] 
    [ERROR] For more information about the errors and possible solutions, please read the following articles:
    [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
    [ERROR] 
    [ERROR] After correcting the problems, you can resume the build with the command
    [ERROR]   mvn <goals> -rf :spark-tags_2.11
    [hadoop@hadoop000 spark-2.3.1]$

检查maven和java版本都没有问题，符合spark-2.3.1版本的编译要求。

[hadoop@hadoop000 spark-2.3.1]$ java -version
    java version "1.8.0_181"
    Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
    Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
    [hadoop@hadoop000 spark-2.3.1]$ javac -version
    javac 1.8.0_181
    [hadoop@hadoop000 spark-2.3.1]$ mvn -version
    Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
    Maven home: /home/hadoop/app/apache-maven-3.3.9
    Java version: 1.8.0_181, vendor: Oracle Corporation
    Java home: /home/hadoop/app/jdk1.8.0_181/jre
    Default locale: en_US, platform encoding: UTF-8
    OS name: "linux", version: "3.10.0-514.21.1.el7.x86_64", arch: "amd64", family: "unix"
    [hadoop@hadoop000 spark-2.3.1]$

后面发现，是因为在编译spark之前我在机器上面编译过hadoop2.6.0-cdh5.7.0，而此版本的hadoop要求使用jdk7来编译。hadoop编译完成之后虽然已经把jdk换回了jdk8。但是忽略了一个问题，hadoop编译时是在jdk7上执行的编译命令，导致后台启动了zinc服务。查看服务进程，它清楚描述了zinc是jdk7启动的并且还一直在运行状态。

[hadoop@hadoop000 spark-2.3.1]$ ps -ef | grep zinc
    hadoop   21816     1  1 10:52 pts/2    00:00:40 /home/hadoop/app/jdk1.7.0_80/bin/java -server -Xmx2g -XX:MaxPermSize=384m -XX:ReservedCodeCacheSize=512m -Dzinc.home=/home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/build/zinc-0.3.15 -classpath /home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/build/zinc-0.3.15/lib/compiler-interface-sources.jar:/home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/build/zinc-0.3.15/lib/incremental-compiler.jar:/home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/build/zinc-0.3.15/lib/nailgun-server.jar:/home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/build/zinc-0.3.15/lib/sbt-interface.jar:/home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/build/zinc-0.3.15/lib/scala-compiler.jar:/home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/build/zinc-0.3.15/lib/scala-library.jar:/home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/build/zinc-0.3.15/lib/scala-reflect.jar:/home/hadoop/sourcecode/spark-2.3.1_cdh5.13.1/spark-2.3.1/build/zinc-0.3.15/lib/zinc.jar com.typesafe.zinc.Nailgun 3030 0 0.0.0.0
    hadoop   28547 12587  0 11:35 pts/2    00:00:00 grep --color=auto zinc
    [hadoop@hadoop000 spark-2.3.1]$

由于spark编译时也需要使用zinc服务，而spark编译时要求jdk版本是jdk8。所以由jdk7启动的zinc服务对后来的spark编译造成了干扰。解决办法是把当前服务器上正在运行的zinc服务kill掉，再重新编译spark。

[hadoop@hadoop000 spark-2.3.1]$ kill -9 21816

重新编译spark，编译成功。

参考：[Spark官网][Spark]。

[Spark]: https://issues.apache.org/jira/browse/SPARK-21075