本地模式下（local），java连接spark调用scala语言打成的jar包

àì夳堔傛蜴生んèń 2022-02-09 09:34 134阅读 0赞

# 本地模式下（local），java连接spark调用scala语言打成的jar包 #

![java.jpg][]

一，环境构建

1、spark版本![70][]

2、Scala语言打成的jar，因为Scala和java都是运行在jvm上的字节码类语言，java可以直接调用。![70 1][]，当然该包中包含了一些算法，LR、GBDT、决策树、随机森林，以便于数据的训练。

3、maven webapp项目，pom引入包如下：

<dependency>  
<groupId>org.apache.spark</groupId>  
<artifactId>spark-core\_2.11</artifactId>  
<version>$\{spark.version\}</version>  
</dependency>  
  
<dependency>  
<groupId>org.apache.spark</groupId>  
<artifactId>spark-sql\_2.11</artifactId>  
<version>$\{spark.version\}</version>  
</dependency>  
<dependency>  
<groupId>org.apache.spark</groupId>  
<artifactId>spark-mllib\_2.11</artifactId>  
<version>$\{spark.version\}</version>  
</dependency>

版本是：<spark.version>2.2.0</spark.version>

3.spark连接工具类：

package cfca.xfraud.afmls.util.spark;

import org.apache.spark.SparkContext;  
import org.apache.spark.sql.SparkSession;  
import cfca.xfraud.afmls.common.Constants;  
import cfca.xfraud.afmls.exception.CodeException;  
import cfca.xfraud.dataprocess.DataProcess;  
import cfca.xfraud.model.DesicionTree;  
import cfca.xfraud.model.GBDTree;  
import cfca.xfraud.model.LR;  
import cfca.xfraud.model.RFTree;  
/\*\*  
\* @Author zhangzhongqiu  
\* @Description spark工具类  
\* @version 1.2.0  
\* @since jdk1.8  
\* @date 2018-03-06  
\*  
\*  
\*/  
public class SparkUtil \{  
**private static SparkSession sparkSession;  
static\{  
SparkContext sparkContext = new SparkContext(Constants.SPARK\_LOCAL\_PATTERN, Constants.APP\_NAME);  
sparkSession = new SparkSession(sparkContext);  
\}**  
/\*\*  
\* 数据预处理运行状态  
\* @param appName  
\* @param dataPattern  
\* @return  
\* @throws Exception  
\* @throws CodeException  
\*/  
public static boolean runDataProcessStatus(int dataPattern,String trainTime) throws Exception \{  
boolean isSuccess=false;  
try \{  
isSuccess = new DataProcess(sparkSession).runDataProcess(dataPattern,trainTime);  
\} catch (Exception e) \{  
throw e;  
\}  
return isSuccess;  
\}

/\*\*  
\* 数据训练过程  
\* @param algorithmPattern  
\* @param depth  
\* @param trainTime  
\* @return  
\* @throws Exception  
\*/  
public static boolean runTrainData(String algorithmPattern,int depth,String trainTime) throws Exception\{  
boolean isRunSuccess=false;

try \{

if(Constants.STR\_ZERO.equals(algorithmPattern))\{  
isRunSuccess = new LR(sparkSession).lrClassificationModel(20,0.2,trainTime);

if(Constants.STR\_ONE.equals(algorithmPattern))\{  
isRunSuccess = new DesicionTree(sparkSession).decisionTreeClassificationModel(20,trainTime);

if(Constants.STR\_TWO.equals(algorithmPattern))\{  
isRunSuccess=new GBDTree(sparkSession).GBDTreeClassificationModel(6,20,trainTime);

if(Constants.STR\_THREE.equals(algorithmPattern))\{  
isRunSuccess = new RFTree(sparkSession).rfTreeClassificationModel(100,trainTime);  
\}  
\} catch (Exception e) \{  
throw e;  
\}  
return isRunSuccess;  
\}  
\}

主要是建立local模式下的sparkSession作为凭证，进行操作，蓝色字体来自于Scala包中的代码，只需要在相应的类中提供函数入口就好了。

原文地址http://www.bieryun.com/2457.html

[java.jpg]: /images/20220206/c4af4cf2a0574bbf99c43593cea12a70.png
[70]: /images/20220206/b582b348121b45afb0b4535173a20385.png
[70 1]: /images/20220206/b33e261a7150452aba01d451efa7755b.png