org.apache.spark.SparkException: Task not serializable

野性酷女 2022-09-04 03:46 113阅读 0赞

spark代码所在的类需要序列化，否则会报错；

即使调用的方法spark代码所在类序列化了，其他spark方法所在的。类没有序列化也会影响。  
had a not serializable result: org.locationtech.geomesa.features.ScalaSimpleFeature  
Serialization stack:  
    - object not serializable (class: org.locationtech.geomesa.features.ScalaSimpleFeature, value: ScalaSimpleFeature:1423477451299749888:1423477451299749888|1624094042320|1624094047600|Mon Aug 09 14:02:35 CST 2021|LINESTRING (121.24744089367137 31.292577146641687, 121.24743638852598 31.292565211676482, 121.45075807931028 31.16781355321965, 121.44961902907794 31.167363434902484, 121.44854210916515 31.16679122385797))  
    - element of array (index: 0)  
    - array (class \[Lorg.opengis.feature.simple.SimpleFeature;, size 47); not retrying  
2021-08-11 18:12:20.181  INFO 13968 --- \[io-12001-exec-6\] org.apache.spark.scheduler.DAGScheduler  : Job 75 failed: collect at GeomesaSparkService.java:116, took 0.062221 s

Spark报错:org.apache.spark.SparkException: Task not serializable

一眼就能看出是没有进行序列化，在spark程序中，所有的class都要序列化。如果不需要序列化的用transient修饰

修改后的代码:

package com.example.controller;  
   
import org.apache.spark.api.java.JavaPairRDD;  
import org.apache.spark.api.java.JavaRDD;  
import org.apache.spark.api.java.JavaSparkContext;  
import org.apache.spark.api.java.function.FlatMapFunction;  
import org.apache.spark.api.java.function.Function2;  
import org.apache.spark.api.java.function.PairFunction;  
import org.springframework.beans.factory.annotation.Autowired;  
import org.springframework.web.bind.annotation.GetMapping;  
import org.springframework.web.bind.annotation.RequestMapping;  
import org.springframework.web.bind.annotation.RestController;  
import scala.Tuple2;  
   
import java.io.IOException;  
import java.io.Serializable;  
import java.util.Arrays;  
import java.util.Iterator;  
import java.util.List;  
   
/\*\*  
 \* @ClassName SparkTest  
 \* @Author yupanpan  
 \* @Date 2020/5/25 16:58  
 \*/  
@RestController  
@RequestMapping("/spark")  
public class SparkTest implements Serializable \{  
   
    @Autowired  
    private transient JavaSparkContext sparkContext;  
   
    @GetMapping("/test")  
    public void test() throws IOException \{  
   
        //创建RDD：从文件中读取待统计文本  
        JavaRDD<String> lines = sparkContext.textFile("D:\\\\yop\_sdk\_config\_default.json");  
   
        //转换RDD：将文本内容按空格分割成单词  
        JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>() \{  
            @Override  
            public Iterator<String> call(String s) \{  
                return Arrays.asList(s.split("\\\\s")).iterator();  
            \}  
        \});  
   
        //转换RDD：将单词<String>转换成<String,Integer>,每个单词初始化次数为1  
        JavaPairRDD<String, Integer> ones = words.mapToPair(new PairFunction<String, String, Integer>() \{  
            @Override  
            public Tuple2<String, Integer> call(String s) \{  
                return new Tuple2<String, Integer>(s, 1);  
            \}  
        \});  
   
        //行动RDD：统计每个单词的次数<String,Integer>  
        JavaPairRDD<String, Integer> counts = ones.reduceByKey(new Function2<Integer, Integer, Integer>() \{  
            @Override  
            public Integer call(Integer i1, Integer i2) \{  
                return i1 + i2;  
            \}  
        \});  
   
        //获取整个RDD的数据  
        List<Tuple2<String, Integer>> output = counts.collect();  
        for (Tuple2<?, ?> tuple : output) \{  
            System.out.println(tuple.\_1() + ": " + tuple.\_2());  
        \}  
        sparkContext.stop();  
        sparkContext.close();  
    \}  
\}

这里只是测试，第一是controller没有序列化，序列化之后，还是报错，然后JavaSparkContext没有序列化，Spark程序处理成员变量也是需要序列化的，由于我是@Configuration @Bean形式先配置了的，注入了sparkContext，sparkContext本身是不需要序列化，加上transient，问题解决

结果:

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lwcDkxenI_size_16_color_FFFFFF_t_70][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lwcDkxenI_size_16_color_FFFFFF_t_70]: /images/20220829/cd6a4fe941a9441292acf11502d60fec.png