scala + spingboot + springcloud + spark jackson和slf4j冲突解决

浅浅的花香味﹌ 2022-05-25 13:26 244阅读 0赞

# scala + spingboot + springcloud + spark jackson和slf4j冲突解决 #

*  scala + spingboot + springcloud + spark jackson和slf4j冲突解决
    
     *  环境
     *  问题的出现
     *  解决过程
     *  dependency
     *  总结

## 环境 ##

*  `jdk`:`1.8`
 *  `spring-boot`:`1.5.2`
 *  `scala`:`2.11.8`
 *  `spark`:`2.2.1`
 *  `bt`:`maven`

## 问题的出现 ##

正常搭建`spring-boot + scala`项目，测试正常，在加入`spark`依赖后，`spring-boot`可以启动，但是会报错：

SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/C:/Users/vabsh/.m2/repository/ch/qos/logback/logback-classic/1.1.11/logback-classic-1.1.11.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/C:/Users/vabsh/.m2/repository/org/slf4j/slf4j-log4j12/1.7.21/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]

执行`spark`任务失败，报错如下：

Exception in thread "Thread-7" java.lang.ExceptionInInitializerError
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
        at org.apache.spark.rdd.RDD.takeSample(RDD.scala:557)
        at com.mongodb.spark.sql.MongoInferSchema$.apply(MongoInferSchema.scala:70)
        at com.mongodb.spark.MongoSpark.toDF(MongoSpark.scala:581)
        at com.mongodb.spark.MongoSpark$.load(MongoSpark.scala:84)
        at github.clyoudu.test.executor.SystemEventTopNExecutor.execute(SystemEventTopNExecutor.scala:21)
        at github.clyoudu.test.controller.ComputeTriggerController$$anon$1.run(ComputeTriggerController.scala:40)
        at java.lang.Thread.run(Thread.java:748)
    Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.8.7
        at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)
        at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
        at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:745)
        at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
        at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
        ... 8 more

大概可以看出是`jackson`版本的问题，要么是`spark 2.2.1`无法使用`jackson 2.8.7`，要么是`spark`中的某些依赖包无法使用`jackson 2.8.7`。

## 解决过程 ##

首先用`mvn dependency:tree Dincludes=com.fasterxml.jackson.core`命令查看编译的包从哪里来，结果如下：

[INFO] github.clyoudu.test:offlineCompute:jar:1.0-SNAPSHOT
    [INFO] \- org.springframework.boot:spring-boot-starter-web:jar:1.5.2.RELEASE:compile
    [INFO]    \- com.fasterxml.jackson.core:jackson-databind:jar:2.8.7:compile
    [INFO]       +- com.fasterxml.jackson.core:jackson-annotations:jar:2.8.0:compile
    [INFO]       \- com.fasterxml.jackson.core:jackson-core:jar:2.8.7:compile

发现是来自`spring-boot-starter-web`，去`mvnrepository`查看[Spring Boot» 1.5.2.RELEASE][Spring Boot_ 1.5.2.RELEASE]，发现真是`jackson-databind-2.8.7`。

第一步想到的是能不能降`spring-boot`版本，让`spring-boot`和`spark`的`jackson`版本相同，浏览了几个版本的`spring-boot`，发现`1.3.2.RELEASE`的`jackson`是`2.5.6`，更改`spring-boot`版本，发现可以启动，并且`spark`任务未见异常。

因此确定是因为`jackson`版本冲突导致的`spark`任务执行失败，于是换回`spring-boot 1.5.2.RELEASE`，强行指定`spring-boot`和`spark`的`jackson`版本。

<dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.8.7</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-core</artifactId>
        <version>2.8.7</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.module</groupId>
        <artifactId>jackson-module-scala_2.11</artifactId>
        <version>2.8.7</version>
    </dependency>

再次执行`mvn dependency:tree Dincludes=com.fasterxml.jackson.core`：

[INFO] github.clyoudu.test:offlineCompute:jar:1.0-SNAPSHOT
    [INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.8.7:compile
    [INFO] |  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.8.0:compile
    [INFO] \- com.fasterxml.jackson.core:jackson-core:jar:2.8.7:compile

所有`jar`都来自指定的版本了。

不过又出了一个问题：`logback`失效，日志未能写到对应文件夹，非常奇怪；做了一些`exclusion`后甚至会报因为包冲突导致的`ClassNotFoundException`。

这时想到前面看到的`slf4j`冲突，因为`spring-boot`自己带有`logback`和`sl4j`，但是`spark-core`又有`slf4j-log4j12`，于是排除`spark-core`的`slf4j-log4j12`：

<dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.2.1</version>
        <exclusions>
            <exclusion>
                <artifactId>slf4j-log4j12</artifactId>
                <groupId>org.slf4j</groupId>
            </exclusion>
        </exclusions>
    </dependency>

重启后，一切正常，`spring-boot`正常启动，`eureka server`注册成功，`spark`任务执行成功，`logback`运行正常。

## dependency ##

<dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.11</version>
            <scope>test</scope>
        </dependency>
    
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <version>1.5.2.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-eureka</artifactId>
            <version>1.3.2.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
            <version>1.5.2.RELEASE</version>
        </dependency>
    
        
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
        </dependency>
    
        
        <dependency>
            <groupId>org.mongodb.spark</groupId>
            <artifactId>mongo-spark-connector_2.11</artifactId>
            <version>2.2.1</version>
        </dependency>
    
        
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.2.1</version>
            <exclusions>
                <exclusion>
                    <artifactId>slf4j-log4j12</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>2.2.1</version>
        </dependency>
        <dependency>
            <groupId>org.codehaus.janino</groupId>
            <artifactId>commons-compiler</artifactId>
            <version>2.7.8</version>
        </dependency>
    
        
        <dependency>
            <groupId>org.mongodb</groupId>
            <artifactId>casbah-core_2.11</artifactId>
            <version>3.0.0</version>
        </dependency>
    
        
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.47</version>
        </dependency>
    
        
        <dependency>
            <groupId>com.github.nscala-time</groupId>
            <artifactId>nscala-time_2.11</artifactId>
            <version>2.18.0</version>
        </dependency>
    
        
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.8.7</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
            <version>2.8.7</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.module</groupId>
            <artifactId>jackson-module-scala_2.11</artifactId>
            <version>2.8.7</version>
        </dependency>
    </dependencies>

## 总结 ##

*  `ClassNotFoundException`要么是真的没添加依赖，要么是包冲突。
 *  解决版本冲突除了`exclusion`，还可以在`pom`里强制指定。
 *  一个包被很多个包依赖的情况下，`exlusion`可能并不好用。

[Spring Boot_ 1.5.2.RELEASE]: https://mvnrepository.com/artifact/org.springframework.boot/spring-boot/1.5.2.RELEASE