Spark Kafka Streaming 给出了不兼容的杰克逊例外

Question

这是我收到的错误：

java.lang.ExceptionInInitializerError
    at org.apache.spark.streaming.dstream.InputDStream.<init>(InputDStream.scala:78)
    at org.apache.spark.streaming.kafka010.DirectKafkaInputDStream.<init>(DirectKafkaInputDStream.scala:62)
    at org.apache.spark.streaming.kafka010.KafkaUtils$.createDirectStream(KafkaUtils.scala:150)
    at org.apache.spark.streaming.kafka010.KafkaUtils$.createDirectStream(KafkaUtils.scala:127)
    at chirpanywhere.stream.classification.service.streaming.StreamingClassificationPipeline.createStreamAndTransform(StreamingClassificationPipeline.scala:69)
    at chirpanywhere.stream.classification.service.streaming.StreamingClassificationPipeline.streamAndUpdateMLModelCache(StreamingClassificationPipeline.scala:58)
    at chirpanywhere.stream.classification.service.streaming.StreamingClassificationPipeline.run(StreamingClassificationPipeline.scala:54)
    at chirpanywhere.stream.classification.service.Boot$.main(Boot.scala:10)
    at chirpanywhere.stream.classification.service.Boot.main(Boot.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.7.8
    at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)
    at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
    at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:730)
    at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
    at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
    at org.apache.spark.streaming.dstream.InputDStream.<init>(InputDStream.scala:78)
    at org.apache.spark.streaming.kafka010.DirectKafkaInputDStream.<init>(DirectKafkaInputDStream.scala:62)
    at org.apache.spark.streaming.kafka010.KafkaUtils$.createDirectStream(KafkaUtils.scala:150)
    at org.apache.spark.streaming.kafka010.KafkaUtils$.createDirectStream(KafkaUtils.scala:127)
    at chirpanywhere.stream.classification.service.streaming.StreamingClassificationPipeline.createStreamAndTransform(StreamingClassificationPipeline.scala:69)
    at chirpanywhere.stream.classification.service.streaming.StreamingClassificationPipeline.streamAndUpdateMLModelCache(StreamingClassificationPipeline.scala:58)
    at chirpanywhere.stream.classification.service.streaming.StreamingClassificationPipeline.run(StreamingClassificationPipeline.scala:54)
    at chirpanywhere.stream.classification.service.Boot$.main(Boot.scala:10)
    at chirpanywhere.stream.classification.service.Boot.main(Boot.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)

有人可以告诉我有什么问题。

我的build.sbt看起来像这样：

   val akkaHttpV = "10.0.0"
  val sparkV = "2.1.0"
 "log4j" % "log4j" % "1.2.17",
    "com.typesafe.akka" %% "akka-http" % akkaHttpV,
    "com.typesafe.akka" %% "akka-http-spray-json" % akkaHttpV,
    "org.slf4j" % "slf4j-api" % "1.7.21",
    "org.apache.spark" %% "spark-core" % sparkV,
    "org.apache.spark" %% "spark-sql" % sparkV,
    "org.apache.spark" %% "spark-streaming" % sparkV,
    "org.apache.spark" %% "spark-mllib" % sparkV,
    "com.github.blemale" %% "scaffeine" % "2.0.0" % "compile",
    "org.apache.spark" %% "spark-streaming-kafka-0-10" % sparkV ,
    "com.datastax.spark" %% "spark-cassandra-connector" % "2.0.0-M3",
    "org.json4s" %% "json4s-native" % "3.5.0" ,
    "org.scalatest" %% "scalatest" % "3.0.0" % "test",

Answer 1

Jackson Scala 模块在检查与其一起部署的jackson-databind版本时会引发异常。Scala 模块需要相同的次要版本;在这种情况下，异常消息不完整(它应该包括两个版本(——我猜你有更新的jackson-databind(2.7.8 是一个很好的最新版本(和较旧的 Scala 模块，也许是一个 Spark 捆绑包(2.5.3？

Scala 模块

在这里非常谨慎，因为通常"足够接近"的次要版本确实有效：但解决方案是确保 databind 和 Scala 模块具有相同的次要版本(也可能具有相同的补丁版本(。