运行spark作业使用Yarn给出错误:com. google.common.uti.concurrent . futu

Question

我正在尝试使用yarn运行spark job，但得到以下错误

java.lang.NoSuchMethodError: com.google.common.util.concurrent.Futures.withFallback(Lcom/google/common/util/concurrent/ListenableFuture;Lcom/google/common/util/concurrent/FutureFallback;Ljava/util/concurrent/Executor;)Lcom/google/common/util/concurrent/ListenableFuture;
at com.datastax.driver.core.Connection.initAsync(Connection.java:176)
at com.datastax.driver.core.Connection$Factory.open(Connection.java:721)
at com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:248)
at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:194)
at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:82)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1307)
at com.datastax.driver.core.Cluster.init(Cluster.java:159)
at com.datastax.driver.core.Cluster.connect(Cluster.java:249)
at com.figmd.processor.ProblemDataloader$ParseJson.call(ProblemDataloader.java:46)
at com.figmd.processor.ProblemDataloader$ParseJson.call(ProblemDataloader.java:34)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$4$1.apply(JavaRDDLike.scala:140)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$4$1.apply(JavaRDDLike.scala:140)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:618)
at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:618)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

集群细节:Spark 1.2.1,hadoop 2.7.1我已经使用spark.driver.extraClassPath提供了类路径。Hadoop用户也可以访问这个类路径。但是我认为yarn没有在那个类路径上得到JAR。我找不到它的根本原因。如有任何帮助，不胜感激。

谢谢。

Answer 1

我遇到了同样的问题，解决方案是遮荫番石榴以避免classpath碰撞。

如果你正在使用sbt程序集来构建你的jar，你可以把这个添加到你的build.sbt:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.**" -> "shadeio.@1").inAll
)

我写了一篇博文，描述了我得到这个解决方案的过程:让Hadoop 2.6 + Spark-Cassandra Driver玩得很好。

希望有帮助!

Answer 2

问题与番石榴版本不匹配有关。

withFallback被添加到版本14的Guava。看起来你有番石榴

Answer 3

添加到@Arjones的回答，如果你正在使用gradle + GradleShadow，你可以添加到你的构建。

shadowJar {
    relocate 'com.google.common', 'com.example.com.google.common'
}