通过火花包装使用闪闪发光的水:com.google.guava ..找不到



我正在尝试通过火花包使用h2o.ai的闪闪发光水。我遵循他们的指南:https://github.com/h2oai/sparkling-water#use-sparkling-water-via-spark-carkages,我在Hortonworks HDP 2.4上,带有Scala 2.10和Spark 1.6.1。我将以下内容放在终端:

spark-shell --packages ai.h2o:sparkling-water-core_2.10:1.6.8,ai.h2o:sparkling-water-examples_2.10:1.6.8

但是,它抱怨找不到两个文物。这里有详细信息:

:: problems summary ::
:::: WARNINGS
        [NOT FOUND  ] com.google.guava#guava;16.0.1!guava.jar(bundle) (1ms)
==== local-m2-cache: tried
  file:/root/.m2/repository/com/google/guava/guava/16.0.1/guava-16.0.1.jar
    [NOT FOUND  ] com.google.code.findbugs#jsr305;3.0.0!jsr305.jar (1ms)
==== local-m2-cache: tried
  file:/root/.m2/repository/com/google/code/findbugs/jsr305/3.0.0/jsr305-3.0.0.jar
    ::::::::::::::::::::::::::::::::::::::::::::::
    ::              FAILED DOWNLOADS            ::
    :: ^ see resolution messages for details  ^ ::
    ::::::::::::::::::::::::::::::::::::::::::::::
    :: com.google.guava#guava;16.0.1!guava.jar(bundle)
    :: com.google.code.findbugs#jsr305;3.0.0!jsr305.jar
    ::::::::::::::::::::::::::::::::::::::::::::::

:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [download failed: com.google.guava#guava;16.0.1!guava.jar(bundle), download failed: com.google.code.findbugs#jsr305;3.0.0!jsr305.jar]
    at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1068)
    at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:287)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:154)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

有人有任何经验我如何解决?非常感谢。

删除~/.m2~/ivy2使用ai.h2o:sparkling-water-core_2.10:1.6.8

修复了问题

如果您查找错误"下载失败:com.google.guava#guava; 16.0.1!guava.jar.jar,您将看到它主要是特定于环境的问题在启动的应用程序期间,无法下载软件包。

您的解决方案可能是下载/安装两个失败的下载软件包,如下所示:

$ mvn install com.google.guava:guava:21.0:get -DrepoUrl=https://mvnrepository.com

看起来像降低jar文件的版本

spark-shell --packages ai.h2o:sparkling-water-core_2.10:1.6.1,ai.h2o:sparkling-water-examples_2.10:1.6.1

或使用苏打水小册子第5.1.1节中建议的fatjar

spark-shell --jars [your path]/sparkling-water-1.6.8/assembly/build/libs/sparkling-water-assembly-1.6.8-all.jar

解决了问题。从H2O

下载sparkling-water-1.6.8的位置

相关内容

最新更新