Spark 2.2 的 SBT 编译失败

Question

我刚刚开始在HDP 2.2上使用Spark 2.6，我在尝试进行sbt编译时遇到了问题

错误

[info] 更新了文件/home/maria_dev/structuredstreaming/project/build.properties：将 sbt.version 设置为 1.3.0 [信息]从/home/maria_dev/structuredstreaming/project 加载项目定义 [信息]获取工件 [信息]获取的工件 [error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts： 错误获取工件： [错误] https://repo1.maven.org/maven2/com/squareup/okhttp3/okhttp-urlconnection/3.7.0/okhttp-urlconnection-3.7.0.jar：下载错误：捕获 java.net.UnknownHost异常：下载 https://repo1.maven.org/maven2/com/squareup/okhttp3/okhttp-urlconnection/3.7.0/okhttp-urlconnection-3.7.0.jar 时 repo1.maven.org (repo1.maven.org(

build.sbt 文件如下

buid.sbt

scalaVersion := "2.11.8"
resolvers ++= Seq(
"Conjars" at "http://conjars.org/repo",
"Hortonworks Releases" at "http://repo.hortonworks.com/content/groups/public"
)
publishMavenStyle := true
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.2.0.2.6.3.0-235",
"org.apache.spark" %% "spark-sql" % "2.2.0.2.6.3.0-235",
"org.apache.phoenix" % "phoenix-spark2" % "4.7.0.2.6.3.0-235",
"org.apache.phoenix" % "phoenix-core" % "4.7.0.2.6.3.0-235",
"org.apache.kafka" % "kafka-clients" % "0.10.1.2.6.3.0-235",
"org.apache.spark" %% "spark-streaming" % "2.0.2" % "provided",
"org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.0.2",
"org.apache.spark" %% "spark-sql-kafka-0-10" % "2.0.2" % "provided",
"com.typesafe" % "config" % "1.3.1",
"com.typesafe.play" %% "play-json" % "2.7.2",
"com.solarmosaic.client" %% "mail-client" % "0.1.0",
"org.json4s" %% "json4s-jackson" % "3.2.10",
"org.apache.logging.log4j" % "log4j-api-scala_2.11" % "11.0",
"com.databricks" %% "spark-avro" % "3.2.0",
"org.elasticsearch" %% "elasticsearch-spark-20" % "5.0.0-alpha5",
"io.spray" %%  "spray-json" % "1.3.3"
)
retrieveManaged := true
fork in run := true

Answer 1

看起来 coursier 正在尝试从被阻止 repo1.maven.org 中获取依赖项。Scala-Metals人在这里解释。基本上，您必须通过设置如下所示的mirror.properties文件来设置指向公司代理服务器的全局 Coursier 配置：

central.from=https://repo1.maven.org/maven2
central.to=http://mycorporaterepo.com:8080/nexus/content/groups/public

根据您的操作系统，它将是：

• Windows： C：\Users\\AppData\Roaming\Coursier\config\mirror.properties
• Linux： ~/.config/coursier/mirror.properties
• MacOS： ~/Library/Preferences/Coursier/mirror.properties

您可能还需要设置 SBT 以使用代理下载依赖项。为此，您需要编辑此文件：

~/.sbt/repositories

将其设置为以下内容：

[repositories]
local
maven-central: http://mycorporaterepo.com:8080/nexus/content/groups/public

这两个设置的组合应该处理将 SBT 指向正确位置所需的一切。