我正在尝试使用 spark-submit 命令在本地机器的 linux 终端中运行我的 jar 文件
> spark-submit
> --master local
> --class RTP
> --packages com.typesafe:config:1.3.2,org.apache.kafka:kafka-clients:2.0.0,
commons-codec:commons-codec:1.9,org.scalaj:scalaj-http_2.11:2.3.0,
com.typesafe.play:play-json:2.4.0
> target/scala-2.11/structured_streaming_2.11-0.1.jar
但是程序失败，我得到以下输出
Ivy Default Cache set to: /home/user/.ivy2/cache
The jars for the packages stored in: /home/user/.ivy2/jars
:: loading settings :: url = jar:file:/home/user/spark-2.3.3-bin-hadoop2.7/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.typesafe#config added as a dependency
org.apache.kafka#kafka-clients added as a dependency
commons-codec#commons-codec added as a dependency
org.scalaj#scalaj-http_2.11 added as a dependency
com.typesafe.play#play-json added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-975a9ed1-7824-48a2-81df-93814b2aa810;1.0
confs: [default]
found com.typesafe#config;1.3.2 in spark-list
found org.apache.kafka#kafka-clients;2.0.0 in spark-list
found org.lz4#lz4-java;1.4.1 in spark-list
found org.xerial.snappy#snappy-java;1.1.7.1 in spark-list
found org.slf4j#slf4j-api;1.7.25 in spark-list
found commons-codec#commons-codec;1.9 in spark-list
found org.scalaj#scalaj-http_2.11;2.3.0 in spark-list
:: resolution report :: resolve 905ms :: artifacts dl 6ms
:: modules in use:
com.typesafe#config;1.3.2 from spark-list in [default]
commons-codec#commons-codec;1.9 from spark-list in [default]
org.apache.kafka#kafka-clients;2.0.0 from spark-list in [default]
org.lz4#lz4-java;1.4.1 from spark-list in [default]
org.scalaj#scalaj-http_2.11;2.3.0 from spark-list in [default]
org.slf4j#slf4j-api;1.7.25 from spark-list in [default]
org.xerial.snappy#snappy-java;1.1.7.1 from spark-list in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 8 | 0 | 0 | 0 || 7 | 0 |
---------------------------------------------------------------------
:: problems summary ::
:::: WARNINGS
module not found: com.typesafe.play#play-json;2.4.0
==== local-m2-cache: tried
file:/home/sushil/.m2/repository/com/typesafe/play/play-json/2.4.0/play-json-2.4.0.pom
-- artifact com.typesafe.play#play-json;2.4.0!play-json.jar:
file:/home/sushil/.m2/repository/com/typesafe/play/play-json/2.4.0/play-json-2.4.0.jar
==== local-ivy-cache: tried
/home/sushil/.ivy2/local/com.typesafe.play/play-json/2.4.0/ivys/ivy.xml
-- artifact com.typesafe.play#play-json;2.4.0!play-json.jar:
/home/sushil/.ivy2/local/com.typesafe.play/play-json/2.4.0/jars/play-json.jar
==== central: tried
https://repo1.maven.org/maven2/com/typesafe/play/play-json/2.4.0/play-json-2.4.0.pom
-- artifact com.typesafe.play#play-json;2.4.0!play-json.jar:
https://repo1.maven.org/maven2/com/typesafe/play/play-json/2.4.0/play-json-2.4.0.jar
==== spark-packages: tried
http://dl.bintray.com/spark-packages/maven/com/typesafe/play/play-json/2.4.0/play-json-2.4.0.pom
-- artifact com.typesafe.play#play-json;2.4.0!play-json.jar:
http://dl.bintray.com/spark-packages/maven/com/typesafe/play/play-json/2.4.0/play-json-2.4.0.jar
::::::::::::::::::::::::::::::::::::::::::::::
:: UNRESOLVED DEPENDENCIES ::
::::::::::::::::::::::::::::::::::::::::::::::
:: com.typesafe.play#play-json;2.4.0: not found
::::::::::::::::::::::::::::::::::::::::::::::
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: com.typesafe.play#play-json;2.4.0: not found]
at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1303)
at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:53)
at org.apache.spark.deploy.SparkSubmit$.doPrepareSubmitEnvironment(SparkSubmit.scala:364)
at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:250)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:171)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
当我在本地计算机上运行 spark-submit version 命令时，我得到以下内容
Welcome to
____ __
/ __/__ ___ _____/ /__
_ / _ / _ `/ __/ '_/
/___/ .__/_,_/_/ /_/_ version 2.3.3
/_/
Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_252
该程序在Intellij上运行良好
我的 build.sbt 文件的详细信息如下
name := "Structured_Streaming"
version := "0.1"
scalaVersion := "2.11.12"
libraryDependencies += "com.typesafe" % "config" % "1.3.2"
libraryDependencies += "org.apache.kafka" % "kafka-clients" % "2.0.0"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.0"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.4.0"
libraryDependencies += "org.apache.spark" % "spark-sql-kafka-0-10_2.11" % "2.4.0"
libraryDependencies += "commons-codec" % "commons-codec" % "1.9"
libraryDependencies += "org.scalaj" % "scalaj-http_2.11" % "2.3.0"
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.4.0"
在--packages中，您有com.typesafe.play:play-json:2.4.0而不是com.typesafe.play:play-json_2.11:2.4.0，因此您正在从中获取内容
https://repo1.maven.org/maven2/com/typesafe/play/play-json/2.4.0/
而不是
https://repo1.maven.org/maven2/com/typesafe/play/play-json_2.11/2.4.0/