scala 2.12 + Spark 3 + sedona-sql-3.0_2.12

Question

我正在尝试将sedona与scala和spark一起使用。这是build.sbt文件：

ThisBuild / scalaVersion := "2.12.12"
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.12" % "3.0.1",
"org.apache.spark" % "spark-sql_2.12" % "3.0.1",
"org.apache.sedona" % "sedona-python-adapter-2.4_2.11" % "1.2.1-incubating",
"org.apache.sedona" % "sedona-core-3.0_2.12" % "1.2.1-incubating",
"org.apache.sedona" % "sedona-sql-3.0_2.12" % "1.2.1-incubating",
"org.apache.sedona" % "sedona-viz-2.4_2.11" % "1.2.1-incubating"
)

该代码与scala2.11&spark 2.4，但当我切换到spark 3时，我在执行代码时出现以下错误：

[error] Provider org.apache.spark.sql.sedona_sql.io.GeotiffFileFormat could not be instantiated

Caused by: java.lang.NoClassDefFoundError: org/apache/spark/sql/execution/datasources/FileFormat$class
[error]         at org.apache.spark.sql.sedona_sql.io.GeotiffFileFormat.<init>(GeotiffFileFormat.scala:54)

有什么想法吗？

Answer 1

根据https://sedona.apache.org/setup/maven-coordinates/#use-sedona脂肪罐，你只需要3罐

请不要将其他jar添加到您的依赖项中。

<dependency>
<groupId>org.apache.sedona</groupId>
<artifactId>sedona-python-adapter-3.0_2.12</artifactId>
<version>1.2.1-incubating</version>
</dependency>
<dependency>
<groupId>org.apache.sedona</groupId>
<artifactId>sedona-viz-3.0_2.12</artifactId>
<version>1.2.1-incubating</version>
</dependency>
<!-- Optional: https://mvnrepository.com/artifact/org.datasyslab/geotools-wrapper -->
<dependency>
<groupId>org.datasyslab</groupId>
<artifactId>geotools-wrapper</artifactId>
<version>1.1.0-25.2</version>
</dependency>