我尝试使用cassandrasqlcontext从Spark查询Cassandra，但是我会发现一个奇怪的丢失依赖关系错误。我有一个如下的火花应用程序：
val spark: SparkSession = SparkSession.builder().appName(appName).getOrCreate()
val cassandraSQLContext = new org.apache.spark.sql.cassandra.CassandraSQLContext(spark.sparkContext)
val path = args(0)
cassandraSQLContext.setKeyspace(args(1))
val dataFrame: DataFrame = cassandraSQLContext.sql(args(2))
dataFrame.write.mode(SaveMode.Overwrite).option("header", "true").csv(path)
我会发现一个缺少的火花Scala类错误：
User class threw exception: java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/analysis/Catalog
at com.test.batch.utils.CSVFromCassandraSQLQuery$.<init>(CSVFromCassandraSQLQuery.scala:19)
at com.test.batch.utils.CSVFromCassandraSQLQuery$.<clinit>(CSVFromCassandraSQLQuery.scala)
at com.test.batch.utils.CSVFromCassandraSQLQuery.main(CSVFromCassandraSQLQuery.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.catalyst.analysis.Catalog
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 8 more
我还尝试在spark-submit命令中明确添加spark-catalyst JAR路径，但是我仍然遇到相同的问题（无论是在本地还是在纱线群集上运行）...
这是我的项目设置build.sbt：
scalaVersion := "2.11.11"
val sparkVersion = "2.3.1"
libraryDependencies ++= Seq(
"log4j" % "log4j" % "1.2.17",
"org.apache.spark" %% "spark-core" % sparkVersion % "provided",
"org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
"org.apache.spark" %% "spark-hive" % sparkVersion % "provided",
"org.apache.spark" %% "spark-catalyst" % sparkVersion % "provided",
"org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",
"com.datastax.spark" %% "spark-cassandra-connector" % "2.3.1",
"org.scala-lang.modules" %% "scala-java8-compat" % "0.9.0",
"com.twitter" % "jsr166e" % "1.1.0"
)
对我缺少的东西有什么想法？
此问题正在发生，因为org.apache.spark.sql.catalyst.analysis. catalog class class> class> class> class> class> class> class> class> class> class> class> cass> spark> spark-catalyst_2.11：2.x.x 。请参阅 Spark-catalyst_2.11：2.0.0
的源代码https://jar-download.com/artifacts/org.apache.spark/spark-catalyst_2.11/2.11/2.0.0/source-code
org.apache.spark.sql.catalyst.Analysis.Catalog class class class class可以使用 spark-catalyst_2.11：1.6.3 版本。
https://jar-download.com/artifacts/org.apache.spark/spark-catalyst_2.11/1.6.3/source-code
我会要求您不使用 cassandrasqlcontext 被弃用。请检查https://datastax-oss.atlassian.net/browse/sparkc-399。
请检查以便发布以使用 spark 2.0 。
中的cassandra上下文如何在Spark 2.0