Apache Spark - 快速入门"java.lang.NoClassDefFoundError: scala/Serializable"

Question

我正在尝试遵循这个指南https://spark.apache.org/docs/latest/quick-start.html (scala)。但是，当我应该将jar文件提交给spark时，我无法完成最后一步。

# Use spark-submit to run your application
$ YOUR_SPARK_HOME/bin/spark-submit 
--class "SimpleApp" 
--master local[4] 
target/scala-2.12/simple-project_2.12-1.0.jar

得到以下异常

Exception in thread "main" java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/Serializable
at SimpleApp$.main(SimpleApp.scala:9)
at SimpleApp.main(SimpleApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NoClassDefFoundError: scala/Serializable
... 14 more
Caused by: java.lang.ClassNotFoundException: scala.Serializable
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 14 more

你知道这是什么原因吗?

Answer 1

我找到问题了。我安装了错误的Spark版本。我已经下载了"预构建的Apache Hadoop 3.3及更高版本(scala 2.13)"&;的版本。安装"预构建的Apache Hadoop 3.3及更高版本";Spark版本解决了问题

Answer 2

您需要将依赖项升级到与您的Scala版本二进制兼容的版本。2.12在本例中