安装后尝试打开 Spark 并收到错误:找不到任何与版本 "1.8" 匹配的 JVM



描述:

我使用Homebrew在我的MacBook上安装了spark。我遵循了以下指导过程:https://www.tutorialkart.com/apache-spark/how-to-install-spark-on-mac-os/.

这个循序渐进的过程包括安装Java,然后是Scala和Spark。Java和Scala安装成功。Spark也成功安装。

当我尝试使用下面的输入命令验证spark安装时,我遇到了一个错误。

输入命令:spark-shell

预期行为:预计Spark将在终端上启动

实际行为:我得到以下错误:

Unable to find any JVMs matching version "1.8".
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/Cellar/apache-spark/2.4.5/libexec/jars/spark-unsafe_2.11-2.4.5.jar) to method java.nio.Bits.unaligned()
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:791)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2422)
at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:79)
at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:348)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$secMgr$1(SparkSubmit.scala:348)
at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:356)
at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:356)
at scala.Option.map(Option.scala:146)
at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:355)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:774)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2
at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3756)
at java.base/java.lang.String.substring(String.java:1902)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:52)

我尝试了什么:

我尝试使用以下命令更改JAVA_HOME:

export JAVA_HOME=/usr/local/opt/java

以前的JAVA_HOME路径是/opt/anaconda3。我可以看到JAVA_HOME被更改为usr/local/opt/java

我仍然会出错。感谢您的回答/反馈。谢谢

对于macOS,请执行以下步骤步骤1。安装java8,因为Spark 2.2以后的版本需要java8。请参阅Spark文档!详细信息。

brew安装openjdk@8

然后设置更新java路径

export JAVA_HOME=/usr/local/opt/openjdk@8/libexec/openjdk.jdk/Contents/Home

我在网上搜索,看到了安装pyspark的说明。

我在终端上运行了这个命令pip install pyspark

安装pyspark后,spark和pyspark都在运行。

不知道发生了什么事,但我现在已经能够点燃火花了。

感谢@Elliott的互动和指导!

相关内容