错误:无法使用Spark提交加载Class main

Question

我的代码低于

import org.apache.spark.SparkContext;
import org.apache.spark.SparkConf;
object WordCounter {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("Word Counter").setMaster("local")
val sc = new SparkContext(conf)
val textFile = sc.textFile("C:/spark/README.md")
val tokenizedFileData = textFile.flatMap(line=>line.split(" "))
val countPrep = tokenizedFileData.map(word=>(word,1))
val counts = countPrep.reduceByKey((accumValue,newValue)=> accumValue + newValue)
val sortedCounts = counts.sortBy(kvPair=>kvPair._2,false)
sortedCounts.saveAsTextFile("C:/Data/WordCountViaApp")
}
}

有人能帮忙吗？

C： \Users\workspace\SparkInScala>spark-submit类；WordCounter"C： \Users\workspace\SparkInScala\target\scale-2.12\SparkInScala_2.12-0.1.0-SNAPSHOT.jar"；错误：未能加载类WordCounter。

Answer 1

我以前也遇到过同样的问题。

我正在使用idea的Maven插件来开发Spark程序

<build>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<executions>
<execution>
<id>compile-scala</id>
<phase>compile</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>test-compile-scala</id>
<phase>test-compile</phase>
<goals>
<goal>add-source</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>2.12.11</scalaVersion>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>testConf</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>assembly</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.6.0</version>
<executions>
<execution>
<goals>
<goal>exec</goal>
</goals>
</execution>
</executions>
<configuration>
<executable>java</executable>
<includeProjectDependencies>true</includeProjectDependencies>
<includePluginDependencies>false</includePluginDependencies>
<classpathScope>compile</classpathScope>
<mainClass>testConf</mainClass>
</configuration>
</plugin>
</plugins>
</build>

您可以通过替换mainClass标签"来尝试它；testConf"；到你的类名，和scala版本等

Answer 2

您应该包含要执行的.scala类/文件的完整路径。例如，如果您的类位于像com/example/xyz.scala这样的文件夹结构中，那么您的命令应该是："--类com.example.xyz">

Answer 3

如果要使用SBT，则必须指定启动执行的主方法。

Main-Class: com.WordCounter.main

上面的内容告诉SBT在运行SBT包时将行添加到JAR中的META-INF/MANIFEST.MF文件中。

使用SBT、运行应用程序时

sbt "run-main com.WordCounter.main"

此外，正如吴所指出的，马文也会做好这项工作。

Answer 4

在用适当的版本对build.sbt文件进行更改后，这一切都很正常。我不再看到错误Failed to Load class。