java.lang.NoClassDefFoundError in spark-submit

Question

我在这里使用

从https://github.com/tribbloid/spookystuff/tree/branch-0.4构建了这个项目

MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize= 512M " mvn package -DskipTests=true

from here https://gist.github.com/titipata/13fad88df5525d607f24

然后我在自己的单独项目中使用core/target/x-jar-with-dependencies.jar中创建的fat-jar。

<dependency>
    <groupId>com.tribbloid.spookystuff</groupId>
    <artifactId>spookystuff-core_2.10</artifactId>
    <scope>system</scope>
    <version>0.4.0-SNAPSHOT</version>
    <systemPath>${basedir}/lib/x-jar-with-dependencies.jar</systemPath>
</dependency>

我还创建了这个项目的脂肪罐，我使用spookystuff在(y-jar-with-dependencies.jar让我们说)。

现在运行

spark-submit——class org.webcrawler.core.Driver——master local[*]/path/to/my/y-jar-with-dependencies.jar

出现

错误

main线程异常java.lang.NoClassDefFoundError:com/tribbloids/spookystuff/actions/Action

调查:

jar -tvf/path/to/my/x-jar-with-dependencies.jar | grep "com/tribbloids/spookystuff/actions/Action"

我得到结果但是

jar -tvf/path/to/my/y-jar-with-dependencies.jar | grep "com/tribbloids/spookystuff/actions/Action"

我没有找到任何类。我到底错在哪里?

Answer 1

在Ykyuen的https://eureka.ykyuen.info/2014/06/10/maven-include-system-scope-dependency-in-maven-assembly-plugin/帖子中解决了我的问题。