Apache Livy 0.7.0无法创建交互式会话



在使用apache Livy 0.7.0创建新会话时,我收到以下错误。我还使用齐柏林飞船笔记本(livy解释器(创建会话。

使用Scala 2.12.10版本,Java HotSpot(TM(64位服务器虚拟机11.0.11
Spark 3.0.2齐柏林飞船0.9.0

知道我为什么会出错吗?

我已经检查过类路径中是否有livy-repl_2.11-0.7.1-incubating.jar,并且jar已经有了找不到的类。

$ jar tf livy-repl_2.11-0.7.1-incubating.jar | grep "ReplDriver"
org/apache/livy/repl/ReplDriver$$anonfun$addFile$1.class
org/apache/livy/repl/ReplDriver$$anonfun$1.class
org/apache/livy/repl/ReplDriver$$anonfun$handle$1.class
org/apache/livy/repl/ReplDriver$$anonfun$addJarOrPyFile$1.class
org/apache/livy/repl/ReplDriver.class
org/apache/livy/repl/ReplDriver$$anonfun$handle$2.class
org/apache/livy/repl/ReplDriver$$anonfun$2.class

错误日志

21/06/17 13:13:28 INFO LineBufferedStream: Exception in thread "main" java.lang.ClassNotFoundException: org.apache.livy.repl.ReplDriver
21/06/17 13:13:28 INFO LineBufferedStream:      at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
21/06/17 13:13:28 INFO LineBufferedStream:      at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
21/06/17 13:13:28 INFO LineBufferedStream:      at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)

21/06/17 13:13:28 WARN ContextLauncher: Child process exited with code 1.
21/06/17 13:13:28 ERROR RSCClient: Failed to connect to context.
java.io.IOException: Child process exited with code 1.
at org.apache.livy.rsc.ContextLauncher$ChildProcess$1.run(ContextLauncher.java:397)
at org.apache.livy.rsc.ContextLauncher$ChildProcess$2.run(ContextLauncher.java:448)
at java.base/java.lang.Thread.run(Thread.java:834)
21/06/17 13:13:28 ERROR SparkProcApp: spark-submit exited with code 1
21/06/17 13:13:28 INFO RSCClient: Failing pending job f05c2bac-6e0a-4732-a419-0bde4dcfd06b due to shutdown.
21/06/17 13:13:28 INFO InteractiveSession: Stopping InteractiveSession 0...
21/06/17 13:13:28 INFO InteractiveSession: Failed to ping RSC driver for session 0. Killing application.
21/06/17 13:13:28 INFO InteractiveSession: Stopped InteractiveSession 0.
21/06/17 13:13:28 WARN InteractiveSession: Fail to get rsc uri
java.util.concurrent.ExecutionException: java.io.IOException: Child process exited with code 1.
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41)
at org.apache.livy.server.interactive.InteractiveSession$$anonfun$18.apply(InteractiveSession.scala:411) 

Spark 3.0.x附带了scala 2.12版本。您将需要使用livy和Spark 3.0.x,使用scal 2.12来解决这个问题。然后你需要调整你的livy.conf这是关于如何使用maven重建你的lify的文章(如何用scala2.12重建apache livy(

祝你好运。我遇到了同样的问题,并能够通过上述步骤解决。

最新更新