SBT测试任务如何管理类路径以及如何从SBT测试正确启动Java过程

Question

在我的一个Scala测试中，使用ProcessBuilder，我在单独的JVM中启动了3个Apache Spark流媒体应用程序。(两个或多个火花流应用程序无法在同一JVM中共存。(一种Spark应用程序处理数据并摄入Apache Kafka，其他人会读取该数据。此外，测试涉及写入NOSQL数据库。

使用ProcessBuilder时，使用：

设置了Spark应用程序的类路径

val classPath = System.getProperty("java.class.path")

在Intellij中运行测试的工作原理，但是在CI系统上，测试由SBT的测试任务调用。在后一种情况下的java.class.path将仅是sbt.jar，因此，孩子JVM再次以NoClassFoundException退出，如预期的那样。： - (

我正在寻找一种使用测试实际使用的类路径从SBT测试中"跨度" JVM的方法。例如，如果在项目core中调用了测试，则应将项目core的类路径提供给SPARK应用程序启动的儿童JVM。不幸的是，我不知道如何检索SBT任务中的正确类路径 - 然后可以将其提供给Child JVM。

Answer 1

Tests.Setup可以用来访问SBT中的类路径：

testOptions in Test += Tests.Setup { classLoader =>
   // give Spark classpath via classLoader
}

例如，在我的计算机上Tests.Setup(classLoader => println(classLoader))给出

> test
ClasspathFilter(
  parent = URLClassLoader with NativeCopyLoader with RawResources(
  urls = List(/home/mario/sandbox/sbt/so-classpath/target/scala-2.12/test-classes, /home/mario/sandbox/sbt/so-classpath/target/scala-2.12/classes, /home/mario/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.4.jar, /home/mario/.ivy2/cache/org.scalatest/scalatest_2.12/bundles/scalatest_2.12-3.0.5.jar, /home/mario/.ivy2/cache/org.scalactic/scalactic_2.12/bundles/scalactic_2.12-3.0.5.jar, /home/mario/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.12.4.jar, /home/mario/.ivy2/cache/org.scala-lang.modules/scala-xml_2.12/bundles/scala-xml_2.12-1.0.6.jar),
  parent = DualLoader(a = java.net.URLClassLoader@3fcb37f1, b = java.net.URLClassLoader@271053e1),
  resourceMap = Set(app.class.path, boot.class.path),
  nativeTemp = /tmp/sbt_741bc913/sbt_c770779a
)
  root = sun.misc.Launcher$AppClassLoader@33909752
  cp = Set(/home/mario/.ivy2/cache/jline/jline/jars/jline-2.14.5.jar, /home/mario/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.12.4.jar, /home/mario/.ivy2/cache/org.scala-lang/scala-compiler/jars/scala-compiler-2.12.4.jar, /home/mario/.ivy2/cache/org.scala-lang.modules/scala-xml_2.12/bundles/scala-xml_2.12-1.0.6.jar, /home/mario/sandbox/sbt/so-classpath/target/scala-2.12/classes, /home/mario/.ivy2/cache/org.scalatest/scalatest_2.12/bundles/scalatest_2.12-3.0.5.jar, /home/mario/.sbt/boot/scala-2.10.7/org.scala-sbt/sbt/0.13.17/test-interface-1.0.jar, /home/mario/.ivy2/cache/org.scalactic/scalactic_2.12/bundles/scalactic_2.12-3.0.5.jar, /home/mario/sandbox/sbt/so-classpath/target/scala-2.12/test-classes, /home/mario/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.4.jar)
)

我们看到的

.../target/scala-2.12/test-classes
.../target/scala-2.12/classes

存在。

另一方面，要从测试本身中检索类路径：

val classLoader = this.getClass.getClassLoader
// give Spark classpath via classLoader

例如，在我的机器上，println(classLoader)在以下测试中给出

class CubeCalculatorTest extends FunSuite {
  test("CubeCalculator.cube") {
    val classLoader = this.getClass.getClassLoader
    println(classLoader)
    assert(CubeCalculator.cube(3) === 27)
  }
}

打印

URLClassLoader with NativeCopyLoader with RawResources(
  urls = List(/home/mario/sandbox/sbt/so-classpath/target/scala-2.12/test-classes, /home/mario/sandbox/sbt/so-classpath/target/scala-2.12/classes, /home/mario/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.4.jar, /home/mario/.ivy2/cache/org.scalatest/scalatest_2.12/bundles/scalatest_2.12-3.0.5.jar, /home/mario/.ivy2/cache/org.scalactic/scalactic_2.12/bundles/scalactic_2.12-3.0.5.jar, /home/mario/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.12.4.jar, /home/mario/.ivy2/cache/org.scala-lang.modules/scala-xml_2.12/bundles/scala-xml_2.12-1.0.6.jar),
  parent = DualLoader(a = java.net.URLClassLoader@6307eb76, b = java.net.URLClassLoader@271053e1),
  resourceMap = Set(app.class.path, boot.class.path),
  nativeTemp = /tmp/sbt_fa64d1a1/sbt_66bd50e2
)

我们再次看到

.../target/scala-2.12/test-classes
.../target/scala-2.12/classes

存在。

在测试中实际将类路径传递给ProcessBuilder：

import java.net.URLClassLoader
import sys.process._
class CubeCalculatorTest extends FunSuite {
  test("CubeCalculator.cube") {
    val classLoader = this.getClass.getClassLoader
    val classpath = classLoader.asInstanceOf[URLClassLoader].getURLs.map(_.getFile).mkString(":")
    s"java -classpath $classpath MyExternalApp".!
    ...
  }
} 

Answer 2

您可以在ProcessBuilder的JVM中获得完整的classpath，从sbt测试中使用：

Thread
  .currentThread
  .getContextClassLoader
  .getParent
  .asInstanceOf[java.net.URLClassLoader]
  .getURLs
  .map(_.getFile)
  .mkString(System.getProperty("path.separator"))