Spark:迭代算法实现中的堆栈溢出错误

Question

我正在尝试使用Scala在Spark中迭代算法，但Spark抛出了StackOverflowError。 我的代码是

object demo {
def main(args: Array[String]): Unit = {
val N =             60        //args(0).toInt      // Population length
val d =              60       //args(1).toInt     //Dimensions ; Number of unknown Decision variables
var Iterations =     400       //args(2).toInt     // Maximum Number of Iteration
val nP =           3         //args(3).toInt      // number of partitions
val MinVal =       -100         //args(6).toInt    //  Lower Bound
val MaxVal =      100          //args(7).toInt    //  Upper Bound
val Fmin = 0                    // Minimum Frequency
val Fmax = 1                    // Maximum Frequency
val Bandwidth = 0.001
val InitialPulseRate = 0.1
val alpha = 0.95
val gyma = 0.95
var GlobalBest_Fitness = Double.PositiveInfinity
val batList = List.fill(N)(new obj(d, MinVal, MaxVal))
batList.map { x =>
x.fitness =SphereFunc(x.position) // Update Fitness
}
GlobalBest_Fitness = batList.minBy(_.fitness).fitness
val conf = new SparkConf().setMaster(locally("local")).setAppName("spark Demo")
val sc = new SparkContext(conf)
val rdd = sc.parallelize(batList, nP)
var partitioned = rdd
partitioned.persist()
var itrU = 0
Fun(Iterations)
@tailrec
def Fun( itr: Int): RDD[obj] = itr match {
case 0 => partitioned// Base case for recursion
case _ => {
itrU = Iterations - itr + 1
partitioned =Updater(partitioned,itrU , Bandwidth)
Fun( itr - 1) //Recursive call
}
}
} 

错误消息的简短片段是

Exception in thread "main" java.lang.StackOverflowError
at java.io.ObjectStreamClass$WeakClassKey.<init>(ObjectStreamClass.java:2505)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:348)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at scala.collection.immutable.List$SerializationProxy.writeObject(List.scala:468)

如何修复此错误？ 我必须运行数千次迭代，维度数 (d( 和人口大小 (N( 的值也大得多。我使用了尾递归函数，因此它可以在常量空间中运行，但它的行为并非如此。

Answer 1

堆栈溢出错误与 Spark.driver 和 Spark.executor 内存分配及其 1GB 的 Spark 默认配置值有关。 sparkConfig 应更新如下。如果全部记忆;尝试增加值仍然不够。

val conf = new SparkConf().setMaster(locally("local")).setAppName("spark Demo").set("spark.executor.memory", "4g").set("spark.driver.memory", "4g");