即使使用20个执行器，每个执行器使用25GB，但由于超过了GC开销限制，Spark执行器丢失

Question

这个GC开销限制错误让我抓狂。我有20个执行器，每个使用25gb，我完全不明白它是如何抛出GC开销的，我也不知道那么大的数据集。一旦这个GC错误发生在执行器，它将丢失，慢慢地其他执行器丢失，因为IOException, Rpc客户端解除关联，shuffle未找到等。

I am new to Spark.

WARN scheduler.TaskSetManager: Lost task 7.0 in stage 363.0 (TID 3373, myhost.com): java.lang.OutOfMemoryError: GC overhead limit exceeded
            at org.apache.spark.sql.types.UTF8String.toString(UTF8String.scala:150)
            at org.apache.spark.sql.catalyst.expressions.GenericRow.getString(rows.scala:120)
            at org.apache.spark.sql.columnar.STRING$.actualSize(ColumnType.scala:312)
            at org.apache.spark.sql.columnar.compression.DictionaryEncoding$Encoder.gatherCompressibilityStats(compressionSchemes.scala:224)
            at org.apache.spark.sql.columnar.compression.CompressibleColumnBuilder$class.gatherCompressibilityStats(CompressibleColumnBuilder.scala:72)
            at org.apache.spark.sql.columnar.compression.CompressibleColumnBuilder$class.appendFrom(CompressibleColumnBuilder.scala:80)
            at org.apache.spark.sql.columnar.NativeColumnBuilder.appendFrom(ColumnBuilder.scala:87)
            at org.apache.spark.sql.columnar.InMemoryRelation$$anonfun$3$$anon$1.next(InMemoryColumnarTableScan.scala:148)
            at org.apache.spark.sql.columnar.InMemoryRelation$$anonfun$3$$anon$1.next(InMemoryColumnarTableScan.scala:124)
            at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:277)
            at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171)
            at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
            at org.apache.spark.rdd.RDD.iterator(RDD.scala:242)
            at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
            at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
            at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
            at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
            at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
            at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
            at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
            at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
            at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
            at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
            at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
            at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
            at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
            at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
            at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
            at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70)
            at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
            at org.apache.spark.scheduler.Task.run(Task.scala:70)
            at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)

Answer 1

当cpu用于垃圾收集任务的开销超过98%时抛出GC overhead limit exceeded。在Scala中，当使用不可变数据结构时就会发生这种情况，因为对于每次转换，JVM都必须重新创建许多新对象，并从堆中删除以前的对象。因此，如果这是您的问题，请尝试使用一些可变数据结构来代替。

请阅读此页面http://spark.apache.org/docs/latest/tuning.html#garbage-collection-tuning以了解如何调优GC。