Windows中的Spark异常:Java.io.ioException:无法重命名Shuffle文件

Question

我在尝试运行火花流代码时面临问题。我正在尝试从KAFKA主题中读取数据，然后将处理的数据推向弹性搜索。我正在Windows上的Eclipse中运行此代码，并配置了Kafka，Spark，Zookeeper和Elasticsearch。我收到以下错误：

  18/02/20 14:52:11 ERROR Executor: Exception in task 0.0 in stage 6.0 (TID 5)
  java.io.IOException: fail to rename file C:UsersshashAppDataLocalTempblockmgr-cb45497b-7f85-4158-815b-852edecbb2c5fshuffle_1_0_0.index.ca3b55d2-6c26-4798-a17b-21a42f099126 to C:UsersshashAppDataLocalTempblockmgr-cb45497b-7f85-4158-815b-852edecbb2c5fshuffle_1_0_0.index
at org.apache.spark.shuffle.IndexShuffleBlockResolver.writeIndexFileAndCommit(IndexShuffleBlockResolver.scala:178)
at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:72)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
at org.apache.spark.scheduler.Task.run(Task.scala:85)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
  18/02/20 14:52:11 WARN TaskSetManager: Lost task 0.0 in stage 6.0 (TID 5, localhost): java.io.IOException: fail to rename file C:UsersshashAppDataLocalTempblockmgr-cb45497b-7f85-4158-815b-852edecbb2c5fshuffle_1_0_0.index.ca3b55d2-6c26-4798-a17b-21a42f099126 to C:UsersshashAppDataLocalTempblockmgr-cb45497b-7f85-4158-815b-852edecbb2c5fshuffle_1_0_0.index
at org.apache.spark.shuffle.IndexShuffleBlockResolver.writeIndexFileAndCommit(IndexShuffleBlockResolver.scala:178)
at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:72)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
at org.apache.spark.scheduler.Task.run(Task.scala:85)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

有人可以给我一些有关如何解决此问题的投入吗？

Answer 1

我通过将Spark.local.dir设置为已重命名权限的另一个路径来解决此问题。

SparkConf conf = new SparkConf().set("spark.local.dir","another path");

我是新来的火花，希望它对您有用。