Spark单机模式:worker不能正常停止

Question

在spark(0.7.0)中使用

停止整个集群时

$SPARK_HOME/bin/stop-all.sh

不是所有的工人都正确停止。更具体地说，如果我想用

重新启动集群

$SPARK_HOME/bin/start-all.sh

我

:

host1: starting spark.deploy.worker.Worker, logging to [...]
host3: starting spark.deploy.worker.Worker, logging to [...]
host2: starting spark.deploy.worker.Worker, logging to [...]
host5: starting spark.deploy.worker.Worker, logging to [...]
host4: spark.deploy.worker.Worker running as process 8104. Stop it first.
host7: spark.deploy.worker.Worker running as process 32452. Stop it first.
host6: starting spark.deploy.worker.Worker, logging to [...]

在host4和host7上，确实有一个StandaloneExecutorBackend仍在运行:

$ jps
27703 Worker
27763 StandaloneExecutorBackend
28601 Jps

简单重复

$SPARK_HOME/bin/stop-all.sh

不幸的是，

也不能阻止工人。Spark刚刚告诉我工人即将被停止:

host2: no spark.deploy.worker.Worker to stop
host7: stopping spark.deploy.worker.Worker
host1: no spark.deploy.worker.Worker to stop
host4: stopping spark.deploy.worker.Worker
host6: no spark.deploy.worker.Worker to stop
host5: no spark.deploy.worker.Worker to stop
host3: no spark.deploy.worker.Worker to stop

no spark.deploy.master.Master to stop

然而,

$ jps
27703 Worker
27763 StandaloneExecutorBackend
28601 Jps

否则说。有人知道stop-all.sh如何正常工作吗?谢谢。

Answer 1

原因似乎是缓存整个数据集的尝试导致Worker机器大量交换数据。在这种情况下，工作机器的数量对于数据集来说太小了。