群集上的 dask-yarn:无法连接到应用程序

Question

我正在尝试使用 dask-yarn 在集群上分发 Python 作业。

我使用以下代码创建群集：

from dask_yarn import YarnCluster
cluster = YarnCluster(environment='.conda/envs/myconda', worker_vcores=2, worker_memory='4GB', n_workers=4)
client = Client(cluster)
cluster.shutdown()

我假设您需要更多信息来确保它连接到纱线。

错误消息如下所示：

---------------------------------------------------------------------------
ConnectionError                           Traceback (most recent call last)
<ipython-input-16-9de74c663703> in <module>()
----> 1 cluster = YarnCluster('myconda.tar.gz')
~/.conda/envs/myconda/lib/python3.6/site-packages/dask_yarn/core.py in __init__(self, environment, n_workers, worker_vcores, worker_memory, worker_restarts, worker_env, scheduler_vcores, scheduler_memory, deploy_mode, name, queue, tags, user, skein_client)
    293                                    user=user)
    294 
--> 295         self._start_cluster(spec, skein_client)
    296 
    297     @cached_property
~/.conda/envs/myconda/lib/python3.6/site-packages/dask_yarn/core.py in _start_cluster(self, spec, skein_client)
    373             app = skein_client.submit_and_connect(spec)
    374             try:
--> 375                 scheduler_address = app.kv.wait('dask.scheduler').decode()
    376                 dashboard_address = app.kv.get('dask.dashboard')
    377                 if dashboard_address is not None:
~/.conda/envs/myconda/lib/python3.6/site-packages/skein/kv.py in wait(self, key, return_owner)
    653                 return res
    654 
--> 655             event = event_queue.get()
    656 
    657         return event.result if return_owner else event.result.value
~/.conda/envs/myconda/lib/python3.6/site-packages/skein/kv.py in get(self, block, timeout)
    279         if isinstance(out, Exception):
    280             self._exception = out
--> 281             raise out
    282         return out
    283 
ConnectionError: Unable to connect to application
```python

Answer 1

这可能是因为您的应用程序无法启动(因此无法连接(。我建议查看日志：

$ yarn logs -applicationId <YOUR APPLICATION ID>

通过查看上面的代码，一件奇怪的事情是指定的 conda 环境。看起来您正在传递目录的路径而不是存档环境的路径 - 您可能需要阅读此处的文档：http://yarn.dask.org/en/latest/environments.html。