纵向扩展 Dask-SSH 集群

Question

我设法使用以下方法创建了一个Dask SSH Cuda集群：

from dask.distributed import Client, SSHCluster
cluster = SSHCluster(
["localhost", "192.168.1.119", "192.168.1.191"],
connect_options={"known_hosts": None,"username": "vinhdiesal"},
worker_options={"nthreads": 40},
scheduler_options={"port": 0, "dashboard_address": ":8797"},
worker_module='dask_cuda.dask_cuda_worker')

它导致 4 个工作人员如下：

dask
+---------------------------+---------+
|                           | version |
+---------------------------+---------+
| client                    | 2.11.0  |
| scheduler                 | 2.11.0  |
| tcp://192.168.1.119:38425 | 2.11.0  |
| tcp://192.168.1.119:41579 | 2.11.0  |
| tcp://192.168.1.191:45641 | 2.17.2  |
| tcp://192.168.1.191:46713 | 2.17.2  |
+---------------------------+---------+

有没有办法扩大规模，这样我就可以获得更多的dask工人？ 我能够增加线程，但我想扩大工人的数量。

Answer 1

你可以通过传递一个nprocs参数来要求工作线程启动多个进程：

from dask.distributed import Client, SSHCluster
cluster = SSHCluster(
["localhost", "192.168.1.119", "192.168.1.191"],
connect_options={"known_hosts": None,"username": "vinhdiesal"},
worker_options={"nthreads": 16, "nprocs": 4},
scheduler_options={"port": 0, "dashboard_address": ":8797"},
worker_module='dask_cuda.dask_cuda_worker')