带火炬的广播字符串.分布式和NCCL后端

Question

使用torch.distributed与nccl后端广播字符串的鲁棒方法是什么?

import torch.distributed as dist
foo = "info"
dist.broadcast(foo, 0)
# everybody has the same foo now

Answer 1

在PT中已经有一个函数，它将广播可pickle(而不仅仅是)对象列表，而无需手动将对象转换为二进制并将其移动到GPU:

import torch.distributed as dist
# init PG or something here
# ...
foostring = "broadcast_this"
dist.broadcast_object_list([foostring], 0)