将池结果添加到一个命令中

Question

i具有一个函数，该函数接受Itertools组合提供的两个输入，并输出解决方案。这两个输入应作为元组存储，形成DICT中的密钥，而结果为值。

。

我可以将其汇总并获取所有结果作为列表，然后我可以一对一地将其插入字典中，但这似乎效率低下。是否有一种方法可以在每个工作完成时获得结果，并将其直接添加到DICE？

中

本质上，我有以下代码：

all_solutions = {}
for start, goal in itertools.combinations(graph, 2):
    all_solutions[(start, goal)] = search(graph, start, goal)

我试图按以下方式并行化它：

all_solutions = {}
manager = multiprocessing.Manager()
graph_pool = manager.dict(graph)
pool = multiprocessing.Pool()
results = pool.starmap(search, zip(itertools.repeat(graph_pool),
                                   itertools.combinations(graph, 2)))
for i, start_goal in enumerate(itertools.combinations(graph, 2)):
    start, goal = start_goal[0], start_goal[1]
    all_solutions[(start, goal)] = results[i]

实际上有效，但迭代两次，一次在池中，然后写给dict(更不用说笨拙的元组拆开包装(。

Answer 1

这是可能的，您只需要切换到使用懒惰映射功能(而不是map或starmap，这些功能必须完成所有结果，然后才能开始使用任何结果(：

from functools import partial
from itertools import tee
manager = multiprocessing.Manager()
graph_pool = manager.dict(graph)
pool = multiprocessing.Pool()
# Since you're processing in order and in parallel, tee might help a little
# by only generating the dict keys/search arguments once. That said, 
# combinations of n choose 2 are fairly cheap; the overhead of tee's caching
# might overwhelm the cost of just generating the combinations twice
startgoals1, startgoals2 = tee(itertools.combinations(graph, 2))
# Use partial binding of search with graph_pool to be able to use imap
# without a wrapper function; using imap lets us consume results as they become
# available, so the tee-ed generators don't store too many temporaries
results = pool.imap(partial(search, graph_pool), startgoals2))
# Efficiently create the dict from the start/goal pairs and the results of the search
# This line is eager, so it won't complete until all the results are generated, but
# it will be consuming the results as they become available in parallel with
# calculating the results
all_solutions = dict(zip(startgoals1, results))