可迭代的 Python 多处理更新大小

Question

>我有以下内容：

从多处理导入池

def process_elements(index_of_data_inputs):
<process>
if <condition>:
# i would like to change the size of data_inputs 

if __name__ == '__main__':
pool = Pool()                         # Create a multiprocessing Pool
pool.map(process_elements, range(0, len(data_inputs))  # process data_inputs iterable with pool

我如何更改data_inputs的大小，从而更改process_elements次数 叫？

我想并行化背后的工作是：

i = 0
while i < len(elements):
new_elems = process_some_elements(x,y)
if len(new_elems) > 0:
elements = elements + new_elems 
i += 1

Answer 1

考虑 Python 中进程与多处理模块之间的通信的简单示例：

import multiprocessing
import queue
import random

def process_elements(num, comq):
val = random.random()
if val > 0.5:
comq.put(1)
return num, int(1000 * val)

if __name__ == '__main__':
# initial data
numbers = list(range(10))
# data structure fot communication between multiple processes
m = multiprocessing.Manager()
q = m.Queue()
with multiprocessing.Pool(processes=4) as pool:
# get answer for original data
ans = pool.starmap(process_elements, [(num, q) for num in numbers])
print(numbers)
print(ans)
# create additional data based on the answer for initial data
new_numbers = numbers[-1:]
try:
while True:
new_numbers.append(new_numbers[-1] + q.get_nowait())
except queue.Empty:
pass
# get answer for additional data
new_ans = pool.starmap(process_elements, [(num, q) for num in new_numbers[1:]])
print(new_numbers)
print(new_ans)