Python MultiProcessing apply_async等待所有进程完成

Question

我一直在尝试并行管理一系列批处理文件进程，同时存在依赖的子进程组。我希望得到的是能够并行运行group1的所有进程，然后等待所有进程完成，然后再运行group2等等。想象一系列进程组，其中每个进程都是一个单独的现有批处理文件(batch_I.bat(

根据我对多进程模块的理解，我有以下代码，所以我希望当调用最终打印命令时，所有日志文件都是完整的，打印完所有数字。然而，我注意到python代码在没有完成批处理的情况下成功地完成了。

Python代码：

import multiprocessing as mp
import subprocess
def worker(cmdlist, log):
with open(log, 'w') as logfile:
p = subprocess.Popen(cmdlist, stderr=logfile, stdout=logfile)
# return p.returncode
# --------------------------------------------
# Main Process (Group 1)
# --------------------------------------------
if __name__ == '__main__':
group1 = [batch_1 , batch_2 , batch_3 , ..., batch_10]
group2 = [batch_11, batch_12, batch_13, ..., batch_20]
group3 = [batch_21, batch_22, batch_23, ..., batch_30]
# Multi-Core Exec
all_process = group1 
all_results = []
pool = mp.Pool(processes=4)
for myProcess in all_process:
print("Starting Process: %s" %myProcess)
log = os.path.splitext(myProcess)[0] + ".log"
res = pool.apply_async(worker, args=[myProcess, log])
all_results.append(res)
pool.close()
pool.join()
print("All sub-processes completed")
for res in all_results:
res.get()
print("All sub-processes completed: %s" % [res.successful() for res in all_results])
# --------------------------------------------
# call group 2 and wait for completion
# --------------------------------------------
....
# --------------------------------------------
# call group 3 and wait for completion
# --------------------------------------------
...

其余代码调用组2中依赖于组1完成的所有进程，依此类推

---

批处理文件：Batch_i.bat：

在这种情况下，批处理文件是一个示例，它只打印出很多数字，我让循环重复几次，以确保批处理文件需要足够长的时间才能完成。

@echo off
echo Start of Loop
for /L %%n in (1,1,40000) do echo %%n
for /L %%n in (1,1,40000) do echo %%n
for /L %%n in (1,1,40000) do echo %%n
for /L %%n in (1,1,40000) do echo %%n
echo End of Loop

---

输出如下：

> *** Running Base Cases: ***
>      on 4 CPUs Process: C:Usersmamo8001ProjectClustering1 CodestestNum.bat Process: C:Usersmamo8001ProjectClustering1
> CodestestNum2.bat Process: C:Usersmamo8001ProjectClustering1
> CodestestNum3.bat Process: C:Usersmamo8001ProjectClustering1
> CodestestNum4.bat Process: C:Usersmamo8001ProjectClustering1
> CodestestNum.bat Process: C:Usersmamo8001ProjectClustering1
> CodestestNum2.bat Process: C:Usersmamo8001ProjectClustering1
> CodestestNum3.bat Process: C:Usersmamo8001ProjectClustering1
> CodestestNum4.bat 
> All sub-processes completed 
> All sub-processes completed: [True, True, True, True, True, True, True,
> True]
> 
> Process finished with exit code 0

当打印最后两行时，我注意到日志文件没有打印出完整的数字列表，即批处理没有完成

Answer 1

问题是您的员工不会等待他们的子流程退出。在worker中的p = subprocess.Popen()调用之后添加一个p.wait()。

Answer 2

使用八个批处理文件，其中只有一个用于循环到40000，我得到了相同的结果，直到我作为上下文管理器运行Popen。

def worker(cmdlist, log):
with open(log, 'w') as logfile:
with subprocess.Popen(cmdlist, stderr=logfile, stdout=logfile) as p:
pass
# return p.returncode

然后，直到所有cmd窗口关闭，最后两个print语句才打印出来。每个日志文件都有所有的数字以及循环行的开始/结束。

当被用作上下文管理器时，文档会说它等待直到过程完成。

如果你有Python 3.5+，文档会说使用subprocess.run()而不是Popen，而.run()文档明确表示它会等待命令完成——我无法测试，我有Python 3.4。

---

批处理文件为

echo off
echo Start of Loop
for /L %%n in (1,1,40000) do echo %%n
echo End of Loop