Python 在循环中并行执行一个函数

Question

我试图缩短脚本的执行时间，该脚本将数据从CSV导入Graphite/Go-Carbon DB时间序列。

这是解析所有zip文件并在函数(execute_run)中读取它们的循环： 它尝试了这段代码，但我收到一个错误：

for idx4, Lst_f in enumerate(full_csvfile_paths):
if lst_metrics in Lst_f:
zip_file = Lst_f
with zipfile.ZipFile(zip_file) as zipobj:
print("Using ZipFile:",zipobj.filename)
#execute_run(zipobj.filename, confcsv_path, storage_type, serial)
output = subprocess.run(execute_run(zipobj.filename, confcsv_path, storage_type, serial),stdout=subprocess.PIPE)
print ("Return code: %i" % output.returncode)
print ("Output data: %s" % output.stdout)

错误：

Traceback (most recent call last):
File "./02-pickle-client.py", line 451, in <module>
main()
File "./02-pickle-client.py", line 361, in main
output = subprocess.run(execute_run(zipobj.filename, confcsv_path, storage_type, serial),stdout=subprocess.PIPE)
File "/usr/lib64/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib64/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/usr/lib64/python3.6/subprocess.py", line 1240, in _execute_child
args = list(args)
TypeError: 'NoneType' object is not iterable

有没有办法执行X 倍函数：">execute_run"并控制正确的运行。

非常感谢您的帮助。

Answer 1

问题可能是并行进程未设置为正确处理可迭代对象。而不是subprocess.run，我建议使用multiprocessing.pool或multiprocessing.starmap这些文档中指定的

。这可能看起来像这样：

import multiprocessing as mp
# Step 1: Use multiprocessing.Pool() and specify number of cores to use (here I use 4).
pool = mp.Pool(4)
# Step 2: Use pool.starmap which takes a multiple iterable arguments
results = pool.starmap(My_Function, [(variable1,variable2,variable3) for i in data])

# Step 3: Don't forget to close
pool.close()