使用Joblib进行多处理:对函数的一个参数进行并行处理

Question

parallel函数有没有一种方法可以接受多个参数，但只能并行处理其中一个参数？

假设我有一些代码：

def my_function(graph,graph_coefficients, thing_i_want_to_parallelise_over):
<do_thing>
return value 

results = Parallel(n_job2=2)(delayed(my_function(one_graph,graph_coefficients)(thing_i_want_to_parallelise_over) for thing_i_want_to_parallelise_over in range(1,3))

有办法做到这一点吗？有多个函数可调用，因此执行简单的环绕函数实际上不是一种选择。

Answer 1

我不知道我是否理解你的问题，但你错误地格式化了

您应该创建具有所有参数的元组

(one_graph, graph_coefficients, x) for x in range(1,3)   # args

然后你应该把它和一起使用

delayed( my_function )

像

results = Parallel(n_jobs=2)( 
delayed( my_function )
(one_graph, graph_coefficients, x) for x in range(1,3)
)

---

最终您可以尝试使用lambda

lambda x: my_function(one_graph, graph_coefficients,x)

然后你可以使用

(x) for x in range(1,3)

像

results = Parallel(n_jobs=2)(
delayed( lambda x: my_function(one_graph, graph_coefficients,x) ) 
(x) for x in range(1,3) 
)

---

或使用functools.partial

partial(my_function, one_graph, graph_coefficients)

像

from functools import partial
results = Parallel(n_jobs=2)(
delayed( partial(my_function, one_graph, graph_coefficients) ) 
(x) for x in range(1,3) 
)

---

最小工作代码

from joblib import Parallel, delayed
def my_function(graph, graph_coefficients, thing_i_want_to_parallelise_over):
print('my_function:', graph, graph_coefficients, thing_i_want_to_parallelise_over)
value = 2 * thing_i_want_to_parallelise_over
return value 
one_graph = 'A'
graph_coefficients = 'B'
# ----
results = Parallel(n_jobs=2)(
delayed( my_function ) 
(one_graph, graph_coefficients, x) for x in range(1,3) 
)
print('results:', results)
# ----
results = Parallel(n_jobs=2)(
delayed( lambda x: my_function(one_graph, graph_coefficients,x) ) 
(x) for x in range(1,3) 
)
print('results:', results)
# ----
from functools import partial
results = Parallel(n_jobs=2)(
delayed( partial(my_function, one_graph, graph_coefficients) ) 
(x) for x in range(1,3) 
)
print('results:', results)