concurrent.futures.ProcessPoolExecutor当函数是lambda或嵌套函数时挂起.<

Question

谁能解释一下为什么使用lambda或嵌套函数(f)会使concurrent.futures.ProcessPoolExecutor挂起以下代码示例?

import concurrent.futures
​
​
def f2(s):
return len(s)
​
​
def main():
def f(s):
return len(s)
​
data = ["a", "b", "c"]
​
with concurrent.futures.ProcessPoolExecutor(max_workers=1) as pool:
# results = pool.map(f, data) # hangs
# results = pool.map(lambda d: len(d), data)  # hangs
# results = pool.map(len, data)  # works
results = pool.map(f2, data) # works
​
print(list(results))
​
​
if __name__ == "__main__":
main()

Answer 1

长话短说，Pool/ProcessPoolExecutor都必须在将它们发送给worker之前序列化所有内容。序列化(有时也称为pickle)实际上是保存函数名的过程，只有当Pool想要访问它时才会再次导入它。为了使这个过程工作，函数必须在顶层定义，因为嵌套函数不能被子进程导入这就是出现以下错误的原因:

AttributeError: Can't pickle local object 'MyClass.mymethod.<locals>.mymethod'

为了避免这个问题，有一些解决方案，我觉得不可靠。如果您可以灵活地使用其他包，那么实际上是的替代方案。例如，下面的代码不会挂起:

import pathos
import os
class SomeClass:
def __init__(self):
self.words = ["a", "b", "c"]
def some_method(self):

def run(s):
return len(s)

return list(pool.map(run, self.words))
pool = pathos.multiprocessing.Pool(os.cpu_count())
print(SomeClass().some_method())

，它确实会输出

[1, 1, 1]