多处理 Pool.map 的奇怪行为

Question

我观察到使用 pool.map 调用方法函数时非常奇怪的行为。只有一个进程，行为与简单的 for 循环不同，我们在 if not self.seeded: 块中输入了几次，而我们不应该。以下是代码和输出：

import os
from multiprocessing import Pool

class MyClass(object):
    def __init__(self):
        self.seeded = False
        print("Constructor of MyClass called")
    def f(self, i):
        print("f called with", i)
        if not self.seeded:
            print("PID : {}, id(self.seeded) : {}, self.seeded : {}".format(os.getpid(), id(self.seeded), self.seeded))
            self.seeded = True
    def multi_call_pool_map(self):
        with Pool(processes=1) as pool:
            print("multi_call_pool_map with {} processes...".format(pool._processes))
            pool.map(self.f, range(10))
    def multi_call_for_loop(self):
        print("multi_call_for_loop ...")
        list_res = []
        for i in range(10):
            list_res.append(self.f(i))

if __name__ == "__main__":
    MyClass().multi_call_pool_map()

输出：

Constructor of MyClass called
multi_call_pool_map with 1 processes...
f called with 0
PID : 18248, id(self.seeded) : 1864747472, self.seeded : False
f called with 1
f called with 2
f called with 3
PID : 18248, id(self.seeded) : 1864747472, self.seeded : False
f called with 4
f called with 5
f called with 6
PID : 18248, id(self.seeded) : 1864747472, self.seeded : False
f called with 7
f called with 8
f called with 9
PID : 18248, id(self.seeded) : 1864747472, self.seeded : False

并使用 for 循环：

if __name__ == "__main__":
    MyClass().multi_call_for_loop()

输出：

Constructor of MyClass called
multi_call_for_loop ...
f called with 0
PID : 15840, id(self.seeded) : 1864747472, self.seeded : False
f called with 1
f called with 2
f called with 3
f called with 4
f called with 5
f called with 6
f called with 7
f called with 8
f called with 9

我们如何解释pool.map的行为(第一种情况(？我不明白为什么我们在 if 块中多次输入，因为self.seeded设置为仅在构造函数中False，而构造函数只调用一次......(我有 Python 3.6.8(

Answer 1

在运行代码并在 f 中打印self时，我们可以看到，在每次输入 if 子句之前，实例实际上都会发生变化：

    def f(self, i):
        print("f called with", i, "self is",self)
        if not self.seeded:
            print("PID : {}, id(self.seeded) : {}, self.seeded : {}".format(os.getpid(), id(self.seeded), self.seeded))
            self.seeded = True

这将输出：

Constructor of MyClass called
multi_call_pool_map with 1 processes...
f called with 0 self is <__main__.MyClass object at 0x7f30cd592b38>
PID : 22879, id(self.seeded) : 10744096, self.seeded : False
f called with 1 self is <__main__.MyClass object at 0x7f30cd592b38>
f called with 2 self is <__main__.MyClass object at 0x7f30cd592b38>
f called with 3 self is <__main__.MyClass object at 0x7f30cd592b00>
PID : 22879, id(self.seeded) : 10744096, self.seeded : False
f called with 4 self is <__main__.MyClass object at 0x7f30cd592b00>
f called with 5 self is <__main__.MyClass object at 0x7f30cd592b00>
f called with 6 self is <__main__.MyClass object at 0x7f30cd592ac8>
PID : 22879, id(self.seeded) : 10744096, self.seeded : False
f called with 7 self is <__main__.MyClass object at 0x7f30cd592ac8>
f called with 8 self is <__main__.MyClass object at 0x7f30cd592ac8>
f called with 9 self is <__main__.MyClass object at 0x7f30cd592a90>
PID : 22879, id(self.seeded) : 10744096, self.seeded : False

如果将chunksize=10添加到.map()它的行为将类似于 for 循环：

    def multi_call_pool_map(self):
        with Pool(processes=1) as pool:
            print("multi_call_pool_map with {} processes...".format(pool._processes))
            pool.map(self.f, range(10), chunksize=10)

这将输出：

Constructor of MyClass called
multi_call_pool_map with 1 processes...
f called with 0 self is <__main__.MyClass object at 0x7fd175093b00>
PID : 22972, id(self.seeded) : 10744096, self.seeded : False
f called with 1 self is <__main__.MyClass object at 0x7fd175093b00>
f called with 2 self is <__main__.MyClass object at 0x7fd175093b00>
f called with 3 self is <__main__.MyClass object at 0x7fd175093b00>
f called with 4 self is <__main__.MyClass object at 0x7fd175093b00>
f called with 5 self is <__main__.MyClass object at 0x7fd175093b00>
f called with 6 self is <__main__.MyClass object at 0x7fd175093b00>
f called with 7 self is <__main__.MyClass object at 0x7fd175093b00>
f called with 8 self is <__main__.MyClass object at 0x7fd175093b00>
f called with 9 self is <__main__.MyClass object at 0x7fd175093b00>

发生这种情况的确切原因是一个非常详细的实现细节，并且与multiprocessing如何在同一池中的进程之间共享数据有关。

恐怕我没有足够的资格确切地回答这在内部是如何以及为什么工作的。

Answer 2

当您将实例方法与 Pool.map 一起使用时，对象实例的副本将在 pickle 模块的帮助下发送到工作进程。您的结果显示了map如何在块中工作，并且对象实例在每个块开始时从腌制窗体重新加载。加载泡菜不会调用__init__.

请参阅 https://thelaziestprogrammer.com/python/a-multiprocessing-pool-pickle 以获取引擎盖下发生的事情的更多解释。