从泡菜文件加载多处理管理器字典时出错

Question

我在从pickle加载multiprocessing.manager.dictionary时遇到错误。我确定泡菜文件存在于我运行 python 的目录中。

Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> import multiprocessing
>>> a = open("test.pkl", "rb")
>>> pickle.load(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1133, in load_reduce
    value = func(*args)
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 879, in RebuildProxy
    return func(token, serializer, incref=incref, **kwds)
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 733, in __init__
    self._incref()
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 783, in _incref
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 169, in Client
    c = SocketClient(address)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 304, in SocketClient
    s.connect(address)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 2] No such file or directory
>>> pickle.load(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1165, in load_put
    self.memo[self.readline()[:-1]] = self.stack[-1]
IndexError: list index out of range

这是将其保存到泡菜中的代码部分。

from multiprocessing import Manager, Pool
import pickle
from functools import partial
def output(dic, s):
    a = open(s + ".pkl", "wb")
    pickle.dump(dic, a)
    a.close()
data_list = [1,2,3,4]
pool = Pool(processes = 4)
m = Manager()
lock = m.Lock()
dic1 = m.dict()
func = partial(f, dic1) # f is a function that takes 3 arguments with the returned result stored in dic1
pool.map(func, data_list)
output(dic1, "test")
pool.close()
pool.join()

如果我在输出之前打印出字典，终端上的结果看起来不错。

Answer 1

Manager.dict()实际上并不返回字典，而是返回由管理器在不同进程中管理的真实字典的代理对象。在该对象上调用方法时，调用将转发到实际字典所在的管理器。

由于管理器和客户端进程之间的通信是使用 pickle 协议进行的，因此这些对象是可pickleable的，当取消pickled时，会再次生成驻留在管理器进程内的实际对象的代理，这就是为什么当您尝试取消pickle字典时会看到尝试创建连接的原因。

因此，如果您想腌制经理字典的内容，请在以下之前将其转换为实际字典：

...
pickle.dump(dict(dic), a)
...