同时进行编译和多处理的问题

Question

我在VSCode中工作，当我运行这个文件时：

from multiprocessing import Process

def mp_setup_and_run(processes_num, *args):
processes = {}
for i in range(processes_num):
processes[i] = Process(
target=function_example,
args=args,
daemon=True,)
processes[i].start()
for i in range(processes_num):
processes[i].join()

def function_example(*data):
print(data)

if __name__ == "__main__":
compiled = compile("z**2 + c", "<string>", "eval")
mp_setup_and_run(3, compiled)

我得到一个例外：

PS C:Pythonprojectsmondebrot_painter>  cd 'c:Pythonprojectsmondebrot_painter'; & 'C:Program FilesPython38python.exe' 'c:UsersASUS.vscodeextensionsms-python.python-2020.5.80290pythonFileslibpythondebugpyno_wheelsdebugpylauncher' '51560' '--' 'c:Pythonprojectsmondebrot_paintertest.py' 
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:Program FilesPython38libmultiprocessingspawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:Program FilesPython38libmultiprocessingspawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
Traceback (most recent call last):
File "C:Program FilesPython38librunpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:Program FilesPython38librunpy.py", line 86, in _run_code
exec(code, run_globals)
File "c:UsersASUS.vscodeextensionsms-python.python-2020.5.80290pythonFileslibpythondebugpyno_wheelsdebugpy__main__.py", line 45, in <module>       
cli.main()
File "c:UsersASUS.vscodeextensionsms-python.python-2020.5.80290pythonFileslibpythondebugpyno_wheelsdebugpy/..debugpyservercli.py", line 430, in main
run()
File "c:UsersASUS.vscodeextensionsms-python.python-2020.5.80290pythonFileslibpythondebugpyno_wheelsdebugpy/..debugpyservercli.py", line 267, in run_file
runpy.run_path(options.target, run_name=compat.force_str("__main__"))
File "C:Program FilesPython38librunpy.py", line 263, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:Program FilesPython38librunpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:Program FilesPython38librunpy.py", line 86, in _run_code
exec(code, run_globals)
File "c:Pythonprojectsmondebrot_paintertest.py", line 45, in <module>
result = mp_setup_and_run(3, compiled)
File "c:Pythonprojectsmondebrot_paintertest.py", line 19, in mp_setup_and_run
processes[i].start()
File "C:Program FilesPython38libmultiprocessingprocess.py", line 121, in start
self._popen = self._Popen(self)
File "C:Program FilesPython38libmultiprocessingcontext.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:Program FilesPython38libmultiprocessingcontext.py", line 326, in _Popen
return Popen(process_obj)
File "C:Program FilesPython38libmultiprocessingpopen_spawn_win32.py", line 93, in __init__
reduction.dump(process_obj, to_child)
File "C:Program FilesPython38libmultiprocessingreduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'code' object

调试器将我重定向到<string>文件：

LOAD_CONST(0), LOAD_CONST(None), IMPORT_NAME(sys), STORE_NAME(sys), LOAD_NAME(sys.path), LOAD_METHOD(insert), LOAD_CONST(0), LOAD_CONST('c:\Users\ASUS\.vscode\extensions\ms-python.python-2020.5.80290\pythonFiles\lib\python\debugpy\no_wheels\debugpy\_vendored\pydevd'), CALL_METHOD{2}, POP_TOP, LOAD_CONST(0), LOAD_CONST(None), IMPORT_NAME(pydevd), STORE_NAME(pydevd), LOAD_CONST('http_json'), LOAD_NAME(pydevd.PydevdCustomization), STORE_ATTR(DEFAULT_PROTOCOL), LOAD_NAME(pydevd.settrace), LOAD_CONST('127.0.0.1'), LOAD_CONST(51592), LOAD_CONST(False), LOAD_CONST(False), LOAD_CONST(True), LOAD_CONST(None), LOAD_CONST('92e8bb604eeece436b2401def85a7ab95455e6c26fd9d660cb8175e691d71bd0'), LOAD_CONST('127.0.0.1'), LOAD_CONST('92e8bb604eeece436b2401def85a7ab95455e6c26fd9d660cb8175e691d71bd0'), LOAD_CONST(True), LOAD_CONST(True), LOAD_CONST(51592), LOAD_CONST(9040), LOAD_CONST(False), LOAD_CONST(('client', 'client-access-token', 'json-dap-http', 'multiprocess', 'port', 'ppid', 'server')), BUILD_CONST_KEY_MAP{7}, LOAD_CONST(('host', 'port', 'suspend', 'trace_only_current_thread', 'patch_multiprocessing', 'access_token', 'client_access_token', '__setup_holder__')), CALL_FUNCTION_KW{8}, POP_TOP, LOAD_CONST(0), LOAD_CONST(('spawn_main',)), IMPORT_NAME(multiprocessing.spawn), IMPORT_FROM(spawn_main), STORE_NAME(spawn_main), POP_TOP, LOAD_NAME(spawn_main), LOAD_CONST(9040), LOAD_CONST(892), LOAD_CONST(('parent_pid', 'pipe_handle')), CALL_FUNCTION_KW{2}, POP_TOP, return None

如果我从控制台运行该程序，则会收到以下消息：

C:Pythonprojectsmondebrot_painter>python set_generator.py
Traceback (most recent call last):
File "set_generator.py", line 121, in <module>
set_ = mp_setup_and_run(senter, length, quality, processes_num, max_iter, compiled, mode)
File "set_generator.py", line 86, in mp_setup_and_run
processes[i].start()
File "C:Program FilesPython38libmultiprocessingprocess.py", line 121, in start
self._popen = self._Popen(self)
File "C:Program FilesPython38libmultiprocessingcontext.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:Program FilesPython38libmultiprocessingcontext.py", line 326, in _Popen
return Popen(process_obj)
File "C:Program FilesPython38libmultiprocessingpopen_spawn_win32.py", line 93, in __init__
reduction.dump(process_obj, to_child)
File "C:Program FilesPython38libmultiprocessingreduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'code' object
C:Pythonprojectsmondebrot_painter>Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:Program FilesPython38libmultiprocessingspawn.py", line 107, in spawn_main
new_handle = reduction.duplicate(pipe_handle,
File "C:Program FilesPython38libmultiprocessingreduction.py", line 79, in duplicate
return _winapi.DuplicateHandle(
PermissionError: [WinError 5] Access Denied

我有些迷茫，不明白发生了什么以及为什么我不能通过编译。

Answer 1

如果您简化multiprocessing代码并从控制台使用它，您将看到您得到的TypeError：

$ python
...
>>> compiled = compile("z**2 + c", "<string>", "eval")
>>> import pickle
>>> pickle.dumps(compiled)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't pickle code objects

这是有道理的，因为文档告诉我们pickle可以处理：

• None、True和False
• 整数、浮点数、复数
• 字符串、字节、字节数组
• 元组、列表、集合和字典仅包含可选取的对象
• 在模块顶层定义的函数(使用 def，而不是 lambda(
• 在模块顶层定义的内置函数
• 在模块顶层定义的类
• 此类类的实例，其__dict__或调用__getstate__()的结果是可选取的(有关详细信息，请参阅拾取类实例一节(。

compiled不是其中之一。¹

这里没有说，但至关重要的是，multiprocessing模块必须能够使用pickle代码来序列化对象，以便将它们从一个 Python 进程发送到另一个进程。 由于compiled表达式不可序列化，因此无法将其从一个 Python 进程发送到另一个进程。

诀窍是序列化表达式，而不是序列化编译的表达式。也就是说，而不是：

mp_setup_and_run(3, compiled)

用：

mp_setup_and_run(3, "z**2 + c")

然后，在mp_setup_and_run中，让它将表达式传递给函数。 让每个函数对compile进行自己的调用。 您将在使用多处理模块运行的三个单独的进程中执行三个单独的compile，但这没关系。

---

¹当然，文档还说：

尝试腌制不可拾取的对象将引发PicklingError异常

当你和我都得到TypeError时。 但这就是TypeError的原因.