Tf-Agents ParallelPyEnvironment 以静默方式失败

Question

我编写了一个自定义环境，所以我可以使用强化学习(PPO(和tf-agent。 如果我包装我的 env(它继承自py_environment，这工作正常。PyEnvironment(在TfPyEnvironment中，但如果我尝试将其包装到ParallelPyEnvironment中，则会失败。我尝试过使用ParallelPyEnvironment的所有关键字参数，但代码只是运行到行，然后什么也没发生 - 没有异常，程序不会终止等。

这是我初始化环境并展示eval_env工作变体的代码：

train_env = tf_py_environment.TFPyEnvironment(
ParallelPyEnvironment(
[CardGameEnv()] * hparams['parallel_environments']
)
)
# this works perfectly:
eval_env = tf_py_environment.TFPyEnvironment(CardGameEnv(debug=True))

如果我通过CTRL+C终止脚本，这就是输出的内容：

Traceback (most recent call last):
Traceback (most recent call last):
File "E:UserstmpDocumentsProgrammingNeural NetsPoker_AIpoker_logictrain.py", line 229, in <module>
File "<string>", line 1, in <module>
train(model_num=3)
File "C:Python37libmultiprocessingspawn.py", line 105, in spawn_main
File "E:UserstmpDocumentsProgrammingNeural NetsPoker_AIpoker_logictrain.py", line 64, in train
[CardGameEnv()] * hparams['parallel_environments']
exitcode = _main(fd)
File "E:UserstmpAppDataRoamingPythonPython37site-packagesginconfig.py", line 1009, in wrapper
File "C:Python37libmultiprocessingspawn.py", line 113, in _main
preparation_data = reduction.pickle.load(from_parent)
KeyboardInterrupt
return fn(*new_args, **new_kwargs)
File "C:Python37libsite-packagestf_agentsenvironmentsparallel_py_environment.py", line 70, in __init__
self.start()
File "C:Python37libsite-packagestf_agentsenvironmentsparallel_py_environment.py", line 83, in start
env.start(wait_to_start=self._start_serially)
File "C:Python37libsite-packagestf_agentsenvironmentsparallel_py_environment.py", line 223, in start
self._process.start()
File "C:Python37libmultiprocessingprocess.py", line 112, in start
self._popen = self._Popen(self)
File "C:Python37libmultiprocessingcontext.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:Python37libmultiprocessingcontext.py", line 322, in _Popen
return Popen(process_obj)
File "C:Python37libmultiprocessingpopen_spawn_win32.py", line 65, in __init__
reduction.dump(process_obj, to_child)
File "C:Python37libmultiprocessingreduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
File "C:Python37libsite-packagestf_agentsenvironmentsparallel_py_environment.py", line 264, in __getattr__
return self._receive()
File "C:Python37libsite-packagestf_agentsenvironmentsparallel_py_environment.py", line 333, in _receive
message, payload = self._conn.recv()
File "C:Python37libmultiprocessingconnection.py", line 250, in recv
buf = self._recv_bytes()
File "C:Python37libmultiprocessingconnection.py", line 306, in _recv_bytes
[ov.event], False, INFINITE)
KeyboardInterrupt
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "C:Python37libsite-packagestf_agentsenvironmentsparallel_py_environment.py", line 289, in close
self._process.join(5)
File "C:Python37libmultiprocessingprocess.py", line 139, in join
assert self._popen is not None, 'can only join a started process'
AssertionError: can only join a started process

由此我得出结论，ParallelPyEnvironment尝试启动的线程没有这样做，但由于我对 Python 线程不是很有经验，我不知道从这里开始，尤其是如何解决这个问题。 当前的训练需要很长时间，并且根本不使用我的 PC 的功能(使用 3GB 32GB RAM，处理器为 3%，GPU 几乎无法工作，但 VRAM 已满(，因此这应该会显着加快训练时间。

Answer 1

解决方案是传入可调用对象，而不是环境，因此ParallelPyEnvironment可以自己构造它们：

train_env = tf_py_environment.TFPyEnvironment(
ParallelPyEnvironment(
[CardGameEnv] * hparams['parallel_environments']
)
)