Windows上的多处理器池:CPU限制为63

Question

当使用具有63个以上内核的python的multiprocessing.pool.pool时，我会得到一个ValueError:

from multiprocessing.pool import Pool
def f(x):
return x
if __name__ == '__main__':
with Pool(70) as pool:
arr = list(range(70))
a = pool.map(f, arr)
print(a)

输出：

Exception in thread Thread-1:
Traceback (most recent call last):
File "C:UsersfischsamAnaconda3libthreading.py", line 932, in _bootstrap_inner
self.run()
File "C:UsersfischsamAnaconda3libthreading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "C:UsersfischsamAnaconda3libmultiprocessingpool.py", line 519, in _handle_workers
cls._wait_for_updates(current_sentinels, change_notifier)
File "C:UsersfischsamAnaconda3libmultiprocessingpool.py", line 499, in _wait_for_updates
wait(sentinels, timeout=timeout)
File "C:UsersfischsamAnaconda3libmultiprocessingconnection.py", line 879, in wait
ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
File "C:UsersfischsamAnaconda3libmultiprocessingconnection.py", line 811, in _exhaustive_wait
res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 72
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69]

这个程序似乎运行良好；结果正是我所期望的。我可以忽略ValueError吗？

背景：我在谷歌上搜索过这个问题，它似乎与Windows上python的限制有关(_winapi.WaitForMultipleObjects(；参见此处。建议的解决方案是将使用的核心数量限制在63个。这并不令人满意，因为我想在我的服务器上拥有100多个核心。我真的需要限制核心吗？为什么？有变通办法吗？

Answer 1

有趣的是，当我从concurrent.futures模块中使用PrcocessPoolExecutor(max_workers=70)查看问题是否仍然存在时，我得到了ValueError:max_workers必须<=61并且程序在提交任何作业之前立即终止这强烈暗示了无法绕过的Windows限制然而，您的程序实际上从未终止，只是在获得异常后挂起。在我的8核计算机上，如果我指定了任何大于60(不是61或63(的工作者，无论我使用multiprocessing.Pool还是concurrent.futures.ProcessPoolExecutor，它都会挂起。找出您的机器上允许其正常终止而不产生异常的最大工人数量，并坚持下去。

from concurrent.futures import ProcessPoolExecutor
def f(x):
return x
if __name__ == '__main__':
with ProcessPoolExecutor(max_workers=70) as executor:
a = list(executor.map(f, range(70)))
print(a)