我有一个脚本，我在其中使用 pool.apply_async 运行一些进程并将它们作为非守护程序运行，以避免"僵尸"进程压倒内存的问题。到目前为止，它一直运行良好，除了现在我已经扩展到内存中更大的数据集，因此通过使用我所有的核心，我正在消耗内存。我想限制在这些情况下使用的内核数量，但无法使其工作
通常我会集成如下内容
pool = Pool(self.nb_cores)
以限制内核数。但是，我似乎找不到将其集成到非demon进程中的位置。
import multiprocessing
import multiprocessing.pool
class NoDaemonProcess(multiprocessing.Process):
"""
Extends the multiprocessing Process class to disable
the daemonic property. Polling the daemonic property
will always return False and cannot be set.
"""
@property
def daemon(self):
"""
Always return False
"""
return False
@daemon.setter
def daemon(self, value):
"""
Pass over the property setter
:param bool value: Ignored setting
"""
pass
class NoDaemonContext(type(multiprocessing.get_context())):
"""
With the new multiprocessing module, everything is based
on contexts after the overhaul. This extends the base
context so that we set all Processes to NoDaemonProcesses
"""
Process = NoDaemonProcess
class NoDaemonPool(multiprocessing.pool.Pool):
"""
This extends the normal multiprocessing Pool class so that
all spawned child processes are non-daemonic, allowing them
to spawn their own children processes.
"""
def __init__(self, *args, **kwargs):
kwargs['context'] = NoDaemonContext()
super(NoDaemonPool, self).__init__(*args, **kwargs)
我知道我需要在某处集成许多内核限制......只是似乎无法在我的上下文中找到我需要的精确功能。
您的自定义NoDaemonPool类派生自multiprocessing.pool.Pool因此将能够接受processes(要使用的工作进程数(作为关键字参数：
pool = NoDaemonPool(processes=nb_cores)