我得到一个系统错误(如下所示),而执行一些简单的numpy为基础的矩阵代数计算并行使用Multiprocessing包(python 2.73与numpy 1.7.0在Ubuntu 12.04 on Amazon EC2)。我的代码对于较小的矩阵大小工作得很好,但是对于较大的矩阵(有足够的可用内存)会崩溃
我使用的矩阵的大小很大(我的代码对于1000000x10浮点密集矩阵运行良好,但是对于1000000x500的矩阵崩溃-顺便说一下,我正在将这些矩阵传递给/从子进程)。10 vs 500是一个运行时参数,其他一切保持不变(输入数据,其他运行时参数等)
我还尝试使用python3运行相同的(移植)代码-对于较大的矩阵,子进程进入睡眠/空闲模式(而不是像python 2.7那样崩溃),程序/子进程只是挂在那里什么也不做。对于较小的矩阵,使用python3可以很好地运行代码。
任何建议都将是非常感谢的(我已经没有主意了)
错误信息:
Exception in thread Thread-5: Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run() File "/usr/lib/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.7/multiprocessing/pool.py", line 319, in _handle_tasks
put(task) SystemError: NULL result without error in PyObject_Call
我使用的Multiprocessing代码:
def runProcessesInParallelAndReturn(proc, listOfInputs, nParallelProcesses):
if len(listOfInputs) == 0:
return
# Add result queue to the list of argument tuples.
resultQueue = mp.Manager().Queue()
listOfInputsNew = [(argumentTuple, resultQueue) for argumentTuple in listOfInputs]
# Create and initialize the pool of workers.
pool = mp.Pool(processes = nParallelProcesses)
pool.map(proc, listOfInputsNew)
# Run the processes.
pool.close()
pool.join()
# Return the results.
return [resultQueue.get() for i in range(len(listOfInputs))]
下面是为每个子流程执行的"proc"。基本上,它使用numpy解决了许多线性方程组(它在子进程中构造所需的矩阵),并将结果作为另一个矩阵返回。同样,对于一个运行时参数的较小值,它可以正常工作,但对于较大的参数会崩溃(或在python3中挂起)。
def solveForLFV(param):
startTime = time.time()
(chunkI, LFVin, XY, sumLFVinOuterProductLFVallPlusPenaltyTerm, indexByIndexPurch, outerProductChunkSize, confWeight), queue = param
LFoutChunkSize = XY.shape[0]
nLFdim = LFVin.shape[1]
sumLFVinOuterProductLFVpurch = np.zeros((nLFdim, nLFdim))
LFVoutChunk = np.zeros((LFoutChunkSize, nLFdim))
for LFVoutIndex in xrange(LFoutChunkSize):
LFVInIndexListPurch = indexByIndexPurch[LFVoutIndex]
sumLFVinOuterProductLFVpurch[:, :] = 0.
LFVInIndexChunkLow, LFVInIndexChunkHigh = getChunkBoundaries(len(LFVInIndexListPurch), outerProductChunkSize)
for LFVInIndexChunkI in xrange(len(LFVInIndexChunkLow)):
LFVinSlice = LFVin[LFVInIndexListPurch[LFVInIndexChunkLow[LFVInIndexChunkI] : LFVInIndexChunkHigh[LFVInIndexChunkI]], :]
sumLFVinOuterProductLFVpurch += sum(LFVinSlice[:, :, np.newaxis] * LFVinSlice[:, np.newaxis, :])
LFVoutChunk[LFVoutIndex, :] = np.linalg.solve(confWeight * sumLFVinOuterProductLFVpurch + sumLFVinOuterProductLFVallPlusPenaltyTerm, XY[LFVoutIndex, :])
queue.put((chunkI, LFVoutChunk))
print 'solveForLFV: ', time.time() - startTime, 'sec'
sys.stdout.flush()
500,000,000相当大:如果你使用float64,那就是40亿字节,或大约4gb。(10,000,000浮点数组将是8000万字节,或大约80mb -小得多。)我希望这个问题与多进程试图pickle数组以通过管道发送给子进程有关。
由于您是在unix平台上,您可以通过利用fork()
的内存继承行为(用于创建多进程的工作者)来避免这种行为。我用这个hack(从这个项目中剥离出来的)取得了巨大的成功,评论中描述了这一点。
### A helper for letting the forked processes use data without pickling.
_data_name_cands = (
'_data_' + ''.join(random.sample(string.ascii_lowercase, 10))
for _ in itertools.count())
class ForkedData(object):
'''
Class used to pass data to child processes in multiprocessing without
really pickling/unpickling it. Only works on POSIX.
Intended use:
- The master process makes the data somehow, and does e.g.
data = ForkedData(the_value)
- The master makes sure to keep a reference to the ForkedData object
until the children are all done with it, since the global reference
is deleted to avoid memory leaks when the ForkedData object dies.
- Master process constructs a multiprocessing.Pool *after*
the ForkedData construction, so that the forked processes
inherit the new global.
- Master calls e.g. pool.map with data as an argument.
- Child gets the real value through data.value, and uses it read-only.
'''
# TODO: does data really need to be used read-only? don't think so...
# TODO: more flexible garbage collection options
def __init__(self, val):
g = globals()
self.name = next(n for n in _data_name_cands if n not in g)
g[self.name] = val
self.master_pid = os.getpid()
@property
def value(self):
return globals()[self.name]
def __del__(self):
if os.getpid() == self.master_pid:
del globals()[self.name]