尝试绕过numpy.core._exceptions._ArrayMemoryError问题在我的代码

Question

我有一个数据帧->形状为(10000,257)的数据。我需要预处理这个数据帧，以便我可以在LSTM中使用它，这需要3维输入- (nrows,ntimesteps,nfeatures)我正在使用这里提供的代码片段:

def univariate_processing(variable, window):
import numpy as np
# create empty 2D matrix from variable
V = np.empty((len(variable)-window+1, window))
# take each row/time window
for i in range(V.shape[0]):
V[i,:] = variable[i : i+window]
V = V.astype(np.float32) # set common data type
return V
def RNN_regprep(df, y, len_input, len_pred): #, test_size):
# create 3D matrix for multivariate input
X = np.empty((df.shape[0]-len_input+1, len_input, df.shape[1]))
# Iterate univariate preprocessing on all variables - store them in XM
for i in range(df.shape[1]):
X[ : , : , i ] = univariate_processing(df[:,i], len_input)
# create 2D matrix of y sequences
y = y.reshape((-1,))  # reshape to 1D if needed
Y = univariate_processing(y, len_pred)
## Trim dataframes as explained
X = X[ :-(len_pred + 1) , : , : ]
Y = Y[len_input:-1 , :]
# Set common datatype
X = X.astype(np.float32)
Y = Y.astype(np.float32)
return X, Y
X,y = RNN_regprep(data,label, len_ipnut=200,len_pred=1)

运行此命令时，得到以下错误:

numpy.core._exceptions._ArrayMemoryError: Unable to allocate 28.9 GiB for an array with shape (10000, 200, 257) and data type float64

我明白这更多的是我的内存在我的服务器的问题。我想知道我可以在代码中更改的任何解决方案，看看我是否可以避免此内存错误或尝试减少此内存消耗?

Answer 1

这就是窗口视图的作用。在这里使用我的配方:

var = np.random.rand(10000,257)
w = window_nd(var, 200, axis = 0)

现在你有一个var的窗口视图:

w.shape
Out[]: (9801, 200, 257)

但是，重要的是，它使用与var完全相同的数据，只是以窗口的方式查看它:

w.__array_interface__['data'] #This is the memory's starting address
Out[]: (1448954720320, False)
var.__array_interface__['data']
Out[]: (1448954720320, False)
np.shares_memory(var, w)
Out[]: True
w.base.base.base is var  #(lots of rearranging views in the background)
Out[]: True

所以你可以这样做:

def univariate_processing(variable, window):
return window_nd(variable, window, axis = 0)

这将显著减少内存分配，没有"魔法"。要求:)

你也可以试试

from skimage.util import view_as_windows
w = np.squeeze(view_as_windows(var, (200, 1)))

几乎做同样的事情。在这种情况下，你的答案应该是:

def univariate_processing(variable, window):
from skimage.util import view_as_windows
window = (window,) + (1,)*(len(variable.shape)-1)
return np.squeeze(view_as_windows(variable, window))