RandomForestClassifier .fit 在 ec2 上因内存错误而失败，但在本地运行时没有错误

Question

我正在将随机森林分类器拟合到包含tfidf和其他维度为(42238,155085(的特征的熊猫数据帧上。大小为 26GB。在本地训练模型时，代码运行没有错误(尽管速度很慢(，但是在内存是 4 倍的 ec2 实例上，当内存利用率达到 37% 时，训练过程会因内存错误而终止。这在 6 次运行时执行中是一致的。在 anaconda 2.7、代码和数据等同的情况下，为什么这个过程在 ec2 上会失败？

我正在16GB Macbook Pro上本地运行该模型。ec2 实例具有 64GB 内存。

我试过什么

1. n_jobs=1
2. n_estimators=10
3. max_depth=10
4. 增加 EC2 上的交换空间
5. 将最大锁定内存设置为无限制

EC2 实例上的 ulimit -a。

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 251728
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 251728
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

法典

model = RandomForestClassifier(max_depth=None, 
max_features=0.2, 
min_samples_leaf=1, 
min_samples_split=2,
n_estimators=10, 
n_jobs=-1,
verbose=1)
model.fit(X_train,y_train)

堆栈跟踪

model.fit(X_train,y_train)
File "/opt/anaconda2/lib/python2.7/site-packages/sklearn/ensemble/forest.py", line 247, in fit
X = check_array(X, accept_sparse="csc", dtype=DTYPE)
File "/opt/anaconda2/lib/python2.7/site-packages/sklearn/utils/validation.py", line 433, in check_array
array = np.array(array, dtype=dtype, order=order, copy=copy)
File "/opt/anaconda2/lib/python2.7/site-packages/pandas/core/generic.py", line 1603, in __array__
return com._values_from_object(self)
File "pandas/_libs/lib.pyx", line 47, in pandas._libs.lib.values_from_object
File "/opt/anaconda2/lib/python2.7/site-packages/pandas/core/generic.py", line 4684, in get_values
return self.values
File "/opt/anaconda2/lib/python2.7/site-packages/pandas/core/generic.py", line 4629, in values
return self._data.as_array(transpose=self._AXIS_REVERSED)
File "/opt/anaconda2/lib/python2.7/site-packages/pandas/core/internals.py", line 3949, in as_array
arr = mgr._interleave()
File "/opt/anaconda2/lib/python2.7/site-packages/pandas/core/internals.py", line 3960, in _interleave
result = np.empty(self.shape, dtype=dtype)
MemoryError

Answer 1

以此为指导：https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-centos-7

调整了以下参数，模型能够完成训练。

鼓励使用交换空间vm.swappiness=70

vm.vfs_cache_pressure=50鼓励操作系统使用更多缓存