持久化 sklearn.feature_extraction.text.TfidfVectorizer 的错误

Question

我使用模块joblib持久化了一个TfidfVectorizer。我通过方法fit_transform运行的对象是一个字符串列表。生成的矩阵具有 263744 列的维度。

我正在通过转换方法运行字符串列表，并收到以下错误。

有什么线索吗？

File "/usr/local/lib/python2.7/dist-      packages/sklearn/feature_extraction/text.py", 
line 1334, in transform
return self._tfidf.transform(X, copy=False)
File "/usr/local/lib/python2.7/dist-packages/sklearn/feature_extraction/text.py", 
line 1037, in transform
X = X * self._idf_diag
File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/base.py", line    
318, in __mul__
return self._mul_sparse_matrix(other)
File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/compressed.py",
line 487, in _mul_sparse_matrix
other = self.__class__(other)  # convert to this format
File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/compressed.py",
line 31, in __init__
arg1 = arg1.asformat(self.format)
File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/base.py", 
line 219, in asformat
return getattr(self,'to' + format)()
File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/dia.py", 
line 241, in tocsr
return self.tocoo().tocsr()
File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/dia.py", 
line 249, in tocoo
num_offsets, offset_len = self.data.shape
AttributeError: 'NDArrayWrapper' object has no attribute 'shape'

Answer 1

假设将训练的转换器或管道保存到磁盘，然后在看到错误之前重新加载它，则可以：

1. 尝试使用 compress 关键字参数参数将原始（工作）对象保存到 joblib.dump ，整数值大于 0：

   _ = joblib.dump(python_object, persisted_file_name, compress=3)
   

2. 如果要将持久化的文件移动到新位置，请将确保复制所有文件片段。 如果它很大，joblib 将将其拆分，例如：

   persisted_model.joblib.pkl
   persisted_model.joblib.pkl_01.npy
   persisted_model.joblib.pkl_02.npy
   

乔布文档