如何解决 python 中稀疏矩阵"NaN or infinity"问题？

Question

我对python完全陌生。我使用了一些在线找到的代码，并尝试对其进行处理。所以我正在创建一个文本-文档-矩阵，我想在训练逻辑回归模型之前添加一些额外的功能。

虽然我已经用 R 检查了我的数据并且没有收到任何错误，但当我运行逻辑回归时，我收到错误"ValueError：数组包含 NaN 或无穷大"。当我不添加自己的功能时，我没有收到同样的错误。我的功能在文件"toPython.txt"中。

请注意对返回"None"assert_all_finite函数的两个调用！

以下是我使用的代码和我得到的输出：

def _assert_all_finite(X):
if X.dtype.char in np.typecodes['AllFloat'] and not np.isfinite(X.sum()) and not np.isfinite(X).all():
    raise ValueError("Array contains NaN or infinity.")
def assert_all_finite(X):
_assert_all_finite(X.data if sparse.issparse(X) else X)
def main():
print "loading data.."
traindata = list(np.array(p.read_table('data/train.tsv'))[:,2])
testdata = list(np.array(p.read_table('data/test.tsv'))[:,2])
y = np.array(p.read_table('data/train.tsv'))[:,-1]
tfv = TfidfVectorizer(min_df=12,  max_features=None, strip_accents='unicode',  
    analyzer='word',stop_words='english', lowercase=True,
    token_pattern=r'w{1,}',ngram_range=(1, 1), use_idf=1,smooth_idf=1,sublinear_tf=1)
rd = lm.LogisticRegression(penalty='l2', dual=True, tol=0.0001, 
                         C=1, fit_intercept=True, intercept_scaling=1.0, 
                         class_weight=None, random_state=None)
X_all = traindata + testdata
lentrain = len(traindata)
f = np.array(p.read_table('data/toPython.txt'))
indices = np.nonzero(~np.isnan(f))
b = csr_matrix((f[indices], indices), shape=f.shape, dtype='float')
print b.get_shape
**print assert_all_finite(b)**
print "fitting pipeline"
tfv.fit(X_all)
print "transforming data"
X_all = tfv.transform(X_all)
print X_all.get_shape
X_all=hstack( [X_all,b], format='csr' )
print X_all.get_shape
**print assert_all_finite(X_all)**
X = X_all[:lentrain]
print "3 Fold CV Score: ", np.mean(cross_validation.cross_val_score(rd, X, y, cv=3, scoring='roc_auc'))

输出为：

loading data..
<bound method csr_matrix.get_shape of <10566x40 sparse matrix of type '<type 'numpy.float64'>'
with 422640 stored elements in Compressed Sparse Row format>>
**None**
fitting pipeline
transforming data
<bound method csr_matrix.get_shape of <10566x13913 sparse matrix of type '<type 'numpy.float64'>'
with 1450834 stored elements in Compressed Sparse Row format>>
<bound method csr_matrix.get_shape of <10566x13953 sparse matrix of type '<type 'numpy.float64'>'
with 1873474 stored elements in Compressed Sparse Row format>>
**None**
3 Fold CV Score: 
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:Python27libsite-packagesspyderlibwidgetsexternalshellsitecustomize.py", line 523, in runfile
execfile(filename, namespace)
File "C:UsersStergiosDocumentsPythonbeat_bench.py", line 100, in <module>
main()
File "C:UsersStergiosDocumentsPythonbeat_bench.py", line 97, in main
print "3 Fold CV Score: ", np.mean(cross_validation.cross_val_score(rd, X, y, cv=3, scoring='roc_auc'))
File "C:Python27libsite-packagessklearncross_validation.py", line 1152, in cross_val_score
for train, test in cv)
File "C:Python27libsite-packagessklearnexternalsjoblibparallel.py", line 517, in __call__
self.dispatch(function, args, kwargs)
File "C:Python27libsite-packagessklearnexternalsjoblibparallel.py", line 312, in dispatch
job = ImmediateApply(func, args, kwargs)
File "C:Python27libsite-packagessklearnexternalsjoblibparallel.py", line 136, in __init__
self.results = func(*args, **kwargs)
File "C:Python27libsite-packagessklearncross_validation.py", line 1064, in _cross_val_score
score = scorer(estimator, X_test, y_test)
File "C:Python27libsite-packagessklearnmetricsscorer.py", line 141, in __call__
return self._sign * self._score_func(y, y_pred, **self._kwargs)
File "C:Python27libsite-packagessklearnmetricsmetrics.py", line 403, in roc_auc_score
fpr, tpr, tresholds = roc_curve(y_true, y_score)
File "C:Python27libsite-packagessklearnmetricsmetrics.py", line 672, in roc_curve
fps, tps, thresholds = _binary_clf_curve(y_true, y_score, pos_label)
File "C:Python27libsite-packagessklearnmetricsmetrics.py", line 504, in _binary_clf_curve
y_true, y_score = check_arrays(y_true, y_score)
File "C:Python27libsite-packagessklearnutilsvalidation.py", line 233, in check_arrays
_assert_all_finite(array)
File "C:Python27libsite-packagessklearnutilsvalidation.py", line 27, in _assert_all_finite
raise ValueError("Array contains NaN or infinity.")
ValueError: Array contains NaN or infinity.

有什么想法吗？谢谢！！

Answer 1

我发现执行以下操作，假设sm是一个稀疏矩阵（我的矩阵CSR，如果您知道，请谈谈其他类型的一些事情！

手动将nan替换为数据向量中的适当数字：

In [4]: np.isnan(matrix.data).any()
Out[4]: True
In [5]: sm.data.shape
Out[5]: (553555,)
In [6]: sm.data = np.nan_to_num(sm.data)
In [7]: np.isnan(matrix.data).any()
Out[7]: False
In [8]: sm.data.shape
Out[8]: (553555,)

因此，我们不再有nan值，但矩阵将这些零显式编码为值索引。

从稀疏矩阵中删除显式编码的零值：

In [9]: sm.eliminate_zeros()
In [10]: sm.data.shape
Out[10]: (551391,)

我们的矩阵现在实际上变小了，耶！

Answer 2

这通常发生在数据中缺少值或处理的结果。

首先，在稀疏矩阵X中找到具有Nan或Inf值的单元格：

def find_nan_in_csr(self, X):
    X = coo_matrix(X)
    for i, j, v in zip(X.row, X.col, X.data):
        if (np.isnan(v) or np.isinf(v)):
            print(i, j, v)
    return None

此函数将为您提供稀疏矩阵中值有问题的行索引和列索引。
然后，"修复"这些值 - 这取决于导致这些值的原因（缺失值等）。

编辑：请注意，sklearn 通常使用 dtype=np.float32 来实现最大效率，所以当它可以的时候，它将稀疏矩阵转换为 np.float32（通过 X = X.astype(dtype = np.float32)）。在从 float64 到 np.float32 的转换中，一个非常高的数字（例如，2.9e+200 ）被转换为 inf 。

Answer 3

我通常使用这个函数：

x = np.nan_to_num(x)

将

nan 替换为零，将 inf 替换为有限数。