ValueError:找到样本数不一致的数组:[4 16149]

Question

嗨，我是scikit学习和数据科学的新手。我在尝试从矢量器中检索信息量最大的功能时遇到了上述问题。我的代码（经过编辑以反映@Gang的评论）：

values = dataset.data
word_vectorizer = CountVectorizer(analyzer='word', stop_words=custom_stop_words)
trainset = word_vectorizer.fit_transform(values)
tags = ['dem','rep','dem','rep']
tags = np.array(tags)
trainset = trainset.toarray()
word_svm = svm.LinearSVC()
word_svm.fit(trainset, tags)

def most_informative_feature_for_binary_classification(vectorizer, classifier, n=10):
class_labels = classifier.classes_
feature_names = vectorizer.get_feature_names()
topn_class1 = sorted(zip(classifier.coef_[0], feature_names))[:n]
topn_class2 = sorted(zip(classifier.coef_[0], feature_names))[-n:]
for coef, feat in topn_class1:
    print class_labels[0], coef, feat
print
for coef, feat in reversed(topn_class2):
    print class_labels[1], coef, feat

most_informative_feature_for_binary_classification(word_vectorizer, word_svm)

终端输出：

Traceback (most recent call last):
File "classification.py", line 251, in <module>
word_svm.fit(trainset, tags)
File "/usr/local/lib/python2.7/site-packages/sklearn/svm/classes.py", line 205, in fit
dtype=np.float64, order="C")
File "/usr/local/lib/python2.7/site-packages/sklearn/utils/validation.py", line 520, in check_X_y
check_consistent_length(X, y)
File "/usr/local/lib/python2.7/site-packages/sklearn/utils/validation.py", line 176, in check_consistent_length
"%s" % str(uniques))
ValueError: Found arrays with inconsistent numbers of samples: [    4 16149]

我非常感谢在这件事上提供任何帮助。如果我没有提供足够的信息，请告诉我。提前感谢您抽出时间！

Answer 1

这是它失败的地方-两个参数应该是相同的类型-数组

word_svm.fit(trainset, tags)

标签不是数组，应转换为数组

tags = ['dem','rep','dem','rep']

您可以使用打印来查看它们是否是相同类型的

print type(tags)
print type(trainset)

下面的代码是在没有运行的情况下用文本编辑器编写的，不能保证工作，但你有这样的想法，我可能对转换为数组是错误的，List很好。

您的列车组正确包含无效数据，请更换

word_svm.fit(trainset, tags)

这个：

trainset_good, trainset_bad = trainset_check(trainset, tags)
print 'Bad datan'
print trainset_bad
if len(trainset_good)==0:
   print 'No good valid data found, exit'
   sys.exit(1)
# use good data
word_svm.fit(trainset_good, tags)

将此功能添加到代码中

def trainset_check(trainset, tags):
    trainset_good = []
    trainset_bad = []
    if not trainset:
        print 'Err trainset is empty'
        return trainset_good, trainset_bad
    if not tags:
        print 'Err - tags empty'
        return trainset_good, trainset_bad
    if len(trainset)==0:
        print 'Err trainset is empty'
        return trainset_good, trainset_bad
    if len(tags)==0:
        print 'Err tags empty'
        return trainset_good, trainset_bad
    for item in trainset:
        if len(item) != len(tags):
            print 'Error - trainset item is not the same length as tags'
            print item
            trainset_bad.append(item)
            # skip to next
            continue
        # filter out None type
        item_new = filter(None, item)
        if len(item_new) != len(tags):
            print 'Error - trainset item is not the same length as tags'
            # bad trainset data, skip to next
            print item
            trainset_bad.append(item)
            continue
         trainset_good.append(item)
    return trainset_good, trainset_bad