sklearn管道匹配:AttributeError: lower not found

Question

我想在sklearn中使用管道，像这样:

corpus = load_files('corpus/train')
stop_words = [x for x in open('stopwords.txt', 'r').read().split('n')]  # Uppercase!
countvec = CountVectorizer(stop_words=stop_words, ngram_range=(1, 2))
X_train, X_test, y_train, y_test = train_test_split(corpus.data, corpus.target, test_size=0.9,
                                                    random_state=0)
x_train_counts = countvec.fit_transform(X_train)
x_test_counts = countvec.transform(X_test)
k_fold = KFold(n=len(corpus.data), n_folds=6)
confusion = np.array([[0, 0], [0, 0]])
pipeline = Pipeline([
    ('vectorizer',  CountVectorizer(stop_words=stop_words, ngram_range=(1, 2))),
    ('classifier',  MultinomialNB()) ])
for train_indices, test_indices in k_fold:
    pipeline.fit(x_train_counts, y_train)
    predictions = pipeline.predict(x_test_counts)

然而，我得到这个错误:

AttributeError: lower not found

我看了这篇文章:

AttributeError: lower not found;在scikit-learn

中使用带有CountVectorizer的Pipeline

但是我正在向矢量器传递一个字节列表，所以这应该不是问题。

编辑

corpus = load_files('corpus')
stop_words = [x for x in open('stopwords.txt', 'r').read().split('n')]
X_train, X_test, y_train, y_test = train_test_split(corpus.data, corpus.target, test_size=0.5,
                                                    random_state=0)
k_fold = KFold(n=len(corpus.data), n_folds=6)
confusion = np.array([[0, 0], [0, 0]])
pipeline = Pipeline([
    ('vectorizer', CountVectorizer(stop_words=stop_words, ngram_range=(1, 2))),
    ('classifier', MultinomialNB())])
for train_indices, test_indices in k_fold:
    pipeline.fit(X_train[train_indices], y_train[train_indices])
    predictions = pipeline.predict(X_test[test_indices])

现在我得到错误:

TypeError: only integer arrays with one element can be converted to an index

2日编辑

corpus = load_files('corpus')
stop_words = [y for x in open('stopwords.txt', 'r').read().split('n') for y in (x, x.title())]
k_fold = KFold(n=len(corpus.data), n_folds=6)
confusion = np.array([[0, 0], [0, 0]])
pipeline = Pipeline([
    ('vectorizer', CountVectorizer(stop_words=stop_words, ngram_range=(1, 2))),
    ('classifier', MultinomialNB())])
for train_indices, test_indices in k_fold:
    pipeline.fit(corpus.data, corpus.target)

Answer 1

您没有正确使用管道。您不需要将数据向量化传递，其思想是管道将数据向量化。

# This is done by the pipeline
# x_train_counts = countvec.fit_transform(X_train)
# x_test_counts = countvec.transform(X_test)
k_fold = KFold(n=len(corpus.data), n_folds=6)
confusion = np.array([[0, 0], [0, 0]])
pipeline = Pipeline([
    ('vectorizer',  CountVectorizer(stop_words=stop_words, ngram_range=(1, 2))),
    ('classifier',  MultinomialNB()) ])
# also you are not using the indices...
for train_indices, test_indices in k_fold:
    pipeline.fit(corpus.data[train_indices], corpus.target[train_indices])
    predictions = pipeline.predict(corpus.data[test_indices])