CountVectorizer在添加停止词后对fit_transform抛出错误

Question

我有两段代码。一个工作，一个不工作。

以下代码按预期运行，没有错误:(注意:postrain, negtrain, postest和negtest是前面定义的字符串列表)

from sklearn.feature_extraction.text import CountVectorizer
vector = CountVectorizer()
train_vector = vector.fit_transform(postrain+negtrain)
test_vector = vector.transform(postest+negtest)
print test_vector.shape

然而，这段代码抛出了一个错误:

import re
stop = [re.split('n|t', open('stop_words.txt').read())]
vector2 = CountVectorizer(stop_words=stop)
train_vector = vector2.fit_transform(postrain+negtrain) # <-- Error occurs here
test_vector = vector2.transform(postest+negtest)
print test_vector.shape

错误:

TypeErrorTraceback (most recent call last)
<ipython-input-43-cf5f4754d58c> in <module>()
      7 
      8 vector2 = CountVectorizer(stop_words=stop)
----> 9 train_vector = vector2.fit_transform(postrain+negtrain)
     10 test_vector = vector2.transform(postest+negtest)
     11 
C:UsersNsthAnaconda2envscs489libsite-packagessklearnfeature_extractiontext.pyc in fit_transform(self, raw_documents, y)
    815 
    816         vocabulary, X = self._count_vocab(raw_documents,
--> 817                                           self.fixed_vocabulary_)
    818 
    819         if self.binary:
C:UsersNsthAnaconda2envscs489libsite-packagessklearnfeature_extractiontext.pyc in _count_vocab(self, raw_documents, fixed_vocab)
    745             vocabulary.default_factory = vocabulary.__len__
    746 
--> 747         analyze = self.build_analyzer()
    748         j_indices = _make_int_array()
    749         indptr = _make_int_array()
C:UsersNsthAnaconda2envscs489libsite-packagessklearnfeature_extractiontext.pyc in build_analyzer(self)
    232 
    233         elif self.analyzer == 'word':
--> 234             stop_words = self.get_stop_words()
    235             tokenize = self.build_tokenizer()
    236 
C:UsersNsthAnaconda2envscs489libsite-packagessklearnfeature_extractiontext.pyc in get_stop_words(self)
    215     def get_stop_words(self):
    216         """Build or fetch the effective stop words list"""
--> 217         return _check_stop_list(self.stop_words)
    218 
    219     def build_analyzer(self):
C:UsersNsthAnaconda2envscs489libsite-packagessklearnfeature_extractiontext.pyc in _check_stop_list(stop)
     92         return None
     93     else:               # assume it's a collection
---> 94         return frozenset(stop)
     95 
     96 
TypeError: unhashable type: 'list'

添加停止词是如何导致错误的?

Answer 1

我很笨。它应该是:

stop = re.split('n|t', open('stop_words.txt').read())

不带括号。虽然不确定为什么它在之后的行中抛出错误