sklearn CountVectorizor的词汇选项带有bigrams，返回零数组

Question

我想从一个数组中提取bigram，取所有频率大于100的bigram，然后使用减少的词汇对第二个数组进行评分。

词汇选项似乎应该能满足我的需求，但它似乎不起作用。即使将一个的输出直接馈送到另一个，也只会产生一个（形状正确的）零阵列。

from sklearn.feature_extraction.text import CountVectorizer
docs = ['run fast into a bush','run fast into a tree','run slow','run fast']
# Collect bigrams
vectorizer = CountVectorizer(ngram_range = (2,2))
vectorizer.fit(docs)
vocab = vectorizer.vocabulary_
# Score the exact same data
vectorizer = CountVectorizer(vocabulary=vocab)
output = vectorizer.transform(docs)
# Demonstrate that the array is all zeros
print "Length of vocab", len(vocab)
print output.A

Length of vocab 5
[[0 0 0 0 0]
 [0 0 0 0 0]
 [0 0 0 0 0]
 [0 0 0 0 0]]

Answer 1

刚刚捕获。您需要在第二个实例中指定ngram_range（它不会自动解释unigram与bigram。）

vectorizer = CountVectorizer(vocabulary=vocab ,ngram_range = (2,2))

Answer 2

创建一个标记化单词语料库。将ngram设置为3有奇数个单词，否则设置为2表示偶数。创建一个单词矩阵袋。用单词矩阵包创建一个数据帧，并获得矢量化单词作为特征。

from sklearn.feature_extraction.text import CountVectorizer
import nltk
docs = ['run fast into a bush','run fast into a tree','run slow','run fast']
str_buffer=" ".join(docs)
#print(str_buffer)
corpus=nltk.word_tokenize(str_buffer)
vectorizer_ng2=CountVectorizer(ngram_range=range(1,3),stop_words='english')
bow_matrix=vectorizer_ng2.fit_transform(corpus)
print(bow_matrix.toarray())
bow_df = pd.DataFrame(bow_matrix.toarray())
bow_df.columns = vectorizer_ng2.get_feature_names()
print(bow_df)

输出：

bush  fast  run  slow  tree
0      0     0    1     0     0
1      0     1    0     0     0
2      0     0    0     0     0
3      0     0    0     0     0
4      1     0    0     0     0
5      0     0    1     0     0
6      0     1    0     0     0
7      0     0    0     0     0
8      0     0    0     0     0
9      0     0    0     0     1
10     0     0    1     0     0
11     0     0    0     1     0
12     0     0    1     0     0
13     0     1    0     0     0