我有一组documents
:
D1 = "The sky is blue."
D2 = "The sun is bright."
D3 = "The sun in the sky is bright."
和一组words
,如:
"sky","land","sea","water","sun","moon"
我想创建一个像这样的矩阵:
x D1 D2 D3
sky tf-idf 0 tf-idf
land 0 0 0
sea 0 0 0
water 0 0 0
sun 0 tf-idf tf-idf
moon 0 0 0
类似这里给出的示例表:http://www.cs.duke.edu/courses/spring14/compsci290/assignments/lab02.html。在给定的链接中,它使用了与文档相同的单词,但我需要使用我提到的words
集。
如果特定的单词出现在文档中,那么我将tf-idf
值,否则我将0
放在矩阵中。
你知道我怎样才能建立这样的矩阵吗?Python是最好的,但R也很受欢迎。
我正在使用以下代码,但我不确定我是否在做正确的事情。我的代码是:
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from nltk.corpus import stopwords
train_set = "The sky is blue.", "The sun is bright.", "The sun in the sky is bright." #Documents
test_set = ["sky","land","sea","water","sun","moon"] #Query
stopWords = stopwords.words('english')
vectorizer = CountVectorizer(stop_words = stopWords)
#print vectorizer
transformer = TfidfTransformer()
#print transformer
trainVectorizerArray = vectorizer.fit_transform(train_set).toarray()
testVectorizerArray = vectorizer.transform(test_set).toarray()
#print 'Fit Vectorizer to train set', trainVectorizerArray
#print 'Transform Vectorizer to test set', testVectorizerArray
transformer.fit(trainVectorizerArray)
#print
#print transformer.transform(trainVectorizerArray).toarray()
transformer.fit(testVectorizerArray)
#print
tfidf = transformer.transform(testVectorizerArray)
print tfidf.todense()
我得到非常荒谬的结果,像这样(值只有0
和1
,而我期望的值在0和1之间)。
[[ 0. 0. 1. 0.]
[ 0. 0. 0. 0.]
[ 0. 0. 0. 0.]
[ 0. 0. 0. 0.]
[ 0. 0. 0. 1.]
[ 0. 0. 0. 0.]
[ 1. 0. 0. 0.]]
我也对计算tf-idf
的其他库开放。我只想要一个正确的矩阵,我在上面提到过。
一个R的解决方案可能是这样的:
library(tm)
docs <- c(D1 = "The sky is blue.",
D2 = "The sun is bright.",
D3 = "The sun in the sky is bright.")
dict <- c("sky","land","sea","water","sun","moon")
mat <- TermDocumentMatrix(Corpus(VectorSource(docs)),
control=list(weighting = weightTfIdf,
dictionary = dict))
as.matrix(mat)[dict, ]
# Docs
# Terms D1 D2 D3
# sky 0.5849625 0.0000000 0.2924813
# land 0.0000000 0.0000000 0.0000000
# sea 0.0000000 0.0000000 0.0000000
# water 0.0000000 0.0000000 0.0000000
# sun 0.0000000 0.5849625 0.2924813
# moon 0.0000000 0.0000000 0.0000000
我相信你想要的是
vectorizer = TfidfVectorizer(stop_words=stopWords, vocabulary=test_set)
matrix = vectorizer.fit_transform(train_set)
(如我前面所说,这不是一个测试集,这是一个词汇表。)