我正在尝试从事TF-idf加权语料库(我希望TF是文档而不是简单计数的比例(。我希望所有经典的文本挖掘库将返回相同的值，但是我得到的值不同。我的代码中是否存在错误(例如，我需要转置一个对象吗？(或TF-IDF的默认参数计数差异不大？
？library(tm)
library(tidyverse)
library(quanteda)
df <- as.data.frame(cbind(doc = c("doc1", "doc2"), text = c("the quick brown fox jumps over the lazy dog", "The quick brown foxy ox jumps over the lazy god")), stringsAsFactors = F)
df.count1 <- df %>% unnest_tokens(word, text) %>%
count(doc, word) %>%
bind_tf_idf(word, doc, n) %>%
select(doc, word, tf_idf) %>%
spread(word, tf_idf, fill = 0)
df.count2 <- df %>% unnest_tokens(word, text) %>%
count(doc, word) %>%
cast_dtm(document = doc,term = word, value = n, weighting = weightTfIdf) %>%
as.matrix() %>% as.data.frame()
df.count3 <- df %>% unnest_tokens(word, text) %>%
count(doc, word) %>%
cast_dfm(document = doc,term = word, value = n) %>%
dfm_tfidf() %>% as.data.frame()
> df.count1
# A tibble: 2 x 12
doc brown dog fox foxy god jumps lazy over ox quick the
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 doc1 0 0.0770 0.0770 0 0 0 0 0 0 0 0
2 doc2 0 0 0 0.0693 0.0693 0 0 0 0.0693 0 0
> df.count2
brown dog fox jumps lazy over quick the foxy god ox
doc1 0 0.1111111 0.1111111 0 0 0 0 0 0.0 0.0 0.0
doc2 0 0.0000000 0.0000000 0 0 0 0 0 0.1 0.1 0.1
> df.count3
brown dog fox jumps lazy over quick the foxy god ox
doc1 0 0.30103 0.30103 0 0 0 0 0 0.00000 0.00000 0.00000
doc2 0 0.00000 0.00000 0 0 0 0 0 0.30103 0.30103 0.30103
您偶然发现了计算术语频率的差异。
标准定义：
tf：术语频率：tf(t(=(术语t的数量t出现在一个 文档(/(文档中的条款总数(。
idf：逆文档频率：idf(t(= log(总数 文档/带有术语t的文档数量(
tf-idf重量是这些数量的产物TF * IDF
看起来很简单，但事实并非如此。让我们计算doc1中的dog一词的tf_idf。
狗的第一个TF：即doc = 0.11111
中的1项/9个术语1/9 = 0.1111111
现在为狗的IDF：(2个文档/1项(的日志。现在有多种可能性，即：log(或自然日志(，log2或log11！
log(2) = 0.6931472
log2(2) = 1
log10(2) = 0.30103
#tf_idf on log:
1/9 * log(2) = 0.07701635
#tf_idf on log2:
1/9 * log2(2) = 0.11111
#tf_idf on log10:
1/9 * log10(2) = 0.03344778
现在变得有趣了。Tidytext根据日志为您提供了正确的加权。tm基于log2返回TF_IDF。我希望从Quanteda的0.03344778值，因为它们的基数为log10。
但要研究Quanteda，它正确返回结果，但使用计数为默认值，而不是比例计数。要获得应有的一切，请尝试以下代码：
df.count3 <- df %>% unnest_tokens(word, text) %>%
count(doc, word) %>%
cast_dfm(document = doc,term = word, value = n)
dfm_tfidf(df.count3, scheme_tf = "prop", scheme_df = "inverse")
Document-feature matrix of: 2 documents, 11 features (22.7% sparse).
2 x 11 sparse Matrix of class "dfm"
features
docs brown fox god jumps lazy over quick the dog foxy ox
doc1 0 0.03344778 0.03344778 0 0 0 0 0 0 0 0
doc2 0 0 0 0 0 0 0 0 0.030103 0.030103 0.030103
看起来更好，这是基于log10。
如果您使用quanteda对参数进行调整，则可以通过更改base参数获得tidytext或tm结果。
# same as tidytext the natural log
dfm_tfidf(df.count3, scheme_tf = "prop", scheme_df = "inverse", base = exp(1))
# same as tm
dfm_tfidf(df.count3, scheme_tf = "prop", scheme_df = "inverse", base = 2)