r——计算字符串中字母的频率和TF-IDF

Question

我有一个名为y的命名字符向量，它看起来类似于

D1        D2        D3        D4        D5 
 "X D X "  "G U V "   "F Q  " "A C U E"  "H I T " 

我想对这个向量做的是创建字母的频率计数和IDF权重。我试着运行这个代码：

dd <- Corpus(VectorSource(y)) #Make a corpus object from a text vector
dtm <- DocumentTermMatrix(dd, control = list(weighting = weightTfIdf))

然而，当我运行此代码时，我会得到错误：

Warning message:
In weighting(x) : empty document(s): 1 2 3 4 5.

所有的文件都有信件，或者至少有空白（我也希望包括在计数中）。我不确定我做错了什么——我能够让这个例子发挥作用——R和手计算中的不同tf idf值。

使用我上面的例子，我想要的是这样的东西：

A C D E F G H I Q T U V X
0 0 1 0 0 0 0 0 0 0 0 0 2 - D1
0 0 0 0 0 1 0 0 0 0 1 1 0 - D2
...

任何帮助都将不胜感激！

Answer 1

你可以在基本R:中完成

y <- c("X D X ",  "G U V ",   "F Q  ", "A C U E",  "H I T ")
names(y) <- paste0("D", 1:5)
# named vector of strings
y
D1        D2        D3        D4        D5 
"X D X "  "G U V "   "F Q  " "A C U E"  "H I T " 
# get rid of spaces, then split every letter
let <- sapply(y, function(x) strsplit(gsub(" ", "", x), ""))
# all possible letters
let_all <- unique(unlist(let))
# uses table on factored x with all possible levels
let_tab <- sapply(let, function(x) table(factor(x, levels=let_all)))
# with some cosmetics
t(let_tab[order(rownames(let_tab)), ])
   A C D E F G H I Q T U V X
D1 0 0 1 0 0 0 0 0 0 0 0 0 2
D2 0 0 0 0 0 1 0 0 0 0 1 1 0
D3 0 0 0 0 1 0 0 0 1 0 0 0 0
D4 1 1 0 1 0 0 0 0 0 0 1 0 0
D5 0 0 0 0 0 0 1 1 0 1 0 0 0

这是你想要的吗？如果是的话，一个完成所有这些的函数：

tabulate_letters <- function(y){
  let <- sapply(y, function(x) strsplit(gsub(" ", "", x), ""))
  # all possible letters
  let_all <- unique(unlist(let))
  # uses table on factored x with all possible levels
  let_tab <- sapply(let, function(x) table(factor(x, levels=let_all)))
  # with some cosmetics
  t(let_tab[order(rownames(let_tab)), ])
}
tabulate_letters(y)
   A C D E F G H I Q T U V X
D1 0 0 1 0 0 0 0 0 0 0 0 0 2
D2 0 0 0 0 0 1 0 0 0 0 1 1 0
D3 0 0 0 0 1 0 0 0 1 0 0 0 0
D4 1 1 0 1 0 0 0 0 0 0 1 0 0
D5 0 0 0 0 0 0 1 1 0 1 0 0 0

Answer 2

我们也可以使用qdapTools 中的mtabulate

library(qdapTools)
mtabulate(strsplit(y, ' '))[-1]
#   A C D E F G H I Q T U V X
#D1 0 0 1 0 0 0 0 0 0 0 0 0 2
#D2 0 0 0 0 0 1 0 0 0 0 1 1 0
#D3 0 0 0 0 1 0 0 0 1 0 0 0 0
#D4 1 1 0 1 0 0 0 0 0 0 1 0 0
#D5 0 0 0 0 0 0 1 1 0 1 0 0 0

在做strsplit之前，我们可以用trimws去除超前/滞后空间。

mtabulate(strsplit(trimws(y), " "))

数据

y <- c("X D X ",  "G U V ",   "F Q  ", "A C U E",  "H I T ")
names(y) <- paste0("D", 1:5)

Answer 3

有一个应用程序：quanteda包。

require(quanteda)
y <- c("X D X ",  "G U V ",   "F Q  ", "A C U E",  "H I T ")
dtm <- dfm(y, toLower = FALSE, verbose = FALSE)
# sort by letter, if that's important
dtm <- dtm[, sort(features(dtm))]
dtm
## Document-feature matrix of: 5 documents, 13 features.
## 5 x 13 sparse Matrix of class "dfmSparse"
##        features
## docs    A C D E F G H I Q T U V X
##   text1 0 0 1 0 0 0 0 0 0 0 0 0 2
##   text2 0 0 0 0 0 1 0 0 0 0 1 1 0
##   text3 0 0 0 0 1 0 0 0 1 0 0 0 0
##   text4 1 1 0 1 0 0 0 0 0 0 1 0 0
##   text5 0 0 0 0 0 0 1 1 0 1 0 0 0

如果你想要tf idf，那也很容易：

tfidf(dtm)
## Document-feature matrix of: 5 documents, 13 features.
## 5 x 13 sparse Matrix of class "dfmSparse"
##        features
## docs          A       C       D       E       F       G       H       I       Q       T       U       V       X
##   text1 0       0       0.69897 0       0       0       0       0       0       0       0       0       1.39794
##   text2 0       0       0       0       0       0.69897 0       0       0       0       0.39794 0.69897 0      
##   text3 0       0       0       0       0.69897 0       0       0       0.69897 0       0       0       0      
##   text4 0.69897 0.69897 0       0.69897 0       0       0       0       0       0       0.39794 0       0      
##   text5 0       0       0       0       0       0       0.69897 0.69897 0       0.69897 0       0       0