使用定制的单词列表从文档语料库中删除单词

Question

tm 包能够让用户"修剪"文档语料库中的单词和标点符号： tm_map（ 语料库文档， 删除单词， 停用词（"英语"） ）

有没有办法为tm_map提供从csv文件中读入并用于代替停用词（"英语"）的定制单词列表？

谢谢。

BSL

Answer 1

让我们拿一个文件 （wordMappings）

"from"|"to"
###Words######
"this"|"ThIs"
"is"|"Is"
"a"|"A"
"sample"|"SamPle"

第一次删除单词;

readFile <- function(fileName, seperator) {
  read.csv(paste0("data\", fileName, ".txt"), 
                             sep=seperator, #"t", 
                             quote = """,
                             comment.char = "#",
                             blank.lines.skip = TRUE,
                             stringsAsFactors = FALSE,
                             encoding = "UTF-8")
}
kelimeler <- c("this is a sample")
corpus = Corpus(VectorSource(kelimeler))
seperatorOfTokens <- ' '
words <- readFile("wordMappings", "|")
toSpace <- content_transformer(function(x, from) gsub(sprintf("(^|%s)%s(%s%s)", seperatorOfTokens, from,'$|', seperatorOfTokens, ')'), sprintf(" %s%s", ' ', seperatorOfTokens), x))
for (word in words$from) {
  corpus <- tm_map(corpus, toSpace, word)
}

如果您想要一个更灵活的解决方案，例如，不仅删除也替换为 then;

#Specific Transformations
toMyToken <- content_transformer( function(x, from, to)
  gsub(sprintf("(^|%s)%s(%s%s)", seperatorOfTokens, from,'$|', seperatorOfTokens, ')'), sprintf(" %s%s", to, seperatorOfTokens), x))
for (i in seq(1:nrow(words))) {
  print(sprintf("%s -> %s ", words$from[i], words$to[i]))
  corpus <- tm_map(corpus, toMyToken, words$from[i], words$to[i])
}

现在是一个示例运行;

[1] "this -> ThIs "
[1] "is -> Is "
[1] "a -> A "
[1] "sample -> SamPle "
> content(corpus[[1]])
[1] " ThIs Is A SamPle "
> 

Answer 2

我的解决方案，可能既麻烦又不优雅：

#read in items to be removed
removalList = as.matrix( read.csv( listOfWordsAndPunc, header = FALSE ) )
#
#create document term matrix
termListing = colnames( corpusFileDocs_dtm )
#
#find intersection of terms in removalList and termListing
commonWords = intersect( removalList, termListing )
removalIndxs = match( commonWords, termListing )
#
#create m for term frequency, etc.
m = as.matrix( atsapFileDocs_dtm )
#
#use removalIndxs to drop irrelevant columns from m
allColIndxs = 1 : length( termListing )
keepColIndxs = setdiff( allColIndxs, removalIndxs )
m = m[ ,keepColIndxs ]
#
#thence to tf-idf analysis with revised m

我们非常感谢寻求任何改进的风格或计算建议。

BSL