NLP-识别和替换R中的单词(同义词)

Question

我在r。

中对代码有问题

我有一个带有4列和超过600k观察的数据集（问题），其中一列命名为" V3"。本专栏有"什么一天？"之类的问题。我有第二列的第二个数据集（VOC），其中一个列名称为" word"和其他列名"同义词"。如果在我的第一个数据集（问题）中存在从"同义词"列中的第二个数据集（VOC）中存在的，则我想从'Word''列中替换Word。

questions = cbind(V3=c("What is the day today?","Tom has brown eyes"))
questions <- data.frame(questions)  
                      V3                                                                                            
1 what is the day today?                                                                                             
2     Tom has brown eyes  
voc = cbind(word=c("weather", "a","blue"),synonyms=c("day", "the", "brown"))
voc <- data.frame(voc)
     word synonyms                                                                                                    
1 weather      day                                                                                               
2       a      the                                                                                                   
3    blue    brown 
Desired output
                      V3                        V5                                                                                  
1 what is the day today?  what is a weather today?                                                                                          
2     Tom has brown eyes         Tom has blue eyes

我编写了简单的代码，但不起作用。

for (k in 1:nrow(question))
{
    for (i in 1:nrow(voc))
   {
      question$V5<- gsub(do.call(rbind,strsplit(question$V3[k]," "))[which (do.call(rbind,strsplit(question$V3[k]," "))== voc[i,2])], voc[i,1], question$V3)
   }
}

也许有人会尝试帮助我？：）

我写了第二个代码，但它也不起作用。

for( i in 1:nrow(questions))
{
    for( j in 1:nrow(voc))
      {
        if (grepl(voc[j,k],do.call(rbind,strsplit(questions[i,]," "))) == TRUE)
        {
            new=matrix(gsub(do.call(rbind,strsplit(questions[i,]," "))[which(do.call(rbind,strsplit(questions[i,]," "))== voc[j,2])], voc[j,1], questions[i,]))
            questions[i,]=new   
        }
    }
    questions = cbind(questions,c(new))
}

Answer 1

首先，在程序级别或数据导入期间使用stringsAsFactors = FALSE选项很重要。这是因为r默认值将字符串变成因素，除非您另行指定。因素在建模方面很有用，但是您想对文本本身进行分析，因此您应该确保您的文本不会被强制为因素。

我接近这一点的方式是编写一个可以将每个字符串"爆炸"到向量的函数，然后使用匹配来替换单词。向量再次重新组合到字符串中。

我不确定这将如何获得您的600k记录。您可能会查看一些处理字符串（例如stringr或stringi）的R软件包，因为它们可能具有执行其中一些功能。match的速度往往可以，但是%in%可以是真正的野兽，具体取决于字符串的长度和其他因素。

# Start with options to make sure strings are represented correctly
# The rest is your original code (mildly tidied to my own standard)
options(stringsAsFactors = FALSE)
questions <- cbind(V3 = c("What is the day today?","Tom has brown eyes"))
questions <- data.frame(questions)  
voc <- cbind(word = c("weather","a","blue"),
             synonyms = c("day","the","brown"))
voc <- data.frame(voc)
# This function takes:
#  - an input string
#  - a vector of words to replace
#  - a vector of the words to use as replacements
# It returns a list of the original input and the changed version    
uFunc_FindAndReplace <- function(input_string,words_to_repl,repl_words) {
    # Start by breaking the input string into a vector
    # Note that we use [[1]] to get first list element of strsplit output
    # Obviously this relies on breaking sentences by spacing
    orig_words <- strsplit(x = input_string,split = " ")[[1]]
    # If we find at least one of the words to replace in the original words, proceed
    if(sum(orig_words %in% words_to_repl) > 0) {
        # The right side selects the elements of orig_words that match words to be replaced
        # The left side uses match to find the numeric index of those replacements within the words_to_repl vector
        # This numeric vector is used to select the values from repl_words
        # These then replace the values in orig_words
        orig_words[orig_words %in% words_to_repl] <- repl_words[match(x = orig_words,table = words_to_repl,nomatch = 0)]
        # We rebuild the sentence again, and return a list with original and new version
        new_sent <- paste(orig_words,collapse = " ")
        return(list(original = input_string,new = new_sent))
    } else {
        # Otherwise we return the original version since no changes are needed
        return(list(original = input_string,new = input_string))
    }
}
# Using do.call and rbind.data.frame, we can collapse the output of a lapply()
do.call(what = rbind.data.frame,
        args = lapply(X = questions$V3,
                      FUN = uFunc_FindAndReplace,
                      words_to_repl = voc$synonyms,
                      repl_words = voc$word))
>
                original                      new
1 What is the day today? What is a weather today?
2     Tom has brown eyes        Tom has blue eyes