在 R 中创建文档特征矩阵需要很长时间

Question

我正在尝试创建一个在 R 中使用字符级双字母的文档特征矩阵。我代码的最后一行需要永远运行并且永远不会完成。其他线路最多需要不到一分钟的时间。我不知道该怎么办。任何建议将不胜感激。

法典：

library(quanteda)
#Tokenise corpus by characters
character_level_tokens = quanteda::tokens(corpus, 
what = "character",
remove_punct = T,
remove_symbols = T,
remove_numbers = T,
remove_url = T,
remove_separators = T, 
split_hyphens = T)
#Convert tokens to characters
character_level_tokens = as.character(character_level_tokens)
#Keep A-Z, a-z letters
character_level_tokens = gsub("[^A-Za-z]","",character_level_tokens)
#Extract character-level bigrams
final_data_char_bigram = char_ngrams(character_level_tokens, n = 2L, concatenator = "")
#Create document-feature matrix (DFM)
dfm.final_data_char_bigram = dfm(final_data_char_bigram)

length(final_data_char_bigram)
[1] 37115571
head(final_data_char_bigram)
[1] "lo" "ov" "ve" "el" "ly" "yt"

Answer 1

我没有您的输入语料库或可重现的示例，但这里是获得所需结果的方法。 如果这在您的较大语料库上也不起作用，我会感到非常惊讶。 第一种方法在quantead中使用选择和 ngram 构造，而第二种方法使用分词器包中的字符瓦分词器。

library("quanteda")
## Package version: 2.0.1
dfm.final_data_char_bigram <- data_corpus_inaugural %>%
tokens(what = "character") %>%
tokens_keep("[A-Za-z]", valuetype = "regex") %>%
tokens_ngrams(n = 2, concatenator = "") %>%
dfm()
dfm.final_data_char_bigram
## Document-feature matrix of: 58 documents, 545 features (26.4% sparse) and 4 docvars.
##                  features
## docs              fe el ll lo ow wc ci  it  ti iz
##   1789-Washington 20 31 34 12 15  3 29  85 118  5
##   1793-Washington  1  1  7  1  4  1  2   8  12  1
##   1797-Adams      24 52 44 25 24  3 23 160 214  7
##   1801-Jefferson  34 49 60 35 31  7 34  91 116  8
##   1805-Jefferson  26 57 64 27 37  8 34 130 163 11
##   1809-Madison    11 29 37 15 17  1 21  62  82  3
## [ reached max_ndoc ... 52 more documents, reached max_nfeat ... 535 more features ]

# another way
dfm.final_data_char_bigram2 <- data_corpus_inaugural %>%
tokenizers::tokenize_character_shingles(n = 2) %>%
as.tokens() %>%
dfm()
dfm.final_data_char_bigram2
## Document-feature matrix of: 58 documents, 701 features (41.9% sparse).
##                  features
## docs              fe el ll lo ow wc ci  it  ti iz
##   1789-Washington 20 31 34 12 15  3 29  85 118  5
##   1793-Washington  1  1  7  1  4  1  2   8  12  1
##   1797-Adams      24 52 44 25 24  3 23 160 214  7
##   1801-Jefferson  34 49 60 35 31  7 34  91 116  8
##   1805-Jefferson  26 57 64 27 37  8 34 130 163 11
##   1809-Madison    11 29 37 15 17  1 21  62  82  3
## [ reached max_ndoc ... 52 more documents, reached max_nfeat ... 691 more features ]