如何在LDA模型中动态传递用户给定的.txt文档？我已经尝试了下面的代码，但它不能给出文档的正确主题。我的.txt的主题与体育相关，因此它应该将主题名称命名为体育。它给出的输出为：
Score: 0.5569453835487366 - Topic: 0.008*"bike" + 0.005*"game" + 0.005*"team" + 0.004*"run" + 0.004*"virginia"
Score: 0.370819091796875 - Topic: 0.016*"game" + 0.014*"team" + 0.011*"play" + 0.008*"hockey" + 0.008*"player"
Score: 0.061239391565322876 -Topic: 0.010*"card" + 0.010*"window" + 0.008*"driver" + 0.007*"sale" + 0.006*"price"*
data = df.content.values.tolist()
data = [re.sub('S*@S*s?', '', sent) for sent in data]
data = [re.sub('s+', ' ', sent) for sent in data]
data = [re.sub("'", "", sent) for sent in data]
def sent_to_words(sentences):
for sentence in sentences:
yield(gensim.utils.simple_preprocess(str(sentence), deacc=True)) # deacc=True removes punctuations
data_words = list(sent_to_words(data))
bigram = gensim.models.Phrases(data_words, min_count=5, threshold=100) # higher threshold fewer phrases.
trigram = gensim.models.Phrases(bigram[data_words], threshold=100)
bigram_mod = gensim.models.phrases.Phraser(bigram)
trigram_mod = gensim.models.phrases.Phraser(trigram)
def remove_stopwords(texts):
return [[word for word in simple_preprocess(str(doc)) if word not in stop_words] for doc in texts]
def make_bigrams(texts):
return [bigram_mod[doc] for doc in texts]
def make_trigrams(texts):
return [trigram_mod[bigram_mod[doc]] for doc in texts]
def lemmatization(texts, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV']):
texts_out = []
for sent in texts:
doc = nlp(" ".join(sent))
texts_out.append([token.lemma_ for token in doc if token.pos_ in allowed_postags])
return texts_out
# Remove Stop Words
data_words_nostops = remove_stopwords(data_words)
# Form Bigrams
data_words_bigrams = make_bigrams(data_words_nostops)
data_lemmatized = lemmatization(data_words_bigrams, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV'])
id2word = gensim.corpora.Dictionary(data_lemmatized)
texts = data_lemmatized
corpus = [id2word.doc2bow(text) for text in texts]
# Build LDA model
lda_model = gensim.models.ldamodel.LdaModel(corpus=corpus,
id2word=id2word,
num_topics=20,
random_state=100,
update_every=1,
chunksize=100,
passes=10,
alpha='auto',
per_word_topics=True)
#f = io.open("text.txt", mode="r", encoding="utf-8")
p=open("text.txt", "r") #document by the user which is related to sports
if p.mode == 'r':
content = p.read()
bow_vector = id2word.doc2bow(lemmatization(p))
for index, score in sorted(lda_model[bow_vector], key=lambda tup: -1*tup[1]):
print("Score: {}t Topic: {}".format(score, lda_model.print_topic(index, 5)))
您的所有代码都是正确的，但我认为您对LDA建模的期望可能有点偏离。您收到的输出是正确的！
首先，你使用了短语"主题名称"；LDA生成的主题没有名称，也没有到用于训练模型的数据标签的简单映射。这是一个无监督的模型，通常你会用没有标签的数据来训练LDA。如果你的语料库包含属于类A、B、C、D的文档，并且你训练LDA模型来输出四个主题L、M、N、O，那么这并不意味着存在一些映射，比如：
A -> M
B -> L
C -> O
D -> N
其次，要注意输出中标记和主题之间的差异。LDA的输出看起来像：
主题1:0.5-0.005*"token_13"+0.03*"token _204"+。。。
主题2:0.07-0.01*"token_24"+0.01*"token _3"+。。。
换句话说，每个文档都被赋予了属于每个主题的概率。每个主题都由以某种方式加权的每个语料库标记的总和组成，以唯一地定义主题。
人们很容易看到每个主题中权重最大的标记，并将这些主题解释为一个类。例如：
# If you have:
topic_1 = 0.1*"dog" + 0.08*"cat" + 0.04*"snake"
# It's tempting to name topic_1 = pets
但这很难验证，而且在很大程度上依赖于人类的直觉。LDA的一个更常见的用法是当你没有标签时，你想识别哪些文档在语义上彼此相似，而不一定要确定文档的正确类标签是什么
经过一番尝试，这对我来说很有效，如果你有什么不同的地方，请评论。
bow_vector = dictionary.doc2bow(preprocess(content))
q= lda_model[bow_vector]
from operator import itemgetter
res = max(q, key = itemgetter(1))[0]
res1 = max(q, key = itemgetter(1))[1]
if (res == 1 ):
print("This .txt file is related to Politics/Government, Accuracy:",res1)
elif (res == 2) :
print("This .txt file is related to sports, Accuracy:",res1)
elif res==3:
print("This .txt file is related to Computer, Accuracy:",res1)
elif..... (so on)
else.