删除停用词并标记用于搭配Bigramfinder NLTK

Question

我不断收到此错误

sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or buffer

当我尝试运行此脚本时。不知道出了什么问题。我本质上是从文本文件中读取，过滤掉停用词并使用NLTK标记它们。

import nltk
from nltk.collocations import *
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
stopset = set(stopwords.words('english'))
bigram_measures = nltk.collocations.BigramAssocMeasures()
trigram_measures = nltk.collocations.TrigramAssocMeasures()

text_file=open('sentiment_test.txt', 'r')
lines=text_file.readlines()
filtered_words = [w for w in lines if not w in stopwords.words('english')]
print filtered_words
tokens=word_tokenize(str(filtered_words)
print tokens
finder = BigramCollocationFinder.from_words(tokens)

任何帮助将不胜感激。

Answer 1

我假设sentiment_test.txt只是纯文本，而不是特定的格式。您正在尝试过滤行而不是单词。您应该首先标记化，然后过滤非索引字。

from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
stopset = set(stopwords.words('english'))
with open('sentiment_test.txt', 'r') as text_file:
    text = text_file.read()
    tokens=word_tokenize(str(text))
    tokens = [w for w in tokens if not w in stopset]
    print tokens

希望这有帮助。