我已经尝试了很多方法来对csv文件中的列进行名称实体识别，我尝试了ne_chunk，但我无法在类似的列中获得ne_chuck的结果
ID STORY PERSON NE NP NN VB GE
1 Washington, a police officer James... 1 0 0 0 0 1
相反，在使用此代码后，
news=pd.read_csv("news.csv")
news['tokenize'] = news.apply(lambda row: nltk.word_tokenize(row['STORY']), axis=1)
news['pos_tags'] = news.apply(lambda row: nltk.pos_tag(row['tokenize']), axis=1)
news['entityrecog']=news.apply(lambda row: nltk.ne_chunk(row['pos_tags']), axis=1)
tag_count_df = pd.DataFrame(news['entityrecognition'].map(lambda x: Counter(tag[1] for tag in x)).to_list())
news=pd.concat([news, tag_count_df], axis=1).fillna(0).drop(['entityrecognition'], axis=1)
news.to_csv("news.csv")
我收到这个错误
IndexError : list index out of range
所以，我想知道我是否可以使用spaCy来做这件事，这是另一件我不知道的事情。有人能帮忙吗？
似乎您检查区块不正确，这就是为什么您会得到一个关键错误。我猜你想做什么，但这会为NLTK返回的每个NER类型创建新的列。预定义每个NER类型的列并将其归零会稍微干净一点(因为如果不存在NER，这会给你NaN(。
def extract_ner_count(tagged):
entities = {}
chunks = nltk.ne_chunk(tagged)
for chunk in chunks:
if type(chunk) is nltk.Tree:
#if you don't need the entities, just add the label directly rather than this.
t = ''.join(c[0] for c in chunk.leaves())
entities[t] = chunk.label()
return Counter(entities.values())
news=pd.read_csv("news.csv")
news['tokenize'] = news.apply(lambda row: nltk.word_tokenize(row['STORY']), axis=1)
news['pos_tags'] = news.apply(lambda row: nltk.pos_tag(row['tokenize']), axis=1)
news['entityrecognition']=news.apply(lambda row: extract_ner_count(row['pos_tags']), axis=1)
news = pd.concat([news, pd.DataFrame(list(news["entityrecognition"]))], axis=1)
print(news.head())
如果你想要的只是计数，那么以下是更高性能的，并且没有NaNs:
tagger = nltk.PerceptronTagger()
chunker = nltk.data.load(nltk.chunk._MULTICLASS_NE_CHUNKER)
NE_Types = {'GPE', 'ORGANIZATION', 'LOCATION', 'GSP', 'O', 'FACILITY', 'PERSON'}
def extract_ner_count(text):
c = Counter()
chunks = chunker.parse(tagger.tag(nltk.word_tokenize(text,preserve_line=True)))
for chunk in chunks:
if type(chunk) is nltk.Tree:
c.update([chunk.label()])
return c
news=pd.read_csv("news.csv")
for NE_Type in NE_Types:
news[NE_Type] = 0
news.update(list(news["STORY"].apply(extract_ner_count)))
print(news.head())