python re.compile and split with ÆØÅ charcters

Question

我是Python的新手。我确实有一个单词列表。它们包含丹麦字母(ÆØÅ)，但是re.compile不理解这些字符。该函数按每个ÆØÅ拆分单词。这些文字是从Twitter和Facebook下载的，并不总是只包含字母。

text = "Rød grød med fløde.... !! :)"
pattern_split = re.compile(r"W+")
words = pattern_split.split(text.lower())
words = ['r', 'd', 'gr', 'd', 'med', 'fl', 'de']

正确的结果应为

    words = ['rød', 'grød', 'med', 'fløde']

我如何得到正确的结果?

完整代码

#!/usr/bin/python 
# -*- coding: utf-8 -*-
import math, re, sys, os
reload(sys)
sys.setdefaultencoding('utf-8')
# AFINN-111 is as of June 2011 the most recent version of AFINN
#filenameAFINN = 'AFINN/AFINN-111.txt'
# Get location of file
__location__ = os.path.realpath(
    os.path.join(os.getcwd(), os.path.dirname(__file__)))

filenameAFINN = __location__ + '/AFINN/AFINN-111DK.txt'
afinn = dict(map(lambda (w, s): (w, int(s)), [ 
            ws.strip().split('t') for ws in open(filenameAFINN) ]))
# Word splitter pattern
pattern_split = re.compile(r"W+")
#pattern_split = re.compile('[ .,:();!?]+')
def sentiment(text):
    print(text)
    words = pattern_split.split(text.lower().strip())
    print(words)
    sentiments = map(lambda word: afinn.get(word, 0), words)
    if sentiments:
        sentiment = float(sum(sentiments))/math.sqrt(len(sentiments))
    else:
        sentiment = 0
    return sentiment

# Print result
text = "ånd ånd med fløde... :)asd "
id = 999
split = "###"
print("%6.2f%s%s%s%s" % (sentiment(text), split, id, split, text))

Answer 1

修改脚本以使用最佳实践:

import csv
import math
import os
import re
LOCATION = os.path.dirname(os.path.abspath(__file__))
afinn_filename = os.path.join(LOCATION, '/AFINN/AFINN-111DK.txt')
pattern_split = re.compile(r"W+")
with open(afinn_filename, encoding='utf8', newline='') as infile:
    reader = csv.reader(infile, delimiter='t')
    afinn = {key: int(score) for key, score in reader}

def sentiment(text):
    words = pattern_split.split(text.lower().strip())
    if not words:
        return 0
    sentiments = [afinn.get(word, 0) for word in words]
    return sum(sentiments) / math.sqrt(len(sentiments))

# Print result
text = "ånd ånd med fløde... :)asd "
id = 999
split = "###"
print('{sentiment:6.2f}{split}{id}{split}{text}'.format(
    sentiment=sentiment(text), id=id, split=split, text=text))

在Python 3中运行此操作意味着text是一个Unicode对象，并且正则表达式将使用re.UNICODE集进行解释。

在Python 2中，您可以使用:

text = u"ånd ånd med fløde... :)asd "

(注意字符串前面的u前缀)和

pattern_split = re.compile(ur"W+", re.UNICODE)

您的AFINN文件仍将被读取为CSV，但在事实之后从UTF8解码key，使用:

with open(afinn_filename, 'rb') as infile:
    reader = csv.reader(infile, delimiter='t')
    afinn = {key.decode('utf8'): int(score) for key, score in reader}

Answer 2

我想指出我的afinn Python包应该可以使用国际字符集，包括丹麦字符集，以及(某些版本的)Python 2和3。有一个英语和丹麦语的单词表。我可能会解决你的问题。

这里Python 2.7或Python 3.4:

>>> from afinn import Afinn
>>> afinn = Afinn(language='da', emoticons=True)
>>> afinn.score(u"ånd ånd med fløde... :)asd ")
4.0
>>> afinn.score('Hvis ikke det er det mest afskyelige flueknepperi...')
-6.0

你可以在这里获得库:

https://github.com/fnielsen/afinn

或pip install afinn的Python包索引