在pandas-df中用其unicode名称替换特殊字符的更有效方法

Question

我有一个大的pandas数据帧，希望对其进行彻底的文本清理。为此，我编写了下面的代码，用于评估字符是表情符号、数字、罗马数字还是货币符号，并将其替换为unicodedata包中的唯一名称。

不过，代码使用了双for循环，我相信肯定有比这更有效的解决方案，但我还没有弄清楚如何以矢量化的方式实现它。

我当前的代码如下：

from unicodedata import name as unicodename 
def clean_text(text):
for item in text:
for char in item: 
# Simple space
if char == ' ':
newtext += char 
# Letters
elif category(char)[0] == 'L':
newtext += char
# Other symbols: emojis
elif category(char) == 'So':
newtext += f" {unicodename(char)} "
# Decimal numbers 
elif category(char) == 'Nd':
newtext += f" {unicodename(char).replace('DIGIT ', '').lower()} "
# Letterlike numbers e.g. Roman numerals 
elif category(char) == 'Nl':
newtext += f" {unicodename(char)} "
# Currency symbols
elif category(char) == 'Sc':
newtext += f" {unicodename(char).replace(' SIGN', '').lower()} "
# Punctuation, invisibles (separator, control chars), maths symbols...
else:
newtext += " "

目前，我正在我的数据帧上使用这个函数，并应用：

df['Texts'] = df['Texts'].apply(lambda x: clean_text(x))

样本数据：

l = [
"thumbs ups should be replaced: 👍👍👍",
"hearts also should be replaced:  ❤️️❤️️❤️️❤️️",
"also other emojis: ☺️☺️",
"numbers and digits should also go: 40/40",
"Ⅰ, Ⅱ, Ⅲ these are roman numerals, change 'em"
]
df = pd.DataFrame(l, columns=['Texts'])

Answer 1

一个好的开始是不做那么多工作：

1. 一旦解析了字符的表示，就缓存它。(lru_cache()会为您执行此操作(
2. 不要给category()和name()打太多次电话

from functools import lru_cache
from unicodedata import name as unicodename, category

@lru_cache(maxsize=None)
def map_char(char: str) -> str:
if char == " ":  # Simple space
return char
cat = category(char)
if cat[0] == "L":  # Letters
return char
name = unicodename(char)
if cat == "So":  # Other symbols: emojis
return f" {name} "
if cat == "Nd":  # Decimal numbers
return f" {name.replace('DIGIT ', '').lower()} "
if cat == "Nl":  # Letterlike numbers e.g. Roman numerals
return f" {name} "
if cat == "Sc":  # Currency symbols
return f" {name.replace(' SIGN', '').lower()} "
# Punctuation, invisibles (separator, control chars), maths symbols...
return " "

def clean_text(text):
for item in text:
new_text = "".join(map_char(char) for char in item)
# ...