我正在使用CountVectorizer/logist回归进行文本处理，并比较没有预处理和预处理的f1分数。我想使用regex进行预处理，所以我构建了一个如下的代码
def better_preprocessor(s):
lower = s.lower()
lower = re.sub(r'^w{8,}$', lambda x:x[:7], lower)
return lower
def a():
cv = CountVectorizer()
train = cv.fit_transform(train_data)
features = cv.get_feature_names()
cv_dev = CountVectorizer(vocabulary = features)
dev = cv_dev.fit_transform(dev_data)
print(features)
lgr = LogisticRegression(C=0.5, solver="liblinear", multi_class="auto")
lgr.fit(train, train_labels)
lgr_pred = lgr.predict(dev)
score = metrics.f1_score(dev_labels, lgr_pred, average="weighted")
print('No preprocessing score:', score)
cv_im = CountVectorizer(preprocessor=better_preprocessor)
train_im = cv_im.fit_transform(train_data)
features_im = cv_im.get_feature_names()
cv_im_dev = CountVectorizer(preprocessor=better_preprocessor, vocabulary = features_im)
dev_im = cv_im_dev.fit_transform(dev_data)
lgr.fit(train_im, train_labels)
lgr_pred_im = lgr.predict(dev_im)
score_im = metrics.f1_score(dev_labels, lgr_pred_im, average="weighted")
print('Preprocessing score', score_im)
print(len(features)-len(features_im))
print(features_im)
a()
我试图将大于或等于8的单词长度截断为7，但当我使用get_feature_names检查词汇表时，并没有任何变化。我不知道该在哪里解决这个问题。
您不需要任何正则表达式。使用
def better_preprocessor(s):
if len(s) >= 8:
return s.lower()[:7]
else:
return s.lower()
re.sub(r'^w{8,}$', lambda x:x[:7], lower)代码采用lower字符串并尝试匹配^w{8,}$:
^-字符串的开头w{8,}-八个或八个以上单词字符$—字符串结束
lambda x:x[:7]然后尝试获取匹配(其中x是匹配数据对象(，然后尝试对匹配数据对象进行切片。也许你想用x.group()[:7]，但在这里还是有些过头了。
如果你计划从一个字符串中提取所有单词并截断它们，你需要指定一个单词对你来说是什么，并使用
def better_preprocessor(s):
return re.sub(r'b(w{7})w+', r'1', s.lower())
查看regex演示
b-单词边界(w{7})-组1(从替换模式中用1表示(：七个字的字符w+-1+字字符
以下是使用analyzer参数的方法：
from sklearn.feature_extraction.text import CountVectorizer
def better_preprocessor(s):
lower = s.lower().split()
lower = [x[:7] for x in lower]
for l in lower:
yield l
lower = ["hello how are you doing today crocodilesaway","hello how are you"]
cv = CountVectorizer(analyzer=better_preprocessor, )
cv.fit_transform(lower)
cv.get_feature_names()
['are', 'crocodi', 'doing', 'hello', 'how', 'today', 'you']