如何在scikit中为cross_validate定制评分指标

Question

我想开发pr_auc作为cross_validate((的评分指标。因此，我遵循了Scikit Learn的用户指南：https://scikit-learn.org/stable/modules/model_evaluation.html#scoring

我的代码如下所示：

from sklearn.datasets import make_classification
from sklearn.model_selection import cross_validate
from xgboost import XGBClassifier
from sklearn.metrics import auc, make_scorer
def cus_pr_auc(x, y):
score=auc(x, y)
return score
X, y = make_classification(n_samples=1000, n_features=2, n_redundant=0,
n_clusters_per_class=2, weights=[0.9], flip_y=0, random_state=7)
model = XGBClassifier(scale_pos_weight=9)
scores = cross_validate(model, X, y, scoring=make_scorer(cus_pr_auc, greater_is_better=True), cv=3, n_jobs=-1)

然而，我收到了以下错误消息：

ValueError:x既不增加也不减少：[1 1 0 0 0 00 1 0 0 0 0 1 0 1 01 0 0 0 00 0 0 00 0 0 00 0 0 0 1 0 1 0 0 00 0 0 0 1 0 0 00 0 0 00 0 0 0 1 0 1 0 0 01] 。

如何修复代码？

Answer 1

将您的度量从auc更改为roc_auc_score，如下所示，即可执行：

from sklearn.datasets import make_classification
from sklearn.model_selection import cross_validate
from xgboost import XGBClassifier
from sklearn.metrics import roc_auc_score, make_scorer
def cus_pr_auc(x, y):
score=roc_auc_score(x, y)
return score
X, y = make_classification(n_samples=1000, n_features=2, n_redundant=0,
n_clusters_per_class=2, weights=[0.9], flip_y=0, random_state=7)
model = XGBClassifier(scale_pos_weight=9)
scores = cross_validate(model, X, y, scoring=make_scorer(cus_pr_auc, greater_is_better=True), cv=3, n_jobs=-1)
scores
{'fit_time': array([0.0627017, 0.0569284, 0.046772 ]),
'score_time': array([0.00534487, 0.00616908, 0.00347471]),
'test_score': array([0.90244639, 0.90242424, 0.94969697])}

您的auc不起作用，因为根据文档：

x坐标。它们必须是单调递增或单调递减的。

因此出现错误(请参阅此处，以获取错误消息的具体来源(。

一般来说，auc是作为auc(fpr,tpr)使用的低级别评分度量。