问题
我正在努力进行学习,以对问题进行分级问题进行评估的预测,但是组评估了模型性能。
更具体地说,估算器输出一个连续变量(非常像回归器(
> y = est.predict(X); y
array([71.42857143, 0. , 71.42857143, ..., 0. ,
28.57142857, 0. ])
但是评分函数需要通过查询进行聚合,即分组预测,类似于发送到GridSearchCV
的groups
参数以尊重折叠分区。
> ltr_score(y_true, y_pred, groups=g)
0.023
障碍
到目前为止一切都很好。在向GridSearchCV
提供自定义评分函数时,事情会向南移动,我无法根据CV折叠动态更改groups
参数:
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import make_scorer
ltr_scorer = make_scorer(ltr_score, groups=g) # Here's the problem, g is fixed
param_grid = {...}
gcv = GridSearchCV(estimator=est, groups=g, param_grid=param_grid, scoring=ltr_scorer)
解决这个问题的最少骇客方法是什么?
一个(失败(方法
在一个类似的问题中,一个评论问/建议:
为什么您不能仅存储{分组列}本地并在必要时使用索引来使用分离器提供的火车测试索引?
OP回答"看起来可行"。我认为这也是可行的,但无法使其起作用。显然,GridSearchCV
将首先消耗所有交叉验证拆分索引,然后执行分裂,拟合,PERES和SCORINGS。这意味着我无法(似乎(尝试猜测创建当前拆分子选择的原始索引。
为了完整,我的代码:
class QuerySplitScorer:
def __init__(self, X, y, groups):
self._X = np.array(X)
self._y = np.array(y)
self._groups = np.array(groups)
self._splits = None
self._current_split = None
def __iter__(self):
self._splits = iter(GroupShuffleSplit().split(self._X, self._y, self._groups))
return self
def __next__(self):
self._current_split = next(self._splits)
return self._current_split
def get_scorer(self):
def scorer(y_true, y_pred):
_, test_idx = self._current_split
return _score(
y_true=y_true,
y_pred=y_pred,
groups=self._groups[test_idx]
)
用法:
qss = QuerySplitScorer(X, y_true, g)
gcv = GridSearchCV(estimator=est, cv=qss, scoring=qss.get_scorer(), param_grid=param_grid, verbose=1)
gcv.fit(X, y_true)
它不起作用,self._current_split
在最后生成的拆分处修复。
,因为我知道评分值是对(值,组(,但是估算器不应与组一起使用。让它们在包装纸中切下,但将它们留给得分手。
简单的估计器包装器(可能需要一些抛光以完全合规(
from sklearn.base import BaseEstimator, ClassifierMixin, TransformerMixin, clone
from sklearn.linear_model import LogisticRegression
from sklearn.utils.estimator_checks import check_estimator
#from sklearn.utils.validation import check_X_y, check_array, check_is_fitted
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import make_scorer
class CutEstimator(BaseEstimator):
def __init__(self, base_estimator):
self.base_estimator = base_estimator
def fit(self, X, y):
self._base_estimator = clone(self.base_estimator)
self._base_estimator.fit(X,y[:,0].ravel())
return self
def predict(self, X):
return self._base_estimator.predict(X)
#check_estimator(CutEstimator(LogisticRegression()))
然后我们可以使用它
def my_score(y, y_pred):
return np.sum(y[:,1])
pagam_grid = {'base_estimator__C':[0.2,0.5]}
X=np.random.randn(30,3)
y=np.random.randint(3,size=(X.shape[0],1))
g=np.ones_like(y)
gs = GridSearchCV(CutEstimator(LogisticRegression()),pagam_grid,cv=3,
scoring=make_scorer(my_score), return_train_score=True
).fit(X,np.hstack((y,g)))
print (gs.cv_results_['mean_test_score']) #10 as 30/3
print (gs.cv_results_['mean_train_score']) # 20 as 30 -30/3
输出:
[ 10. 10.]
[ 20. 20.]
更新1:hackers way ,但估算值没有变化:
pagam_grid = {'C':[0.2,0.5]}
X=np.random.randn(30,3)
y=np.random.randint(3,size=(X.shape[0]))
g=np.random.randint(3,size=(X.shape[0]))
cv = GroupShuffleSplit (3,random_state=100)
groups_info = {}
for a,b in cv.split(X, y, g):
groups_info[hash(y[b].tobytes())] =g[b]
groups_info[hash(y[a].tobytes())] =g[a]
def my_score(y, y_pred):
global groups_info
g = groups_info[hash(y.tobytes())]
return np.sum(g)
gs = GridSearchCV(LogisticRegression(),pagam_grid,cv=cv,
scoring=make_scorer(my_score), return_train_score=True,
).fit(X,y,groups = g)
print (gs.cv_results_['mean_test_score'])
print (gs.cv_results_['mean_train_score'])