计算mlR中的AUC留一交叉验证

Question

这是一个简短的问题，只是为了确保我没有用愚蠢的方式来做。我想用auc作为mlr的度量，由于样本量小，我也用LOO。当然，在LOO交叉验证方案中，测试样本总是只有一个实例，因此无法计算auc。当然，我们可以在之后计算它，看看预测，当我们想要在嵌套交叉验证的内循环中使用它作为度量时，问题就发生了。像这样(你必须定义你自己的binaryTask):

require(mlr)    
#for example purposes we will decide which one is better, vanilla LDA or
#vanilla SVM, in the task specified below
bls = list(makeLearner("classif.lda"),makeLearner("classif.svm"))
#modelMultiplexer allows us to search whole parameter spaces between models
#as if the models themselves were parameters
lrn = makeModelMultiplexer(bls)
#to calculate AUC we need some continuous output, so we set 
#predictType to probabilities
lrn = setPredictType(lrn, "prob")
lrn = setId(lrn, "Model Multiplexer")
#here we could pass the parameters to be tested to both SVM and LDA,
#let's not pass anything so we test the vanilla classifiers instead
ps = makeModelMultiplexerParamSet(lrn)
#finally, the resample strategy, Leave-One-Out ("LOO") in our case
rdesc = makeResampleDesc("LOO")
#parameter space search strategy, in our case we only have one parameter:
#the model. So, a simple grid search will do the trick
ctrl = makeTuneControlGrid()
#The inner CV loop where we choose the best model in the validation data
tune = makeTuneWrapper(lrn, rdesc, par.set = ps, control = ctrl, measure = auc, show.info = FALSE) 
#The outer CV loop where we obtain the performace of the selected model 
#in the test data. mlR is a great interface, we could have passed a list 
#of classifiers and tasks here instead and do it all in one go 
#(beware your memory limitation!)
res = benchmark(tune, binaryTask, rdesc, measure = auc)

你不能像那样在两个循环中使用auc。我们如何让mlr在所有测试样本上评估测量值，而不是每次只评估一个样本?

Answer 1

您可以对内部循环使用不同的重采样策略，然后使用auc:

library(mlr)    
ps = makeParamSet(
  makeNumericLearnerParam(id = "cp", default = 0.01, lower = 0, upper = 1)
)
ctrl = makeTuneControlRandom(maxit = 10)
inner = makeResampleDesc("Subsample")
lrn = makeLearner("classif.rpart", predict.type = "prob")
tune = makeTuneWrapper(lrn, resampling = inner, par.set = ps, control = ctrl, measure = auc)
outer = makeResampleDesc("LOO")
r = resample(tune, bc.task, resampling = outer, extract = getTuneResult, measure = auc)

你也可以取一个样本结果，然后计算一个任意的性能指标，例如performance(r$pred, auc)。