I’ve found a lot of different procedures to calculate the AUC confidence interval of a cross-validated model. it may sound quite theoretical but it is not clear to me which parameter these CI refers to. Several options (assuming all cases sampled by the same population):
- the average test AUC when the current trained model is used to make prediction in infinite samples of new cases as large as the training one.
- the test AUC when the trained model is used to make prediction in all new cases of the population
- the average cross-validated AUC of infinite models trained by infinite number of sample of size n
Thank you!
Statistical researchers explain the important statistics very difficult.
See my idea about 100-fold cross-validation for mall sample by "New Theory of Discriminant Analysis after R. Fisher" on RG.
Thank you for the suggestions.
My questions actually wanted to be purely theoretical on exactly what is the parameter that we aim to estimate
Let's consider an regularized logistic regression model (so we don't have hyper-parameters to choose) and we apply a Leave-one-out-cross-validation procedure.
At the end we have exactly a cross-validated probabilistic prediction for each case in the train dataset. We can pool them and calculate the cross-validated AUC of our model applied to our sample.
As we are interested in providing an estimate of the predictive perfomance of our model when applied much beyond our sample, potentially to the whole population of interest, we want to make an inference and provide 95% CI of the AUC of our model.
The question is: which kind of AUC parameter we really want to estimate as performance metric of our model?
- the average test AUC when the current trained model is used to make prediction in infinite samples of new cases as large as the training one.
- the test AUC when the trained model is used to make prediction in all new cases of the population
- the average cross-validated AUC of infinite models trained by infinite number of sample of size n
It is not clear to me which is the one usually considered of true interest.
- Badges
- Science topic