I have to calculate the CI of the AUC (Roc) for a series of classifiers (e.g. Lasso, Random Forest, SVM) learned using the same test dataset, in order to identify the best model for this problem (prediction of a dichotomous variable).
Considering the small size of the dataset, I used the less known but almost unbiased Leave-Pair-Out-Cross-Validation (Airola et al.,2010). In brief, you learn and validate the model holding-out all possible combinations made of one case of one class and another case of the other class. Thus the (several) folds of the cross-validation procedure overlaps, with each case re-used in different validation folds.
The AUC is calculated according the Wilcoxon statistic (i.e as the average of all folds results, considering 1 if the p(C1) > p(C2) and 0 otherwise) as indicated in the papers where LPOCV was proposed. However I couldn't find in the literature any formal procedure to calculate the CI.
I thus decided to use a bootstrap procedure but I'm not sure that the design of the resampling procedure I applied is the correct one given the structure of LPOCV.
I independently resampled (with repetition) the cases belonging to each of the two classes. Then I considered the folds according to the combination of two resamples, calculating the AUC as before. I performed these several times to obtain a distribution of bootstrapped AUC.
Is this correct? are there better resampling scheme to apply in this case?
Thank you!
http://www.jmlr.org/proceedings/papers/v8/airola10a/airola10a.pdf
Note that you will get a confidence interval by this bootstrapping, but that this will only be useful if you have enough samples in your dataset, i.e. it represents your population well.
Bootstrapping cannot overcome the problem of generalization if you have a very small sample size compared to your population.
Hello, Massimo,
I agree with Vikas here, building a stronger model this way can be deceiving. You are only recycling the same (small) set of data to fit a better model in many ways. In this case, the AUC may be only a measure of self-perfection...
Regarding the whole problem, I suppose you will find all the answers and tools (including how to calculate the CI for the AUC) here:
https://cran.r-project.org/web/packages/pROC/pROC.pdf
Cheers!
C.
Hi Vikas and Cristian,
thank you for your answers!
Yes, you're absolutely right. I wrote "small" referring the usual sample size used in ML nowadays :-). It is actually more than 120 cases, with more than 2000 combinations/validation folds.
Regarding the R package, there is also a specific package to calculate the CI of the AUC in case of overlapping cross-validated folds (cvAUC). However, it gives me no variation in the CI considering the scheme of the LPOCV I used. Yes, LOPCV generates all the possibile combinations of interest from the available dataset but of course I care about the expected CI when generalized to a totally new test data.
Thanks again!
https://cran.r-project.org/web/packages/cvAUC/index.html
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis