I understand that we use cross-validation to estimate uncertainty in the model selection phase. Would the technique be used to produce confidence intervals, as well?
I am trying to estimate the plantation area of a crop in an area, using satellite data, and want to choose a model first and then to be sure about the uncertainty in the final model at the end of the season. I finally decided on K-fold cross-validation in a training set [stratifiedly sampled from an area to approximate real situation] for the model selection and bootstrapping method in a test set [separately and randomly sampled from the area] for confidence intervals.
What confuses me is the question of why I don't merge training set with the test set and perform cross-validation to estimate uncertainty. Would cross-validation estimation be biased in that case since we do not sample training set randomly or for other reasons? On the other hand, could it be a more reliable estimation since the sample size is larger if we merge the training set with the test set?
Any resource or suggestion on this topic is highly appreciated, thanks.
Check this! https://stats.stackexchange.com/questions/88809/significance-testing-of-cross-validated-classification-accuracy-shuffling-vs-b?noredirect=1&lq=1
Hello Batuhan,
The bootstrap (resampling) method is just an extension of your idea, such that many samples are taken and (in the traditional approach) confidence intervals are derived from the observed distribution of resultant values. So, yes, re-estimation with a "different" data set can certainly be a path towards determining (what turns out to be pretty robust) estimates of CIs.
Good luck with your work.
Hi Mohammed Deeb, could you inform me more about it? I know we use a test set to prevent bias in our results, and we use Cross-Validation to prevent bias in our split. Do you mean this is why we use a randomly collected test set?
Mohammed Deeb Thank you for clarification. My problem was exactly what Luis Ernesto Cervera-Gómez refer. The dependence of the folds in cross-validation is thought to add bias in error estimation. I guess it is still an open issue for research and which situation it is reliable is not clear yet.
Batuhan Kavlak
If you tend to use stratified cross-validation, then it is not necessary to have an independent test set. Meaning that if you need to merge the training with the test set, the cross-validation alone is enough.
It is worthy to highlight that even if you use 10-fold cross-validation, for instance, to estimate expected performance on unseen data for a model built from the full dataset, there will be bias (which people typically ignore). This due to the fact that each training set in 10-fold cross-validation only contains 90% of the full dataset. This leads to getting an estimation of the expected performance of a classifier that learned from 90% of the full dataset.
Mohammed Deeb, accuracy alone is not a reliable approach to assess the performance of the machine learning algorithm.
HTH.
Dr. Samer Sarsam
- Badges
- Science topic