Hi!
I am a bit confused with the term of "repeated" in repeated CV. In a typical cross validation problem, let's say 5-fold, the overall process will be repeated 5 times: at each time one subset will be considered for validation. In repeated n-fold CV, the above procedure will be repeated in N times. My question is: why and when a typical 5-fold CV needs repetition? and how many times (N?) this repeating process would satisfy our validation? and why a 5-fold CV per se is not enough for validation?
Appreciate any comment in advance.
Regards.
Hi,
N-fold validation are often used to get an estimate of your expected error (+ variance). Statistically speaking, a problem in using n-fold validation is that the n repetitions are not independent of one another (the data you use for training are also used for testing at other times, etc...), meaning that your estimator is biased (maybe not a lot, but it is).
Making multiple n-fold cross validation may lower this bias and give you a better estimate in terms of statistics, although the best (theoretical) way to get rid of the bias is to build n independent data sets and make the training/test procedure on each of them and then average, but this means much, much less data per training/test repetition, meaning that your learned model may not be as good as possible.
If you want more details on this, I would recommend this paper
http://www.jmlr.org/papers/volume5/grandvalet04a/grandvalet04a.pdf
Although other may speak of the problem.
Best wishes
Sebastien
P.S.: in general, a non-repeated 10-fold cross validation is still sufficient to get an idea of what is going on.
Hi,
N-fold validation are often used to get an estimate of your expected error (+ variance). Statistically speaking, a problem in using n-fold validation is that the n repetitions are not independent of one another (the data you use for training are also used for testing at other times, etc...), meaning that your estimator is biased (maybe not a lot, but it is).
Making multiple n-fold cross validation may lower this bias and give you a better estimate in terms of statistics, although the best (theoretical) way to get rid of the bias is to build n independent data sets and make the training/test procedure on each of them and then average, but this means much, much less data per training/test repetition, meaning that your learned model may not be as good as possible.
If you want more details on this, I would recommend this paper
http://www.jmlr.org/papers/volume5/grandvalet04a/grandvalet04a.pdf
Although other may speak of the problem.
Best wishes
Sebastien
P.S.: in general, a non-repeated 10-fold cross validation is still sufficient to get an idea of what is going on.
Dear Sebastien,
Thanks for your comprehensive response. Is it possible to use 5-fold CV as following?:
Repeat CV e.g. for 100 times,and at each time select 20% of observation data randomly. This will generate unbiased/independent sets of data, and at the same time covers multiple CV. This would be similar to do a 5 fold CV for 20 times (?).
Kindly regards.
Hello,
I never really thought about it, but I guess the same problem would happen, as you would use multiple times the same data (you could probably use statistical properties of bootstrap methods in this case, though).
However, I guess both strategies would give you reasonable estimates and a good idea of the method performances. So unless you really want to have specific statistical properties, no one woud probably question the validity of your conclusions.
Best regards
Sebastien Destercke Can you please explain whats the difference between `25-fold cross-validation' and `repeating 5-fold cross-validation 5 times' ?
Thanks
- Badges
- Science topic