Hi!
I am a bit confused with the term of "repeated" in repeated CV. In a typical cross validation problem, let's say 5-fold, the overall process will be repeated 5 times: at each time one subset will be considered for validation. In repeated n-fold CV, the above procedure will be repeated in N times. My question is: why and when a typical 5-fold CV needs repetition? and how many times (N?) this repeating process would satisfy our validation? and why a 5-fold CV per se is not enough for validation?
Appreciate any comment in advance.
Regards.
Hi,
N-fold validation are often used to get an estimate of your expected error (+ variance). Statistically speaking, a problem in using n-fold validation is that the n repetitions are not independent of one another (the data you use for training are also used for testing at other times, etc...), meaning that your estimator is biased (maybe not a lot, but it is).
Making multiple n-fold cross validation may lower this bias and give you a better estimate in terms of statistics, although the best (theoretical) way to get rid of the bias is to build n independent data sets and make the training/test procedure on each of them and then average, but this means much, much less data per training/test repetition, meaning that your learned model may not be as good as possible.
If you want more details on this, I would recommend this paper
http://www.jmlr.org/papers/volume5/grandvalet04a/grandvalet04a.pdf
Although other may speak of the problem.
Best wishes
Sebastien
P.S.: in general, a non-repeated 10-fold cross validation is still sufficient to get an idea of what is going on.
Hi,
N-fold validation are often used to get an estimate of your expected error (+ variance). Statistically speaking, a problem in using n-fold validation is that the n repetitions are not independent of one another (the data you use for training are also used for testing at other times, etc...), meaning that your estimator is biased (maybe not a lot, but it is).
Making multiple n-fold cross validation may lower this bias and give you a better estimate in terms of statistics, although the best (theoretical) way to get rid of the bias is to build n independent data sets and make the training/test procedure on each of them and then average, but this means much, much less data per training/test repetition, meaning that your learned model may not be as good as possible.
If you want more details on this, I would recommend this paper
http://www.jmlr.org/papers/volume5/grandvalet04a/grandvalet04a.pdf
Although other may speak of the problem.
Best wishes
Sebastien
P.S.: in general, a non-repeated 10-fold cross validation is still sufficient to get an idea of what is going on.
Dear Sebastien,
Thanks for your comprehensive response. Is it possible to use 5-fold CV as following?:
Repeat CV e.g. for 100 times,and at each time select 20% of observation data randomly. This will generate unbiased/independent sets of data, and at the same time covers multiple CV. This would be similar to do a 5 fold CV for 20 times (?).
Kindly regards.
Hello,
I never really thought about it, but I guess the same problem would happen, as you would use multiple times the same data (you could probably use statistical properties of bootstrap methods in this case, though).
However, I guess both strategies would give you reasonable estimates and a good idea of the method performances. So unless you really want to have specific statistical properties, no one woud probably question the validity of your conclusions.
Best regards
- Badges
- Science topic