We often use leave-one-subject-out cross-validation (LOSOXV) for machine learning experiments involving human subjects to allow for the subject-to-subject variation that occurs and also the tendency for autocorrelation for time series data involving a single subject. I am familiar with the paper by Kohavi recommending 10-fold XV but is there any equivalent paper that supports LOSOXV for this type of subject-dependent data?
I think the best paper on cross validation so far has been written by Sylvain Arlot and Alain Celisse: http://arxiv.org/abs/0907.4728 (Section 6.1)
They also discuss leave-one-out (LOO) cross validation (CV) in detail. LOO is probably the best method to estimate the risk when learning a model. For model selection, however, 10-fold CV should be used (see Breiman and Spector 1992).
http://www.jstor.org/discover/10.2307/1403680
Although both papers I mention do not talk about human subjects, it is clear that, from a statistical point of view, these papers support LOO for your analysis.
This paper maybe of interest to you
"Asymptotic optimality and efficient computation of the leave-subject-out cross-validation" Ann. Statist. Volume 40, Number 6 (2012), 3003-3030.
Full text is available here: http://arxiv.org/abs/1302.4607
Great topic, I am early in my learning curve but I hope to contribute soon. Thanks
- Badges
- Science topic