I'm performing LOOCV for Support Vector Machines on fMRI data using R and am not sure if the data indices for the input to the training set should be randomized at each step.
While this is just a short line of code, it may influence computation time since I have a rather large group of subjects (around 300 Patients). Any ideas?
Strictly speaking, you have to choose the parameter C at each iteration by k-food CV. You can use the libsvm packages whose computational efficiency is very high.
If you have a rather small set of training data, and a single step (online) learning mechanism, then you should randomize the sequence when presenting the data to avoid, that the learning procedure will generate a "loop" within parameter or weight space (learning the sequence by heart).
When the data is somehow ordered within your training set, and the Leave-One-Out scheme is following the very same ordering, then it might be that your system is adapting to this sequence.
For the case you describe (300 data sets, classifier SVM) the chance, that you will encounter a loop in parameter or weight space, is between low and neglectable.
The results of leave-ONE-out performance estimation do *not* depend on random mixing of the cases: there are only n such surrogate models with the corresponding test possible and they are all evaluated. Nevertheless LOO error estimation is known to be subject to an unneccesary high variance (random) uncertainty. It is therefore recommended to use e.g. iterated k-fold cross validation or out-of-bootstrap validation instead. This allows to lower the the part of the variance that is due to instability of the surrogate models. If your models are very stable, this doesn't matter (but iterated k-fold cv allows you to prove this).
IMHO the cross validation routine in libsvm has certain drawbacks.
1. AFAIK it does not allow to specify splitting the data according to patients. Andreas Alfons' cvTools package provides this option.
2. libsvm does a single split for the whole grid search. With that you have no idea of the random uncertainty due to model (in)stability. (which can lead to choosing between negligible differences compared to the error estimate's random uncertainty e.g. in grid search for the SVM's hyperparameters).
@Nils Goerke: that would apply to model training in general, not only to the training of the surrogate models during resampling validation. So, with LOO surrogate models it should be done if it is necessary for the "usual" model training, right?
- Badges
- Science topic