Many type of Cross Validation test options (k-fold, 2 fold, 5 fold, 10 fold, training vs testing ratio, LOOCV, LOUOCV, Bootstrap, ...) are used to evaluate the system performance. But which one is more appropriate when dataset includes repeated samples in multiple sessions for each subjects to train and evaluate the machine learning model.
Hi,
Basically, stratified 10-fold cross-validation is the most popular one.
HTH.
Dr. Samer Sarsam
.
i would build folds as partitions of the subjects (one subject, ie all the data related to a given subject, is put in one fold and only one and all subjects are in one fold) trying to balance the fold content (some subjects might have few maesurements and others a lot ... so i'd try to have a balanced mix in each fold)
.
the purpose of that is to have some robust estimate of the performance on new subjects : train on a set of subjects, test on a disjoint set of subjects
.
Hi Soumen,
If you have repeated observations on the same subject in your training data (whether they are repeated over time or space) then your data set no longer satisfies the i.i.d assumption that most machine learning/data mining techniques are based on. Depending on how many times each subject is repeated there are ways to get around this. E.g. if your subject are repeated over time you could create your own folds based on time so that each subject appears once in each fold. Otherwise if all the observation pertaining to a single subject falls in the same fold cross-validation wouldn't work well.
- Badges
- Science topic