In designing classifiers (using ANNs, SVM, etc.), models are developed on a training set. But how to divide a dataset into training and test sets? With few training data, our parameter estimates will have greater variance, whereas with few test data, our performance statistic will have greater variance. What is the compromise? From application or total number of exemplars in the dataset, we usually split the dataset into training (60 to 80%) and testing (40 to 20%) without any principled reason. What is the best way to divide our dataset into training and test sets?
Generally, k-fold cross validation; e.g. 10-fold cross validation, is the best . For a minimal dataset, then LOO (leave one out) should be preferred.
The justification that I have always read in the literature is that you have the 60 to 80% for training to better model the underlying distribution and then test the results with the reamining 40-20%. The rationale given is that if you lower the number of samples in the training , the samples for the model being built will have too few samples. One of the shortcomings that I have always found in these techniques is that one of the assumptions is that by random sampling you will achieve independence and also a smooth generation of samples without any bias of the dataset. There are no previous testing to see if the set is unbalanced or not, if there are correlations within samples, etc.
You are right, we need to make a more principled approach to test our assumptions in how we divide the dataset. I think that by adding additional testing before random selection is done might alleviate the problem of such assumptions.
Just a thought
What to do if the amount of data is limited?Maybe as Olegas Niaksu said n-fold cross-validation would be the best approach. But my main concern has not been answered yet. "How to find the optimal splitting proportion?" that can be applied to a particular problem?".. "How does it impact on MSE of the prediction accuracy estimate or accuracy of the classifier?"
How to find the optimal splitting proportion can also be good research problem. Start from 50/50 and go on changing the sets as 60/40, 70/30, 80/20, 90/10. declare all the results and come to some conclusion. In one of my work on Devanagari numeral recognition, I used 50/50 database and got 86.8 % accuracy using MLP NN.
My problem is how to use n-fold cross-validation. NN automatically validates the result as shown in its performance graph. Can guide me regarding 5 fold or 10 fold cross validation please ?
Thank you so much.You elaborated the n-fold cross-validation in very elaborate words which I could easily understand. I will implement it in my work
it depends on the size of our dataset.
If it is large enough, 66% split is a good choice (66% for training and the others for test).
if it is a moderated dataset, 10-fold cross validation (or leave-one-out) can be a good choice.
if your dataset is small then Bootstraping is good.
I think its depend on the size of the data set and if its balance or not , also depend on the main problem needs to built many samples that is mean you should increase the training.
k fold cross valdation is goog and very used, but yo can also make a loop if you want to devide dataset randomly
Generally, k-fold cross validation; e.g. 10-fold cross validation, is the best . For a minimal dataset, then LOO (leave one out) should be preferred.
The application of LOOCV or k-fold cross validation with a large number of folds or with a large set of data in the training phase drastically increases the training time.
To add to this, it still cannot solve the problem of leakage.
As for overfitting, if the data has some noise in it, CV can prove to be counter productive as the noise data will also be taken into account while modelling.
- Badges
- Science topic
- Similar topics
- Engineering
- Modeling