"k fold cross validation should be done in Artificial neuron network if input data are limited i .In this process data is divide in to K-equal size and the one is use as test and remaining k-1 is used for the training set". But I am confused with that point. What parameters should we compare to evaluate the performance of a neural network and then how we conclude the the performance of our network is best or not? Please help me to understand this.....if any paper related to this please share me to find way from this amazon jungle....
In general, k-fold cross-validation is a statistical technique used for estimating prediction error. In the context of Artificial Neural Networks, you can use several network models, i.e., networks with different learning rates, number of hidden layers or number of hidden neurons, to address a given prediction problem. Then, you estimate the prediction error of each network model by k-fold cross-validation. Finally, the model with the smallest prediction error is chosen.
Rather than papers, you could refer either to statistical or machine learning books.
Neural Networks for Pattern Recognition by Chris Bishop (OxfordUniversity Press) is a seminal book about Neural Networks.
The Elements of Statistical Learning by Hastie, Tibshirani and Friedman (Springer) is a book on Machine Learning techniques described by a strong statistical point of view.
hello friend
k fold cross validation
=================
its one of the common type of the validation schemes.
the entire data set is divided into K blocks- then in turn each block is used as test set and the rest nine will be used as training set.
Then the final result is the average of the K folds.
Thats it.
I hope this answer helps you.
Best regards
Dr.Indrajit Mandal
Dear Dr. Indrajit Mandal
you said: "and the rest nine will be used as training set." this is true if k=10, I am sure you answered the question having in your mind that k=10,
instead your sentence should be: "the rest (k-1) blocks will be used as training set".
I'm not sure about the problems you face. But if you have issues with understanding the concept of k-fold cross-validation, I'd like to refer to wikipedia.
Antonino Staiano already provided you two excellent books (which you can download for free) including a good understanding of neural networks and cross-validation.
If you are not satisfied yet please explain your question more elaborate.
http://en.wikipedia.org/wiki/Cross-validation_(statistics)
Lila:
This is a very good material... you should look at it...
http://www.cs.tau.ac.il/~nin/Courses/NC05/pr_l13.pdf
Dear Lila,
Hope I understood your question. After k-fold cross validation, compute sensitivity, specificity, over all accuracy, MCC etc, for each fold and average it, if it is satisfactory proceed else work on your classifiers parameter for improved result. if possible compute ROC & AUC for each fold and compare. If your input data is limited, then try leave one out cross validation test, it is similar to k-fold validation but the k will be equal to N (total input instance). Note LOOCV is computationally costly if instances are more. Hope the answer helps!
after having collected the k results from the cross validation, the most conservative way to draw conclusions is to keep the worst in terms of training error and to use that trained weights as the best case. This is because the most important role of cross-validation is to avoid any possibility to induce overfitting on your network. By considering the worst training result, this goal is guaranteed.
Take a look into the two papers. You will also find some cited literature regarding your topic.
https://www.researchgate.net/publication/264273404_Fast_Classification_in_Industrial_Big_Data_Environments
https://www.researchgate.net/publication/265906866_Combinatorial_Refinement_of_Feature_Weighting_for_Linear_Classification
Conference Paper Fast Classification in Industrial Big Data Environments
Conference Paper Combinatorial Refinement of Feature Weighting for Linear Cla...
Break data into 10 sets of size n/10. (k=10)
Train on 9 datasets and test on 1.
Repeat 10 times and take a mean accuracy
Hi Lila.
I think what you are referring to is what confuses many people: once you have run k-fold CV and estimated the expected out-of-sample error as the average across the k-folds, say k=5, which specific neural network of the 5 ones you have estimated on diffferent data (and which will have different weights and possibly a different architecture) do you take to actually predict?
The answer is that k-fold cCv is not done to select the best model but to compare the out of sample accuracy and its variance more accurately. So you would still need to pick one, possibly trained on all data. Or you follow our approach and take the ensemble prediction from averaging the predictions for the out-of-sample data from each of the neural networks trained on different folds, getting an ensemble of 5 networks.
Hope this helps
Sven
One remark towards all the comments that suggest to average the prediction error on the k left-out folds:
After k training/evaluation cycles, every sample was part of the held-out set exactly once. Hence, no averaging over folds is needed, but all the predictions on the k held-out sets can simply be concatenated and one performance measure computed.
Actually, you might introduce quite some bias if you average over folds in case of unlucky (or very lucky) splitting.
You can quantify the variance of the outcome by repeating the procedure with different random splitting into k folds (e.g. 50 repetitions of five-fold CV).
Another thing to keep in mind is proper stratification, i.e. balancing the number of instances per class in each fold. Otherwise, a held-out fold may e.g. end up with instances of only one class, in the worst case.
- Badges
- Science topic