I am trying to train a Back propagation NN with Bayesian regularisation with input vector has 10 features and single output of continuous value between(-1 to +1)(regression). It is a shallow network with a single hidden layer of 30-50 neurons. Dataset size is 2000. Traing and test ratio is 80:20 as BR does not requires validation. Dataset is randomized and randomly picked 20% is chosen for testing. Now I am curious to know if I should go with 10-fold validation or simply run the existing implementation 10 times and present the average performance? What is the norms in ANN application publications?
Another related question if k-fold is suggested then should I first hold out a training set and then use remaining 80% data for traing and testing/validating? And once test all models separately using out of sample hold out data? As BR does not require validation so if I have to use k-fold then I can only use K-1 set for training and remaining one for testing( not validating as we would do in other learning like LM). Please suggest a way that would minimize rework during revision phase of journal submission.
Dear Giridhar Maji
Because it ensures that every observation from the original dataset has the chance of appearing in training and test set. This is one among the best approach if we have a limited input data.
Cross-validation is usually used in machine learning for improving model prediction when we don't have enough data to apply other more efficient methods like the 3-way split (train, validation and test) or using a holdout dataset. This is the reason why our dataset has only 100 data points.
Kind Regards
Qamar Ul Islam
We usually use k-fold CV during the tuning process of the network, e.g. defining the architecture of BPNN or learning rate, etc. For this purpose, it is better to apply the k-fold CV on the training data (i.e. 80% of the dataset).
If you are using Pyrhon, you can use scikit-learn wrapper for keras to use cross validation.
See these links:
https://arxiv.org/pdf/2006.16409.pdf
https://machinelearningmastery.com/k-fold-cross-validation/
https://towardsdatascience.com/why-and-how-to-cross-validate-a-model-d6424b45261f
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression