I used ANFIS for forecasting. Totally, 75% of this data is used for training, 15% is used for testing and 10% is used for validation.
The results were excellent (R^2 =0.98).
Does the change of samples in the training, learning and testing affect the forecasting performance?
Well it is completely depend on your data. If your dataset is consistent, whatever you choose for your 75% in training should be able to create a model that predict your testing data well.
if your training set was, by chance, very close to the testing set, it may overfit and have artificially high r2.
doing x-fold cross-validation reduces the risk of having drawn training set and test set that match too well by chance. But if your overall set is not representative of what you will have to predict, it may works very well even on x-fold cross-validation but failed to predict real data. A good practice is to analyse your overall set to detect possible correlation between individuals (e.g. using a PCA) and to create training, testing and validation sets with very different individuals, to make sure you algorithm generalize training enough.
Dear El-sayed,
I suggest to you a link and attached files in topics.
-Application of neuro-fuzzy networks to forecast innovation ...
https://www.researchgate.net/.../222739236_Application_of_neuro...
Best regards
Dear El-sayed,
Usually, the parts of one's whole data that one takes as training data, validation data and testing data do affect the results of one's model; this situation is more pronounced on small datasets. Hence, various cross validation training schemes are employed to overcome spurious results which arise as a result of the samples which end up in the training, validation, testing datasets.
Hope this helps.
I think it will only effect when training data samples are less.
This problem was accure for me. Using of the cross validation cofficents such as Nash sutklif or baias or MAE you can realize the realy performance. also by Gamma test you can choise the best training set.
- Badges
- Science topic