I have doubts regarding selection of best suitable predictive regression model. I trained my data with 4 different algorithms with bayesian optimization technique and i used 20/15/10 percent of data for validation and remaining i used for training for these three different validation sets. I am getting good result from the ensemble regression model, decision tree model and also some what satisfiable result with GPR model for all three cases and in these tree model and ensemble models are performing well with my validation data set now am getting doubt about which model i can select for my data set. Because i need to implement the suitable algorithm in real time hardware system and i have to test the performance of the model.
Please help me out regarding this.
Thank you
Hi,
The selected model is the model that has the best performance on the test set (or the validation set in case you have three datasets: training, validation, and test sets).
In general, to evaluate the model, 10-fold cross-validation is recommended to go with. Holdout method is another approach to rely on if you have a large dataset (e.g., 66% training and 34% test set).
Regards,
Dr. Samer Sarsam
Thank you Dr. Samer Sarsam and Desmond Bala Bisandu.
I used 10 fold cross validation and i used three data set test, train and 15% of data for validation. During validation when i compare with training set both decision tree and ensemble models are performing well and losses in training and test set is less there is no much difference between these two losses also (for ex MAE 0.1269 for training and 0.1867 for testing and similarly for RMSE 0.7703 for training and 1.1321 for testing these for tree losses and for ensemble losses training - 0.1817 and testing - 0.3260 and RMSE for training - 0.3498 and testing 0.9003) from this how can i select best suitable one am getting more confusions in this please help me out regarding this. can i go with these models or try with GPR model also here am getting bit higher error comparing with above models.
Sanjeevkumar,
Since you used 10-fold cross-validation, then rely on its result; no need to have independence datasets. Cross-validation performs training, testing, and validation. (Normally, we don't consider the result of the model on the training set.)
If you have 3 datasets--50/25/25; training, validation, and test set, respectively--where the cross-validation is not applied, consider the result of the model on the validation set to select the model.
In sum, you can either rely on the result of cross-validation or the result of the model on the test set (but not the training set).
Note that you can also consider the size of the validation set to be even larger than the test set.
Dr. Samer Sarsam
Thank you Dr. Samer Sarsam
I used 80% of data for training and applied bayesian optimization to reduce the cross validation error and 20% of the data i used for final validation of the model and i got decision tree and ensemble models performing well. is it right what i am following.
Sanjeevkumar T.M ,
Please find my reply below.
Q: I used 80% of data for training and applied bayesian optimization to reduce the cross validation error and 20% of the data
Answer
As I said, load your *full* dataset and then let bayesian optimization method works on this data with the assessment of tenfold cross-validation (cross-validation splits your data into training/test sets, so no need to do that yourself). From there, you can find the best model with the associated hyperparameters.
NB: kindly make sure that bayesian optimization is suitable for your research.
Dr. Samer Sarsam
thank you Dr. Samer Sarsam.
For my research bayesian is suitable because i saw similar papers related to my work. Thank you sir.
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training