- What is the best metric for model selection?
- Does accuracy derived by cross-validation is a good metric? *
- Does the selected model in the model selection process based on the metrics, surely leads to better results?
It depends on which problem you are trying to solve if you are trying to solve a classification problem then commonly use metrics are AUC; precision/Recall, F1 Score. If you dealing with a regression problem then Mean squared prediction error (MSPE) or R Square could be your choice.
Cross-validation is normally not metric for selecting a model it is used to divide the dataset into training and test in the most optimal way.
Of course, it yields better i recommend to check ensemble learning methods to do this automatically.
Hasan -
I like to suggest comparing performances of alternative models - perhaps picked based on subject matter knowledge or some other method, but not forward or backward selection which can leave out good possibilities - by using a "graphical residual analysis," which you can research. Such a comparison, on a single scatterplot, would be with regard to one specific sample, so you do then need to avoid overfitting to that particular sample, and that is where "cross-validation" is important.
Some comparison statistics may automatically assume homoscedasticity. That is not a good blanket assumption. See https://www.researchgate.net/publication/354854317_WHEN_WOULD_HETEROSCEDASTICITY_IN_REGRESSION_OCCUR.
Best wishes - Jim
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics