Say I have two models for a regression task and from each model I get a RMSE value. The first value of RMSE is smaller than the other; however I need to check if the difference is statistically significant to be able to say that one model is better than the other. How can I do it?
Hello Salah,
If the competing regression models are nested (for example, Model 1 uses IVs x1, x2, x3; Model 2 uses IVs x1, x2, x3, x4, and x5), then the question of whether the more inclusive model explains significantly more of the variation in the DV (and therefore, has a smaller MSE) is pretty simple: F = [(R2f - R2r) / (dff = dfr)] / [(1 - R2f) / (N - dff -- 1)]
where:
R2f = R-squared from the "full" model
R2r = R-squared from the "reduced" or smaller, nested model
dff = df for full model (number of IVs used)
dfr = df for reduced model
N = Total number of cases in data set used for analysis
the F-ratio has (dff - dfr, and N - dff - 1) df.
Good luck with your work!
I would start by checking to see if both models satisfy the assumptions of the model. Try a goodness of fit test. Try a LOO (leave one out): a technique where you leave one (or more) values out of the data set and then refit the model and then using the new model see if the model correctly predicts the values that were left out. You could gather more data and use the new data to test model accuracy (though I understand that this is seldom a real option).
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression