I've been statistically validating the performance of different Deep Learning models. In the process, I could find that there is no statistically significant difference in performance between the pre-trained CNNs. In the absence of such significance, what model am I suppose to choose?
Choosing the correct linear regression model can be difficult. After all, the world and how it works is complex. Trying to model it with only a sample doesn’t make it any easier. In this post, I'll review some common statistical methods for selecting models, complications you may face, and provide some practical advice for choosing the best regression model.
It starts when a researcher wants to mathematically describe the relationship between some predictors and the response variable. The research team tasked to investigate typically measures many variables but includes only some of them in the model. The analysts try to eliminate the variables that are not related and include only those with a true relationship. Along the way, the analysts consider many possible models.
They strive to achieve a Goldilocks balance with the number of predictors they include.
Too few: An underspecified model tends to produce biased estimates.
Too many: An overspecified model tends to have less precise estimates.
Just right: A model with the correct terms has no bias and the most precise estimates.
Statistical Methods for Finding the Best Regression Model
For a good regression model, you want to include the variables that you are specifically testing along with other variables that affect the response in order to avoid biased results. Minitab statistical software offers statistical measures and procedures that help you specify your regression model. I’ll review the common methods, but please do follow the links to read my more detailed posts about each.
Adjusted R-squared and Predicted R-squared: Generally, you choose the models that have higher adjusted and predicted R-squared values. These statistics are designed to avoid a key problem with regular R-squared—it increases every time you add a predictor and can trick you into specifying an overly complex model.
The adjusted R squared increases only if the new term improves the model more than would be expected by chance and it can also decrease with poor quality predictors.
The predicted R-squared is a form of cross-validation and it can also decrease. Cross-validation determines how well your model generalizes to other data sets by partitioning your data.
P-values for the predictors: In regression, low p-values indicate terms that are statistically significant. “Reducing the model” refers to the practice of including all candidate predictors in the model, and then systematically removing the term with the highest p-value one-by-one until you are left with only significant predictors.
Stepwise regression and Best subsets regression: These are two automated procedures that can identify useful predictors during the exploratory stages of model building. With best subsets regression, Minitab provides Mallows’ Cp, which is a statistic specifically designed to help you manage the tradeoff between precision and bias.
Real World Complications
Great, there are a variety of statistical methods to help us choose the best model. Unfortunately, there also are a number of potential complications. Don’t worry, I’ll provide some practical advice!
The best model can be only as good as the variables measured by the study. The results for the variables you include in the analysis can be biased by the significant variables that you don’t include. Read about an example of omitted variable bias.
Your sample might be unusual, either by chance or by data collection methodology. False positives and false negatives are part of the game when working with samples.
P-values can change based on the specific terms in the model. In particular, multicollinearity can sap significance and make it difficult to determine the role of each predictor.
If you assess enough models, you will find variables that appear to be significant but are only correlated by chance. This form of data mining can make random data appear significant. A low predicted R-squared is a good way to check for this problem.
P-values, predicted and adjusted R-squared, and Mallows’ Cp can suggest different models.
Stepwise regression and best subsets regression are great tools and can get you close to the correct model. However, studies have found that they generally don’t pick the correct model.
Dear Sivarama Krishnan Rajaraman,
First of all, thank you for your interesting question. This is a hard selection. I would try different databases and other external validation ways. If you do not have this possibility, I would create different databases with different sizes from the same main database and check the main statistical results in the external validations. The model that needs a smaller database would be the selected one.
I hope this answer will be helpful to you. if you need additional information, please do not hesitate to contact me.
Cheers,
José S. Torrecilla
- Badges
- Science topic