Recommendations for Finding the Best Regression Model

Choosing the correct regression model is as much a science as it is an art. Statistical methods can help point you in the right direction but ultimately you’ll need to incorporate other considerations.

Theory

Research what others have done and incorporate those findings into constructing your model. Before beginning the regression analysis, develop an idea of what the important variables are along with their relationships, coefficient signs, and effect magnitudes. Building on the results of others makes it easier both to collect the correct data and to specify the best regression model without the need for data mining.

Theoretical considerations should not be discarded based solely on statistical measures. After you fit your model, determine whether it aligns with theory and possibly make adjustments. For example, based on theory, you might include a predictor in the model even if its p-value is not significant. If any of the coefficient signs contradict theory, investigate and either change your model or explain the inconsistency.

Complexity

You might think that complex problems require complex models, but many studies show that simpler models generally produce more precise predictions. Given several models with similar explanatory ability, the simplest is most likely to be the best choice. Start simple, and only make the model more complex as needed. The more complex you make your model, the more likely it is that you are tailoring the model to your dataset specifically, and generalizability suffers.

Verify that added complexity actually produces narrower prediction intervals. Check the predicted R-squared and don’t mindlessly chase a high regular R-squared!

Residual Plots

As you evaluate models, check the residual plots because they can help you avoid inadequate models and help you adjust your model for better results. For example, the bias in underspecified models can show up as patterns in the residuals, such as the need to model curvature. The simplest model that produces random residuals is a good candidate for being a relatively precise and unbiased model.

In the end, no single measure can tell you which model is the best. Statistical methods don't understand the underlying process or subject-area. Your knowledge is a crucial part of the process!

If you're learning about regression, read my regression tutorial!

* The image of Rodin's The Thinker was taken by flickr user innoxius and licensed under CC BY 2.0.

Thank you Adriana. I was wondering if you have any insights and thought about the issue that I'm facing also.

Amir,

You mentioned a binary variable, is flipping when weighted regression method is used: Does this mean 0's become 1's and 1's are becoming 0's? Could you please explain your situation a little bit more, perhaps provide relevant software output?

Armin,

The coefficient of a binary variable will change while using different statistical methods. Here is the coefficient for different methods:

Method 1: 78.12

Method 2: 74.63

Method 3: 78.94

Method 4: 45.1

Method 5: -17.02

Let me know if you need further information.