Hi,
I have a set of predictors and a continuous outcome variable and I am trying to find the dominant predictors that affect the outcome through multiple regression.
My question is that when modelling the analysis for the predictors and the outcome, if there is a difference in the predictors that have significant associations with the outcome when using different methods of entering variables (for example Enter, Forward and Backward method) then on what basis do you decide which variables should be reported and which method should be used?
Any suggestions or references to the literature would be appreciated.
A number of good ideas can be found here: https://www.google.com/search?client=firefox-b-1-d&q=how+to+choose+regression+methods
and i hope in the attachments. See the Austin and Tu reference in the attached.
STEPWISE Methods are UNSTABLE AND SHOULD NOT BE USED. The attached and their references should give you good advice. Please ask if you have questions. Best, David Booth
A number of good ideas can be found here: https://www.google.com/search?client=firefox-b-1-d&q=how+to+choose+regression+methods
and i hope in the attachments. See the Austin and Tu reference in the attached.
STEPWISE Methods are UNSTABLE AND SHOULD NOT BE USED. The attached and their references should give you good advice. Please ask if you have questions. Best, David Booth
David Eugene Booth Thank you for your quick response. I just read through the first link you have given and checked my models. I noticed that for both my models (i.e. one created by using Forward and one using Backward method of entering variables) the adjusted R2 as well as the histogram and scatter plot of the residuals are almost same. However both the models are predicting different set of variables that are significantly associated to the the outcome.
Can you suggest which model to choose in this case?
Well, don't use enter, forward, backwards. They should be based on your research question(s) and whatever theories you have about your variables. If you don't have any ideas about how the variables relate (which happens with some machine learning applications), you could use something like the lasso and cross-validation to decide how much to shrink the betas. First though, it is probably worth thinking if you have some ideas about your variables.
Astha Shah Since you said predictors I assumed your goal was prediction and my initial reply concerns that. Daniel Wright assumes more explanatory goals and I certainly agree with him in that case. You might look at the Article To Explain or to Predict?
Best, David BoothPS On the difference between Dirichlet and multinomial logistic regression see:
https://stats.stackexchange.com/questions/263202/what-are-the-differences-between-dirichlet-regression-and-log-ratio-analysis. Based on this I would prefer multinomial because I think in your case that is the better choice.
I agree with David Eugene Booth , if the goal is simply to say which set of predictors produces the maximum of some statistics, like adjusted R-square, then best subset regression with exhaustive search (https://rdrr.io/cran/leaps/man/regsubsets.html) works, but requiring predictors to be all-or-none won't make sense for lots of problems. To add another paper to explain v predict (sometimes associated science v machine learning), Pearl has some good thoughts here: .https://ftp.cs.ucla.edu/pub/stat_ser/r489.pdf
One last comment on Daniel Wright 's last post. While in predictive cases subset regression methods (other than the stepwise family) can be very helpful one must use caution. Throwing in all variables in a data set usually doesn't work well. Thought is always needed. Best, David Booth
the best subset checks all sets of the Xs. One of those would include all the variables, one would include none (if is there are k variables, 2^k sets). I see no advantage of the backwards, forwards, and stepwise over searching them all since those algorithms can get caught in local minima. But yes, the inclusion of ANY variable can be problematic!
Astha -
You said that "I noticed that for both my models (i.e. one created by using Forward and one using Backward method of entering variables) the adjusted R2 as well as the histogram and scatter plot of the residuals are almost same." So if I understand correctly, you looked at fairly different models and did graphical residual analyses (estimated residuals on the y-axis and predicted y values on the x-axis) and got about the same results. You mention a "...different set of variables...." Daniel then noted "...it is probably worth thinking if you have some ideas about your variables." In particular, I think it sounds like a possibly complex collinearity situation where perhaps nearly the same information is contained in one set of variables as is contained in another set. You might put your subject matter knowledge to work with that in mind.
To know of you have perhaps close to the best set of variables for prediction, a graphical residual analysis with two or more models' results on the same scatterplot might be very helpful. A graphical residual analysis may also find heteroscedasticity, which I think is most likely if you have good predictions for y. And the cross-validation Daniel also mentioned is a good way to try to avoid overfitting.
Cheers - Jim
- Badges
- Science topic
- Similar topics
- Engineering
- Modeling