Hello Matthew,

You didn't indicate the purpose for running a stepwise regression analysis, but there are some very real technical concerns about using step methods in the first place. Chief among these is the reality that a stepwise solution is in no way guaranteed to: (a) identify the best ensemble of independent variables; (b) yield a solution that replicates in other samples. It is a highly opportunistic method which can over-fit models; validation of a resultant model with different data is strongly suggested.

Now, to your questions!

1. In my experience, changing the criteria for variable entry and variable exclusion can result in different final models. (Which brings up another complaint about step methods--many software packages misrepresent the computed probability of a variable's inclusion/exclusion value at a given step). Choosing a more liberal (larger) significance level for entry is less likely to overlook variables than can be constructive contributors. The trick would be to balance that against a lower significance level for exclusion later in the process.

2. The answer depends on the criterion used for variable exclusion. SPSS offers either a fixed F-ratio value or (mis)computed probability as criteria options. A not-yet-included variable having high VIF with variables in the model is likely to have a nonsignificant F-enter. An included variable with high VIF with later entered variables is likely to have a nonsignificant F-remove (and therefore be dropped).

3. If zero is a legitimate value for a predictor, I don't see how dropping one because your population has numerous instances of that value is justified. Of course, a variable with no variation (e.g., all values were zero), would never be included in a step model anyway.

Good luck with your work; don't forget to validate!

I agree with everything that David Morse said. In addition, it is my experience that stepwise regression is generally considered to be an outdated approach, that is out of favor in most fields. So, unless it is indeed widely used in your own field, I would avoid it. Instead, the currently favored standard is to use straightforward multiple regression and concentrate on the variables that have significant effects.

I agree with the two Davids on the problems with stepwise. If you are building a predictive model, then I would suggest lasso regression. If an explanatory model then David Morgan's suggestion is good. For details, see the attached papers. Best wishes, David

@ David Morse Thank you for your suggestions, I am using dozens of lexical measurements generated automatically by tools to predict essay scores, also automatically generated by a scoring system, the purpose of the regression model is to see which measurements can best predict essay scores. I will rerun the same procedure using another group of essays with almost identical scores (statistically non-significant, from a different student population, same genre), to see whether some measurements are constant across essay groups. The problem with the zero values is that they seem to lack the predictive power a predictor should have (the predicted essay scores vary, but the predictor remains zero), so I exclude a measurement if it contains many zero values from entering the model.

@ David L Morgan David Eugene Booth Thank you for your reply and suggestions.

Hi

Please refer to chapter 28, pages 259-267, Howitt, D. and Cramer, D.(2008). Introduction to SPSS.

Regards,

Zuhair

Hi again! I have a BS Statistics degree, and had been practicing it for some time in both the manufacturing industry and academe in the Philippines and other in countries.

Am new here. Amused reading through your interesting comments. The comment of Sir David Morgan caught my imagination, to wit, that "A STRAIGHT-FORWARD MULTIPLE REGRESSION SHOULD BE PERFORMED IN LIEU OF STEP-WISE REGRESSION".

With due respect, may I comment for the first time?

As far as I can recall as taught by our Stat Profs. at the University of the Philippines (UP), the Stepwise Regression is done to SELECT THE SIGNIFICANT INDEPENDENT VARS (IV) SET VIS-A-VIS THE DEPENDENT (DV) OR RESPONSE VARIABLE (Neter, Wasserman; 1989?)

In short, there is a role of Stepwise Regression which a straightforward Multiple Regression can not do - to determine the "BEST" combination of SIGNIFICANT IV that explain the response DV. The straight Multiple Regression would only give you just ONE combination. Hence, no menu of combinations to choose the BEST from. In short, the 2 Stat methods can not replace each other for they have independent, though not mutually exclusive, clearly defined uses.

Hope I made a point here.

Thank you all!

Stepwise regression is a technique that went out of favor in the U.S. at least as long ago as the 1970s. Consider two cases.

First, two of your variables are highly correlated, so that you want to remove one of them But if you have a problem with multi-collinearity, how would know which one to remove? Stepwise regression does this mechanically by comparing effects on the dependent variable, but these aren't reliable due to the multi-collinearity. Further, it ignores the pattern of correlations with the other independent variables. So, the effects of the remaining variables might be quite different, depending on which of the co-linear variables your remove. In this case, you should first consider your theory to determine what to do with the highly correlated variables (perhaps they can be combined into a single indicator). If that doesn't answer your question, then you should at least compare the resulting equations for removing either one.

Second, suppose you remove a variable because it has little impact of the dependent variable. But once again, this fails to take into account its relationships to the other independent variables. In this case, you would be better to leave it in the equation to control for those effects.

Either way, stepwise regression is a limited, mechanical approach that may distort the partial coefficients that you want to observe.

When two of your variables are highly correlated, so that you want to remove one of them. But if you have a problem with multi-collinearity, how would you know which one to remove?

- being highly correlated, either one of the two IVs may be removed. There should be no problem with the choice. But, if there is multi-collinearity issue, can we not use the Variance Inflation Factor (VIF) as a decision criterion? Your thoughts, Sir @David L Morgan...

I would examine the two equations that you get when you remove one variable or the other, to see how that affects the other variables in the equation.

Also, if the two variables are conceptually similar, I would consider combining them into a single measure.

thank you, sir david....