Dear Community,
I am facing a rather odd problem. I am running an OLS multivariate regression, my independent variables change signs compared to signs of univariat regression, although VIF is below 2,0. Can somebody please help me and explain the matter to me? Many thanks.
Do you mean multiple regression or multivariate regression?
This may be some form of suppression effect, one indicator may be that the sign for the predictors switches between bivariate and multiple(!) regression. Another indicator may be that both predictors have the same bivariate sign in the association with the dependent variable (so both correlate either positively or both negatively) but the predictors are negatively correlated with each other (which is odd if you think about it). Typically, suppression leads to an R2 which is larger than the sum of ryx12 and ryx22 which is also odd. But please have a look at the seminal book by Cohen, Cohen, West, & Aiken – Applied Multiple Regression/Correlation Analysis for further explanation.
Yuko -
In multiple regression, if that's what you meant, the 'independent' variables do influence one another, not just by collinearity between two independent variables, but also between groups of variables, and also due to such occurrences as moderator and as Rainer said, suppressor variables, etc. So there are various influences which may occur. VIF itself as a measure for collinearity seems to be a bit nebulous. Further, it is noticeable when a coefficient changes sign, but perhaps more to the point, how much did the coefficients actually change?
Because variables are going to influence one another, their relative importance is made somewhat blurry. But you can consider that to some extent. I recommend looking at the example in the appendix in Galit Shmueli's paper, made available in a public form in the following:
https://www.researchgate.net/publication/48178170_To_Explain_or_to_Predict.
However, for prediction, including variance, I recommend you study a "graphical residual analysis" to consider model fit to your sample, including heteroscedasticity, and consider a "cross-validation" to study whether you may have overfit to your particular sample, when you are trying to model a wider subpopulation or population. Whether or not you wish to include certain variables should be less a matter of VIF than how your variables predict as a group. No one predictor is the point, but how they work together. You want the best predicted-y, that is the model part, that you can have. That means not just not too many variables, and not too few, but also the best ones as a group. If you use principal components, then you have a problem in interpretation, so obtaining the ideal set of variables, and excellent quality data for them, is problematic. (However, if you were to have the ideal variables and the ideal predicted-y from them, with good quality data, then my latest article argues you should likely have heteroscedasticity: https://www.researchgate.net/publication/354854317_WHEN_WOULD_HETEROSCEDASTICITY_IN_REGRESSION_OCCUR.)
You might want to show a graphical residual analysis scatterplot, and some other scatterplots comparing variables, or groups of variables, for discussion, if you are allowed by your data source(s).
Cheers - Jim
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics