It is understandable to me that such a problem arises when highly correlated predictors are entered into regression equation. At this point, predictor sign changes to opposite to correlation coefficient between this predictor and dependent variable.
But it is not clear to me why this happens. What mechanism is the base of of this sign-change? Can anyone explain this to me? I read here:
https://stats.stackexchange.com/questions/1580/regression-coefficients-that-flip-sign-after-including-other-predictors
https://stats.stackexchange.com/questions/31841/coefficients-change-signs
that problem may lie in calculations based on covariance between variables, and more specifically on residues. But I was not able to understand this mechanism even though the described paradoxes are familiar and clear to me. I have not found any literature on this subject.
When you have a regression of Y on X only, the sign of the coefficient will depend on the correlation between X and Y (or, if your prefer, coincide with the sign of their covariance). Then, when you include a further predictor Z, the sign of the coefficient of X will depend on:
Cov(Y;X)*Var(Z)-Cov(Y;Z)*Cov(X;Z).
You may see that, if Z is uncorrelated with either Y or X, the sign of the coefficient will not change (in the latter case, the coefficient itself will not change indeed); otherwise, it depends. Basically with the introduction of a covariate you are no more using X to explain Y, but to explain the residual part of Y that is not explained by Z. If X has a spurious correlation with Y, which is entirely explained by Z, then introducing Z will send the coefficient of X to 0. If you accept this, then accepting that the coefficient may change sign will only be a step of length "epsilon" forward.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression