I have two models. The second one is the first plus more control variables. I see that the coefficient of a variable that I have in the two models has decreased in this second model and I want to know if this difference is statistically significant. For this to be true, is it necessary that the confidence intervals of the variable in the two models don't include the value of it's coefficients? Or the condition is that not even the limits of the confidence intervals cross with each other?
Hello Santiago,
Any time you add or remove variables from a model, the coefficients associated with a variable common to both models is quite likely to change. The meaning of that change may or may not be important, since you have to judge it in the context of the specific variable ensemble under consideration in a specific model. As well, unstandardized coefficients depend on the scale of both the IV in question and the DV.
That said, you are comparing two dependent estimates (dependent, since they come from the same cases). A very conservative test would be that of non-overlapping CIs for the two estimates. A less conservative test would be an ordinary t- or z- type test. Most programs will furnish the coefficient and an estimated standard error of the coefficient (from whence the CIs are generated).
From this, generate a pooled variance estimate:
Vpooled = (n*SE1^2 + n*SE2^2) / 2
Then t = (Coeff1 - Coeff2) / sqrt(Vpooled / n)
Note that this pooled variance is most likely an overestimate, since the two values are correlated. In absence of information about the value of that correlation, this (independent groups) variance approach is an option.
An alternate option would be to run many bootstrap/resampling trials estimating each of the two models, and recording the observed difference in the target coefficient between the two models to derive an empirical distribution of the observed coefficient differences. Then you could either compare your observed difference to that distribution and determine whether it represented an extreme (and therefore, "rare") instance, or was not noteworthy in magnitude, OR compute the SD of the observed differences and use that as the SE for your statistical comparison. This could be pretty easily done in R (as well as other statistical software).
Good luck with your work.
All David Morse Says is quite true. I might however suggest a different approach since you appear to be interested in building one prediction model. I would run an adaptive lasso variable selection regression on your full set of predictors. This guarantees you the optimal model from your full set of predictors. See Attachments The Refrerence By Boos contains an easy to use R program to do this. This also prevents multicollinearity and overfitting. The two attachments and their references should give full details on how to carry this out. Best wishes, David Booth.
You say you have the "same variable" in both models.
Although the values are identical, the coefficients in the models for this variable do have different meanings! It is usually not very sensible to compare the coefficients of these "same variable" across different models. The meanings may be as different as "velocity" and "mass". How would you compare such coefficients? Is 5 m/s more or less than 6 kg ? Are 3 m/s equal to 3 kg?
- Badges
- Science topic