If I'm running a regression with one dependent variable and 6 independent variables of which 4 came back as significant, then I added 5 more independent variables, and these turned out to be insignificant. Can the result of both regressions be exactly the same (ie. same R square, same significant variables with same beta coefficient)? or there is something wrong?
Thank you
I do not think that exists such possibility of having EXACTLY the same results by running two almost completely different regressions... from two regressions you could probably have the same significance if the coefficients and might the same r^2 but not the same coefficients... maybe I am wrong, I don't know...
Thank you Dimitrios. I also felt that there is something wrong here.
Just to explain, the two models have the same dependent variable (one variable), while the first model has 6 independent variables and the second model has 5 more independent variables (total of 11 variable), and these 5 newly added variables were not significant.
Unless you have a coefficient of exactly 0 for the 5 added variables, it seems difficult that the coefficients and so on are exactly the same for the old variables. And even if so, the degrees of the freedom should change for the residual term, hence all tests should have different p-values.
Check your analysis of variance table, are your 5 new variables really added? (May be they are perfect linear combinations of the old ones and the software removed them to avoid a singular design matrix).
Something is wrong in the program or in the model. A basic assumption is the independence of all regressor variables (the X's).
Can you list all variables here so we can better understand the model?
Run a correlation analysis for y, x1, x2, .... x11 so that we can see the relationships.
Did you SAS or SPSS or did you use R?
p= number of parameters in the model.
As p increases, (n-p) decreases.
MSE= SSE/( n-p). If the last 5 X's don't contribute at all to SS Regression, then MSE will increase.
Thanks Raid and Emmanuel for the answer. I used SPSS. I will try to check the program and repeat the analysis. Will inform you of the outcome.
I can see this happening if the 5 new variables are redundant (correlation coefficient of exactly 1).
An alternative is to have a correlation coefficient of exactly zero (as suggested by Emmanuel).
Mostly, my bet is with the general consensus that there is something wrong.
Dear Muna,
Regarding the insignificant coefficients that you mentioned, it is not something wired! They found to be insignificant and thus they do not explain your Y, that's all !
@Dimitrios
What is weird is that there was no effect. So I have an equation y=x1+x2 and I get an R2 of 0.68 with the slope of X1 of 2.3789 and the slope of x2 of 5.7692. If I now add a couple of new variables to get y=x1+x2+x3+x4+x5 would you expect to get an R2 of 0.68 and that the slope of x1 would remain 2.3789, and slope of x2 remain at 5.7692? At least in biological data this is not a common effect. It means that x3, x4, and x5 have no correlation with y -- not just close to zero, but equal to zero. Alternatively, that x3, x4, and x5 are perfectly correlated with x1 and/or x2 (not just close, but equal to 1). If it is just close then the coefficients will change.
I did think of another scenario that could produce the observed outcome. It could be caused by round-off error. So an R2 of 0.6 could be anything from 0.556 through 0.6499. You could in this way get values that appear to be the same but aren't the same.
I hope a lot of this answer is not redundant. In multiple regression, added causal variables make R^2 (hence R itself) inflate meaninglessly. Unless some action is taken to correct this spurious R^2 inflation, embarrassingly optimistic untrue outcomes can be, and are, frequently reported. Theil, Henri 1961 Economic Forecasts and Policy. Holland, Amsterdam: North, includes a popular attempt at correcting for this, However, although corrected R^2 is accepted it is not quite the same as R^2 and becomes less so as the number of independent variables increases (well actually it sometimes comes out negative!). In multiple regression it is very important to eliminate as many variables as possible. The number of useless variables can be greatly reduced by techniques such as factor analysis, or principal component analysis. And yes, your results are weird (one in infinity chance?),
I agree with Mr, Raid, it is very strange to get the same exact regression coefficients with different models....
Thank you all for your explanations and suggestions.
I repeated the test using R and results came slightly different. 2 more independent factors were significant and coefficients changed though not substantially. The model looks reasonable now. I think there was something wrong with the SPSS.
Thanks again
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression