I have two multiple regression models. y ~ sim + ctl + covariate and y ~ sim + ctl + covariate. Between both models, variable 'ctl' differs in value. Could you suggest me the methods for comparing multiple regression models having same predictor variables? Should I directly compare R2 or adjusted R2 values between these two models? Thank you very much for your kind help.
Edit:
My predicted variable, y, is always the same between two models. I tried to vary the values of ctl predictor variable and look at how much it affects R2. They do give different R2 values. My doubt is if the difference between R2 is enough to say one ctl is better than other in predicting y OR do I need to look at other things and do other tests? Thank you very much.
Why not just combine the data sets and run one model? I assume the y values are different. I'm not sure I understand what you're doing.
Jatin -
Perhaps you want to research "multivariate regression." That's multivariate multiple regression, where multiple regression refers to more than one regressor, and multivariate refers to more than one y variable. Not an area I know well, but it sounds like what you are doing, I think, where you are using the same regressors and different dependent variables in a matrix. ???
I think the purpose of multivariate regression is largely to do hypothesis testing, however, and note that p-values are sample size dependent, so without a type II error analysis, you can be mislead. Further, heteroscedasticity often makes sense, and for cross-sectional surveys is generally substantial, so OLS should often be replaced by WLS.
So perhaps you want to just compare coefficients and their standard errors between the two sets of regression equations so that you can see how they vary in relating the same independent variables to different dependent variables.
Perhaps you can do some graphical analyses, as they are often informative.
You also do not want to have omitted variable bias in either regression. You could also look at the estimated variances of the prediction errors, and validate results with test data.
I'm not sure how valuable R-square would be, as that is a rather volatile statistic.
I hope that I have not too badly misinterpreted your question, and that some of this might be of some use.
Cheers - Jim
Thank you very much for your replies, and sorry for confusion.
My predicted variable, y, is always the same between two models. I tried to vary the values of ctl predictor variable and look at how much it affects R2. They do give different R2 values. My doubt is if the difference between R2 is enough to say one ctl is better than other in predicting y OR do I need to look at other things and do other tests? Thank you very much. I hope my question makes more sense now.
Still not completely clear, but if you are changing something and want to know if the change in R-square is substantial, as I said, that statistic is rather volatile anyway, I think, and I don't know that you can reliably measure or evaluate the meaningfulness of any fairly small change.
If you don't have omitted variable bias, estimating variance of the prediction errors can tell you much more about how you are doing, I think. Validation by using test data when you can might be helpful. But it depends on exactly what you are trying to do.
Cheers.
I am not entirely sure how you want to compare them, but there seem to be a few options. If you are trying to compare the two ctl variables in the same model to see which accounts for the largest amount of variance, you can include or exclude them and look at the change in rsquare value. If you are trying to compare model by model, then yes, you can look at which model has the highest r-square (but some of the rsquare in the individual models would be explained by your covariates, keep that in mind).
Jatin,
You should look behind R2.
Compute analysis of variance (or deviance) tables for fitted models.
comparing is possible based on RSS (residual sum of squares) and degree of freedom (Df), when You use linear regression.
Let me assume that ctl of the first model is the quantitave variable , and ctl of the second model is the factor with three levels ( low-medium-high). So Df's are different and RSS's are are different too. Use F-test.
What statistical tools is used? Some tools have function for compare two models based on the same set of the response variable and the same model class.
The result is inerperted as: what model reduces variability of the response variable better.
I'm sorry for being so dense but you did two experiments with different controls in each?
Dear absolutely model results are change due to different values of ctl, If ctl is significant in both models. If you want to compare such models, I think you should note the MSE and adjusted R2 without considering other model assumptions. If you want to get better results, I think you should consider the other models assumptions and note the results.
I believe that if you ask a simple question then it becomes easier to get a simple answer. That said then, why don't you explain exactly what you are doing. I don't really understand the model you are building. Give us the name of the variables and what you want to explain or predict!
Read about Akaike Information Criterion (AIC). That will help you to decide which model is better explaining a dependent/predicted/outcome variable.
For instance, using a entry-forced method in a multivariable model you'll see two AIC and the smaller the number the better is the model explaining your outcome. Of course, be aware that AIC is also used to variable selection when you choose a backward elimination method. Just do not use forward selection.
You also will find more information in the TRIPOD statement (Annals of Internal Medicine). Everyone working with multivariable models should know this last paper.
Cheers!