I have seen that some researchers just compare the difference in R2 in two models: one in which the variables of interest are included and one in which they are excluded. However, in my case, I have that this difference is small (0.05). Is there any method by which I can be sure (or at least have some support for the argument that) this change is not just due to luck or noise?
See almost any reference on adaptive lasso such as found in the attached. Best, D. Booth
Santiago -
So you have compared R-squared for one model versus the other with more independent variables. The difference was small, but did you use adjusted R-squared? What about accounting for heteroscedasticity (weighted least squares, WLS)? "Another Cautionary Note about R2: Its Use in Weighted Least-Squares Regression Analysis,"
John B. Willett and Judith D. Singer,
The American Statistician (Taylor & Francis), Vol. 42, No. 3 (Aug., 1988), pp. 236-238, showed that WLS does not make much difference to R-squared, though I know that for an individual prediction, variance of the prediction error can vary a lot under WLS, but I don't know about the relative contribution of the variance for the estimated 'error' part, sigma.
But another thing that Willett & Singer (1988) points out is that R-squared is not a good measure of fit.
I think what is more important is to use a graphical residual analysis to check fit. You can plot more than one model on the same scatterplot for comparison. You can then do some kind of cross-validation, perhaps saving data to check how well you would have predicted for them, to avoid overfitting. There are various methods.
Note that on pages 109-110 of Brewer, KRW (2002), Combined survey sampling inference: Weighing Basu's elephants, Arnold: London and Oxford University Press, we see that "It is well known that regressor variables, when introduced for reasons other than that they may have appreciable explanatory power, tend to increase rather than decrease the estimates of variance." So Ken Brewer is saying in a survey statistics setting what you also hear in statistical learning, that increased complexity usually increases variance. (Then you also need to remember that you can have omitted variable bias.) So this may be an important consideration for you, though I understand that you were interested in comparing model variance and estimated 'error' variance.
Cheers -
Jim
- Badges
- Science topic