I am researching the impact of living experiences on depression. In model 1, only personal characteristic variables were included. Model 2 added three living experience variables on the basis of model 1. The final results showed that the R square change was 0.005 and the Sig. F change was 0.004. I want to know whether this small change is enough to indicate that model 2 fits better than model 1, given that the change in R-squared is very small.
This seems like an extremely small R-square increase, especially given that it is based on adding three more predictors. It appears like the additional predictors don't improve the prediction of the outcome very much.
Unless you have some theoretical reason to include a predictor this is all arbitrary, and it doesn't really matter. The model is likely pretty much mis-specified anyway (missing relevant predictors, missing interactions, missing nonlinear relationships, wrong probability model of the outcome), you are reading tea leaves here.
The small change in R2 from .238 to .243 is large enough to be statistically significant by an F-test with 3 and 1853 degrees of freedom. Assuming you set alpha = .05 (or some other value > .004), then yes, model 2 does fit better from a purely statistical point of view.
But whether that small change in R2 is practically important is a separate, non-statistical question. To answer it, one needs content expertise. Even so, when the change in R2 is so tiny, I think it very unlikely content experts would deem it practically important. YMMV.
Bruce Weaver Jochen Wilhelm Christian Geiser thanks for your advice! So is there a standard value to judge that the newly added variable is significant? For example, after I adjusted the above model, the R square change amount became 0.015
Also, is it a problem for the model to decrease in F value as R square increases? I've seen some papers and tutorials where the F value gets smaller while the R square gets larger
Bruce Weaver I mostly agree but would just add that it depends on context. Increasing R^2 from 2% to 3% is a 50% improvement. There are contexts where 2% or 3% R^2 is practically useful. Mostly this would be where there is a cumulative, ongoing effect (possibly in educational context). The real issue here is that small effects are usually poorly estimated unless you have huge samples and huge samples can be biased (so the magnitude of any bias might overwhelm the improvement in R^2).
More promisingly in something like an engineering context you might have a process that accounts for 99% variation. Improving that to 99.5% variation might well be of practical importance (e.g., a low cost change that decreases wastage in production). This is more plausible because physical processes are typically low in variability relative to data from humans.
- Badges
- Science topic
- Similar topics
- Psychology
- Emotion