Hi all,
I have been wracking my brain and searching the literature for answers, but yet none seem clear to me so I feel I must ask. Are "fit statistics (e.g., RMSEA, TLI, CFI, etc..) important or even useful for mediation models?
So my issue is as follows:
When I specify the following model in Lavaan, using DWLS estimation (due to ordinal factors and mediators):
outcome ~ a*fact1+fact2+fact3+fact4+fact5+fact6+b*med1+c*med2 # DIRECT EFFECT
med1 ~ a2*fact1+fact2+fact3+fact4+fact5+fact6
med2 ~ a3*fact1+fact2+fact3+fact4+fact5+fact6
#indirect effect(s)
indirect_mod1:=a2*b
indirect_mod2:=a3*c
#total effect(s)
total_crp:=a +(b*c)
I fail to meet the thresholds for the fit statistics, but the modification indicies suggest allowing covariance between mod1 and mod2, or one predicting the other (which technically could make sense), however, this then renders my model saturated and the fit statistics become useless.
Any ideas what I can do please? The output of the model is interesting, but I am concerned that the fit is too far off, but can't think of any other way to assess mediators without SEM. I have tried looking at the model-implied (fitted) covariance matrix and the residuals of a fitted model, but I cant see anything drastically out of the ordinary there.
Below are Fit outputs for both the model before mediator covariance and after, for reference.
Unsaturated model:
lavaan 0.6-7 ended normally after 53 iterations
Estimator DWLS
Optimization method NLMINB
Number of free parameters 54
Number of observations 9619
Model Test User Model:
Test statistic 35.907
Degrees of freedom 1
P-value (Chi-square) 0.000
Model Test Baseline Model:
Test statistic 474.621
Degrees of freedom 24
P-value 0.000
User Model versus Baseline Model:
Comparative Fit Index (CFI) 0.923
Tucker-Lewis Index (TLI) -0.859
Root Mean Square Error of Approximation:
RMSEA 0.060
90 Percent confidence interval - lower 0.044
90 Percent confidence interval - upper 0.078
P-value RMSEA
I can't quite tell if you are just concerned about significance of misfit, or also about the misfit values that are unaffected by sample size. Your sample is so large (9619 it appears) that a significant lack of fit is almost guaranteed for any non-saturated model.
You might test the difference between your df=1 model and models constraining one or both of your mediators to have zero effect, which should establish that mediation is occurring.
And/or, maybe try a model that specifies causal direction between the two mediators (which you say makes sense), and constrain the indirect effect of fact1 to operate only through whichever of the mediators causes the other. For example, if med1 --> med2 --> outcome, then specify fact1 --> med1 but with zero direct effect of fact1 on med2. That model isn't saturated, so it would give you non-zero df, and therefore you'd have fit statistics to interpret. The modification indices suggest that such a model might fit somewhat better.
I was particularly concerned about why, no matter what I did, my model did not "fit" based on the fit statistics like TLI, CFI or RMSEA, yet all my diagnostics showed nothing that seemed to be concerning and would suggest assumption violations or errors in input variables. Knowing that my sample size would likely lead to a misfit for any non saturated model is very useful to know. Thank you so much Burke D. Grandjean I really do appreciate you taking the time to answer my question
Hello Steven,
The test of RMSEA, the sheer value of RMSEA, and the observed value of SRMR are all very promising values from the un-modified model results. The chi-square test is near useless with a sample size as large as yours, as it scales absolute discrepancy between observed & reproduced relationships by a factor proportional to N. Given that, I'd probably ignore the modification indices (since these are all about what change, if any, would have largest impact on chi-square) and suggest that model did an adequate job of reproducing the observed relationships in the data.
If your concern is more about how to make the case for or against moderation, then the suggestions of Burke D. Grandjean are helpful guidelines.
Good luck with your work.
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Techniques
- Structural Equation Modeling