Hi, I am running a multi-group SEM analysis using AMOS to investigate the moderation effect of 3 categorical varaibles (discipline: hard/soft, experience: low/high, participation in training: yes/no) on a model of continuous variables.
First, I checked the measurement and the structural models for the whole sample, and they showed a good fit.
Then I run three seperated measurment invariance test to see whether the model is invariant across groups of different disciplines, experience, participation in training.
The results showed the models were invariant in terms of delta CFI (all delta CFI < .01). In other words, the groups are not different at the model level, but they may be different at the strength of each path regression weight.
I run path by path analysis (constraining one regression path at a time) and checked the delta chi-squared, the results yielded significant p value (p < .05) in some paths.
My question is:
If the results showed that the models are invariant at the model leveIs, is it OK to check path by path analysis to see whether the groups are different in terms of the strength of the regression weight?
Should also delta CFI be > .01 to reject the assumption of invariance when constraining one regression path at a time?
Any help would be appreiected, as I am still beginner in the multigroup analysis.
Thanks,
Tahani
Hello Tahani,
I came across your thread several times and one time I had already started typing but then....
Ok here we go: First I would wonder why you relied on the CFI-threshold? Yes, one can have hours-long debates about the usefulness of statistical tests of differences but I am always puzzled if someone routinely uses significance test--also for testing moderator or interaction effects but when it comes to testing invariance, then suddenly the criteria change.
Hence I would use the usual statistical invariance testing approach.
Start with the configural invariance test and strive for a non-significant chisquare test. Yes, this will probably hurt (if it is significant) but that's not the end of the story. You can re-specify the model and come up with a reasonable new and preliminary model (which as to be replicated and tested in the future).
Then test for metric invariance for all factor loadings. If the overall difference test is non-significant proceed to substantial differences (e.g., for the effects), if not, use the modification indices to find the invariant loadings.
In this paper we explicitly outline this sequence, and how to do it and which forms of invariance are necessary (in 99% of the cases, you need metric invariance).
In the paper however, we had a badly fitting model. This was a long time ago and I changed my opinion about model testing quite.
HTH
Holger
Steinmetz, H., Schmidt, P., Tina-Booh, A., Wieczorek, S., & Schwartz, S. H. (2009). Testing measurement invariance using multigroup CFA: Differences between educational groups in human values measurement. Quality and Quantity, 43(4), 599-616. doi:10.1007/s11135-007-9143-x
Thanks so much Holger for your comprehensive answer!
I followed your suggestions and I run configural, metric and scalar invariant tests. In each time, I got insignificant chisquare test. This means that the groups are invariant, and the model works well across groups. Does this result indicate that there is no moderation effect?
What makes me confused is that I learnt from some resources that when we have two groups similar at the model level (global level), we can still check invariance at the path level (local level), but we have to interpet results WITH CAUTION.
Accordingly, I constrained one regression path at a time and checked delta chisquared each time. I got significant chisquare test, and the results showed that this regression path is significant in one group, but not in the other group.
What do you think? Please correct me, if I am wrong.
If it can be interpreted that there is no moderation, there are also studies that restrict some loads in case they do not meet the criteria of equivalence to evaluate the partial imvariance.
Hi,
1) you don't need to test for scalar invariance. Scalar invariance is a presumption when you investigate mean differences
2) I am not quite clear whether you really tested for DIFFERENCES in the chisq test when comparing one step (e.g., configural) with the more restricted step (e..g, metric).
3) No, measurement invariance concerns the measurement model not the effect of the latent variables. This is tested in the third step (and would require a SEM (instead of the mere CFA) with latent variables connected by effects (instead of correlations). Hence, you would have three steps (I would do all of them in a SEM (instead of CFA)
a) test the configural model
b) restrict loadings to be equal (and test difference)
c) restrict effects based on the bestfitting version of step "b"
4) Yes it is possible to have one effect significant and another non-sign. and yet the difference is not significant. Think of "significance" as a kind of border: The one effect is slightly above the border--the other is slightly below (but both are not so far away from each other.
All the best,
--Holger
Hi again Holger Steinmetz and sorry for bothering you again.
I did the three steps a, b, c and the difference was not significant. It could be the case as you described in point 4 (that "it is possible to have one effect significant and another non-sign. and yet the difference is not significant").
So as a conclusion now:
In my article, would I say that the test of the moderation effect failed (there is no moderation effect) OR could I report it with caution?
Note: I read your comments in the following thread and they were much appreciated
https://www.researchgate.net/post/How-to-test-moderation-effect-in-AMOS
Hi Tahani,
yes, the statistical test of the moderation was rejected. That is, the observable difference between the groups was not large enough to reject the possibility that they are actually identical in the population (I say that so that you avoid making the mistake of concluding that non-signifikant means "no effect" or "no difference".
Best,
Holge
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Techniques
- Structural Equation Modeling