Hello. In structural equation modeling (SEM) or path analysis, the most important test for fit is the p value. Some argue that in fact this is the ONLY true test of fit. As the p-value is very often significant due to large sample size, etc. (indicative of poor fit) we often use other fit indexes such as RMSEA, CFI, etc. But it sometimes happens that the p for the model IS non-significant which seems to suggest that the model has passed the test for fit. Should I care in such situations for other fit indexes and report them? I wonder what is your opinion about this :)
Pay attention to formulas. Many indices are based on noncentrality indices, which will be 0 if chi-square < degrees of freedom. When that happens, then trivially RMSEA = 0 and CFI = 1 almost certainly.
A weak CFI with a low chi-square may be an indication that power is low and observed correlations are weak.
Joreskog (1969), in the paper that introduced confirmatory factor analysis and the chi-square, noted that models should be expected to fail the chi-square test, if the model was nontrivial and sample size was large enough. That is not a myth.
Dear Lukasz,
I would certainly recommend considering and reporting the other fit statistics. There are situations in which variations between ways of calculating fit can be informative. As a reader/reviewer I can exclude some problems simply by seeing what it is, to you, entirely redundant.
The other reason would be that some values of p are not, on their own, entirely convincing. Imagine p = .06. Again, with my reader's hat on I will now investigate exactly how many parameters were left free, and with what justifications were others fixed - because a p value is not "scale free" or independent of context any more than the other measures.
Can you give me a convincing argument as to how alpha should be set so that the p value would be more useful?
Best of luck,
Global fit indices may still be relevant only if conducting model-comparison approach.
Hope this helps,
Matt
Hi Lukasz,
first, I strongly appreciate your focus on really testing models instead of trying to index the amount of misfit (although you still stick to the myth that misfit may be "due to sample size").
I guess Bruce made some valid points: Evaluation of a model has always to consider the amount of saturation (=df). An almost saturated, fitting model implies still minor causal support for critical portions of the model. The most extreme case is in latent partial mediation models in which the df and thus the test only refers to the measurement model while the latent mediational structure is *completely untested* and hence not supported. As a consequence, you could flip all the arrows and the fit would be the same.
More important, however, than the overall consideration of saturation is which specific aspects are tested (see the mediation example). In a model, for instance, that contains a single X --> Y effect but with an instrument W having an effect on X, the only degree of freedom is critical because it refers to the test of conditional independence of W and Y given X. An alternative model with either an error covariance between e_X and e_Y (reflecting an omitted confounder) or a reverse effect of Y on X would not pass this test.
In this regard, intensifying the incorporation of graph theoretical issue into SEM would strongly improve inspecting models in this regard.
With regard to your main question:
1) Reviewers want to see indexes but I do not know if they add anything to the overall clean fit
2) Yes, p slightly passing the .05-limit are not a good support (but fit indexes, which would also look very nice would again not add anything) especially in almost-saturated models.
3) Check the testability of critical assumptions (for instance, you can do this with the web software dagitty.net that lets you draw the model and which prints the specific testable features of the model. So this recommendation is a slight "yes-but"-version of #2 as sometimes almost saturated models have still the testable implications on the right /critical spot of the model.
Elwert, F. (2013). Graphical causal models. In S. L. Morgan (Ed.), Handbook of causal analysis for social research. (pp. 245-273). Dordrecht Heidelberg New York London: Springer.
Kline, R. B. (2015). The mediation myth. Basic and Applied Social Psychology, 37(4), 202-213. doi:10.1080/01973533.2015.1049349
Thoemmes, F. (2015). Reversing arrows in mediation models does not distinguish plausible models. Basic and Applied Social Psychology, 37(4), 226-234. doi:10.1080/01973533.2015.1049351
Thoemmes, F., Rosseel, Y., & Textor, J. (2018). Local fit evaluation of structural equation models using graphical criteria. Psychological Methods, 23(1), 27-41. doi:10.1037/met0000147
Best,
Holger
Pay attention to formulas. Many indices are based on noncentrality indices, which will be 0 if chi-square < degrees of freedom. When that happens, then trivially RMSEA = 0 and CFI = 1 almost certainly.
A weak CFI with a low chi-square may be an indication that power is low and observed correlations are weak.
Joreskog (1969), in the paper that introduced confirmatory factor analysis and the chi-square, noted that models should be expected to fail the chi-square test, if the model was nontrivial and sample size was large enough. That is not a myth.
Agree with edward , see what other indicies (FORMULA) tell you. What information they carry about fit of the model and at what stage i.e CFA stage ( measuremeNt model) or Structural model stage.
In structural equation modeling (SEM) it is important to pay attention in choice of index for testing the model ( measuremeNt or structural) due to the size of sample.
Hope this helps
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Techniques
- Structural Equation Modeling