Thanks, Patrick, for commenting.

It is briefly explained in Section 15 of the article I published recently on Researchgate. Have a look at it, if you like. It is fully explained in another article which I submitted more than three months ago. I still have no decision of the editor.

Besides, from the literature I concluded that the indices are quite unreliable. So why continue with CFA if it provides such a poor feedback on one's predictions of item clustering?

Peter,

I will add a few thoughts, I suggest that SEM, or the specific case of CFA is a particularly powerful tool and that the limits you identify are those of the user more than the technique. For example you can correlate the errors of two or more items which with careful thought are reasonable, this is not possible in EFA.

Philosophically, I suggest that EFA is only acceptable with new instruments and these should be without a strong theoretical base, and I am not certain such areas exist any more. The logic of CFA is that you are testing a known structure, and that should include the cross loadings etc. these can be modeled. If you only have a theoretical structure, as opposed to a previous empirical structure, test that structure and use MI's to add cross loading etc. To add a further thought, SEM tools are now so powerful that all the questionnaires constructed with 2, 3, 4, 5, 6, etc point response scales have been tested assuming they could be treated as continuous. They all need to be re-examined using ordinal methods and logically with CFA methodology, resorting to EFA when CFA cannot find an acceptable solution.

In regard to GOF measures, I followed SEMNET for several years and gave up after years of vitriol around this and other issues. The conclusions I have drawn and rely on is that Chi Sq is the only truly acceptable measure of fit, unfortunately it is sample size dependent, but most studies particularly CFA studies are much smaller than than 1000, The only other measure I rely on at all is RMSEA, the other measures are useful in specific contexts when they may specifically apply, the problem is that they are reported generally, and I have had reviewers insist I provide them even though I have acceptable chi sq.

I agree with the previous comments that you can have cross loadings in CFA. What troubled me most about SEM methods is the vast number of measures of fit and the sensitivity to the assumptions when using asymptotic methods, especially with small sample sizes. The latter I have personally resolved by adopting a Bayesian approach as described in [1]. Obtaining the cross loadings was as straight forward as correlating the observed item scores with the latent variable.

[1] Song, X.-Y., & Lee, S.-Y. (2012). A tutorial on the Bayesian approach for analyzing structural equation models. Journal of Mathematical Psychology, 56(3), 135–148. doi:10.1016/j.jmp.2012.02.001

Martin, I will check the article you referenced, I must admit I am not strong on the Bayesian approaches

I have rejected papers in the past which have had significant chi sq but apparently acceptable fit statistics and suggest that papers must justify why not chi sq even with large samples. As I previously indicated I withdrew from SEMNET and do not know where thinking is this regard is on that forum anymore

one additional bit of thinking about the CFA approach in general is that if applied properly it can reduce the number of data specific variations reasonably,

I published a factor structure for and instrument supposedly uni-dimensional, which had 4 first order and two second order factors with one correlated error terms; two independent samples, (n = 1200, & n=10,000), EFA in the first and CFA in the second non-significant chi sq, someone replicated my results and tried to claim the correlated error term was the wrong one, unfortunately for them the straight CFA based on the empirical data was non-significant and the result should simply have been accepted as being consistent with previous empirical research, I believe this was also the editors view, so even though the CFA could be improved, in their data by changing some minor relationships, this was not considered acceptable practice and thus avoiding a pointless argument about what is the best structure!

Robert,

Thanks for the comment. I have the impression (from your first comment) that you think I am advocating EFA as an alternative for CFA. That, however, is not what I attempt to communicate. My article is meant to incite a critical attitude towards the GOF-indices, including chi-square, as a proof of construct validity of a measuring instrument. CFA (SEM in general) may be a great technique when used properly, but its use would be greater, so it seems to me, if the problems with chi-square would be solved and if a reliable measure of the degree of model fit would be developed.

Regarding the reliability of chi-sqyuare: Paradoxical results in case of large samples are not the sole problem with chi-square. The most serious problem is that it easily produces false-positives when the unique variance of the observed variables is high because of a low reliability of the measuring instrument, and/or because of unmodeled specific factors. SRMR does not escape this fate either. I suspect this is the case because generally low within-cluster item correlations (which are a "consequence" of a high unique variance) will produce tiny residuals, even if the tested model is far from perfect.

Not being a multivariate expert, I am not at all sure whether this is really the clue to the riddle, but if it is, why not invent a correction for the average of the (absolute) inter-correlations values, in order to cope with this problem?

Hi Peter,

I am not sure I would be as hard on chi sq as you seem to be, again I consider that it is a function of the researcher doing their job properly.

I am assuming that if the unique variance is included in the modelling, that this should reduce the impact on chi sq of having less than desirable latent factors. I would appreciate your thoughts on this, as it is an assumption I have made.

Consequently, I now always model the unique variance in my analyses, further, if the reliability is too low I reanalyze the factor structure, or start thinking about using single items of interest because without reasonable inter item correlations there is no latent structure to use.

I have recently been using R to compute alpha for ordinal variables, and I am finding the alpha is for 5/6 point or less scales often significantly higher, suggesting a much better reliability than initially thought eg an alpha of 0.63 assuming linear vs 0.86 treating the items as ordinal, a paper that gives a step by step guide on how to do this is

Gaderman, A. M., M. Guhn and B. D. Zumbo (2012). "Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical and practical guide." Practical Assessment, Research & Evaluation 17: 1-13

in regard to incorporating unique variance the best way is to do a full SEM, however in the field I work in the concepts are not clearly distinguished and the items cross load all the time (eg depression and PTSD), hence I model the scale scores and include the reliability information in the analysis, a reference I use for this is

MacKinnon, D. P. (2008). Introduction to Statistical Mediation Analysis. New York and London, Lawrence Erlbaum Associates

This is a good resource on the concepts of mediation etc but a small section talks about the approach he takes. There are better statistical references but I find this one useful

If I understand your point about un-modeled specific factors I believe this is a problem for all research endeavors, and is not unique to the world of SEM. The unknown influence impacts the general linear model, correlational research, and the gold standard of randomized clinical trials it is even a problem in qualitative research. That is, the interviewer doesn't provide the appropriate prompt so that the participants can provide the necessary insight, hence experienced qualitative researchers get better results than the in-experienced, they miss fewer of the important questions, either in interview and/or the analytical stage.

Similarly leave a key item out of a factor analysis and the apparent meaning of the concept can shift substantially

Hi Robert, I am not a SEM-user, so not the right person to judge the way of using it. Nevertheless, a modest attempt:

As an outsider I would say that it is excellent solution to include the unique variance in the model, if that is possible. Since SEM translates every predicted parameter into an implied correlation matrix, it seems to me that a researcher should predict as many details as possible of the factor structure to be tested: number of factors, variable assignment to factors, factor loadings of the variables, factor correlations, cross-loadings, even if he is uncertain about these relations. Because, if he does not, there will be many free parameters, which will reduce the average height of the residuals (if I understand the procedure rightly), suggesting better fit than warranted. I doubt whether the penalties for free parameters in chi-square, RMSEA, etc., are sufficient to guard against the suggestion of good fit in these cases.

It seems to me that it should not be difficult for brilliant guys like Steiger, Bentler, MacCallum, Browne, Satorra, etc. to devise measures that correct for the average height of the within-cluster variable correlations, that penalize directly proportionate to the number of predicted versus unpredicted variable correlations in the implied correlation matrix, and that base the model fit solely on the predicted variable correlations.

With such measures, researchers would be no longer rewarded for not doing the testing job properly.