Basically, the following statistics assesses the adequacy of the model, indirectly construct validity, as the degree of fit between estimated and observed variance:  Tucker Lewis Index (TLI) (>0.90 acceptable, >0.95 excellent), the Comparative Fit Index (CFI) (>0.90 acceptable, >0.95 excellent), and root mean square error of approximation (RMSEA) (

Thanks for your answer!

I agree with those cutoffs (although the .90 is a little lenient, see Barrett 2007), however these articles fail on those grounds.  CFI is the only place they come close, and CFI is the most liberal and susceptible to data quirks unrelated to theory.  So why are so many of these articles getting through? 

Have you ever seen reviewer and editor comments for articles published in psychology and psychiatry journals? I hate to say it, but reviewers and editors of these journals more often than not don't really understand psychometric methods or how to use them.  See Denny Borsboom's article in Psychometrika for further elaboration: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2779444/

(Other journals in healthcare are probably no better, but they publish fewer of these kinds of studies.)

In this case it sounds like the authors did SEM and did something more than just try to fit an SEM model; they tried to compare models, and the editors and reviewers may have had no idea how to interpret these results. What I find surprising is that the authors reported statistics that show questionable fit. I think it is more typical that authors cherry pick fit indices post-hoc that support their model (and then claim it was an a-priori decision).

Heh... I know what you mean.  Once I had a reviewer criticize a correlation (a zero order correlation) because I had not standardized the scores before I ran the analysis.  I asked for clarification, and he wanted me to subtract the mean and standard deviation from each score, then do a correlation (i.e. z-scores).  I told him that was done automatically as part of the calculation of a correlation, and he dismissed me as ignorant.  So I ran the correlation, and expressed my amazement (tongue in cheek) that the correlation was the same.  He wisely speculated it must be because the data was completely normally distributed.  yeah.... not so much.  He was confusing a standardized score with a standard normal score.  Maybe.  I have no idea, I never asked because he might have come back with an even more bizarre request like I should add a constant to a variable to correct the variance (another reviewer request which is a story for another day...).

Thanks!

Acceptable RMSEA values depend on your data. For 0,1 dichotomies, http://www.rasch.org/rmt/rmt254d.htm suggests values less than 0.1. But for the 1,2,3,4,5 Netflix Challenge rating-scale, the very best RMSEA was 0.9.

John - your first link demonstrated that polyotomous data can actually be fit well.  That the Netflix rating scale struggled to get under .9 (and some did get under .9), probably suggests more that the variables involved were poorly measured and understood than a failure of RMSEA.  If anything, you could argue that RMSEA successfully demonstrated that what was really needed wasn't a better model of combining the variables to achieve a better estimate, but a better measurement of the variables to build a model on.  I don't think a dichotomous measure would give you a better model.

See http://math.arizona.edu/~hzhang/math574m/Read/KddNetflixWorkshop.pdf for more of the Netflix challenge.

Timo - to the extent that it is biased, especially when N

As far as I know, the cut off scores derive from an article by Hu & Bentler (1999), mysteriously there is a ton of papers which state different cut off scores for RMSEA which states that their models fit the data which fails to meet Hu & Bentler  criteria.

I am trying to get something out, and am taking the arguably more rigorous approach of comparing my model's RMSEA with two competing models' RMSEAs (versus the standard approach of comparing to the generic model) and will probably get slammed for it...  

In the article you mention, to be fair the authors do say that the selected model does not fit the data well. There's nothing inherently contradictory in saying that the model fit poorly, but had the best fit of those examined. (Although this would tend to suggest that they should search for a better model!)

For your own research, comparing models based on the RMSEA sounds ok, but you may want to consider using a fit statistic specifically designed for comparing models. E.g., the difference in BIC values between the two models can readily be converted to a Bayes Factor comparing the models by the formula exp(BIC difference/2); see attached article by Wagenmakers, p. 796. The Bayes Factor directly tells you about how strong the weight of the evidence is in favour of one model over the other (see Wagenmakers p. 791 for interpretation guidelines).

To me that sounds preferable to looking at two different RMSEA values and trying to work out if the difference is big enough to merit saying that one model is actually favoured over the other.

PS. The story about the reviewer wanting to see a Z-score transformation before calculating a correlation is hilarious. (Or scary? Not quite sure).

http://www.ejwagenmakers.com/2007/pValueProblems.pdf

That was very helpful Matt, and thanks for taking the time to read the original article as well!