The obvious problem with using EFA deductively is that you have only your subjective intuition to determine what degree of fit is good enough. In addition, you are going to have the same problem with "tinkering" due to issues like cross-loadings or single-item factors -- let alone determining a cut-off for how much a item has to load on a factor before you assign it to that factor.

The advantage of CFA is that it gives you precise fit indices and statistical tests to resolve issues surrounding your model, rather than rely on subjective judgement. If you are worrying about over-fitting your model due to "tinkering," one option is to split your sample in half and use the second half to "validate" the model you fit in the first half (of course, that requires a reasonably large sample size).

As for construct validity, the primary point of factor analysis is to specify how you are going to measure the construct. So, if you cannot resolve that question, you cannot assess validity.

Hello Nick,

There is indeed a fit index available from EFA, and that is the discrepancy between observed and reproduced variable correlations. A reproduced correlation between any pair of variables may be derived by summing the products of loadings (for the two variables in question) on all factors (assuming orthogonal factors). If you wanted, you could easily take that information and determine the standardized root-mean square residual (SRMR), though when doing this it is customary to include the "1.0" values along the main diagonal.

Otherwise, I concur with David L Morgan 's comment.

Good luck with your work!

Hi Nick,

as you remember we had a similar discussion some while ago in which I argued to test everytime you have some idea about the structure. The post (which took me a time can be found here: https://www.researchgate.net/post/EFA_exploratory_factor_analysis_CFA_confirmatory_factor_analysis_or_both#view=5c9937470f95f1a905604215

With regard to using EFA in a deductive manner.

First, an EFA has an almost saturated model structure, and hence, has not a decent amount of falsifiability. If I remember correct, Popper called this "empirical content" ("empirischer Gehalt"). That is, you have a fully saturated loading structure and the only restrictions concern those of the uncorrelated errors. You can model any set of variables by some factor structure. The only question is how many factors you need to achieve this. This has nothing to do with the most essential questions whether the factors mean anything substantial, see

Hayduk, L. A., & Glaser, D. N. (2000). Jiving the four-step, waltzing around factor analysis, and other serious fun. Structural Equation Modeling, 7(1), 1-35. doi:10.1207/S15328007SEM0701

Hayduk, L. A. (2014). Seeing perfectly fitting factor models that are causally misspecified: Understanding that close-fitting models can be worse. Educational and Psychological Measurement, 1-22. doi:10.1177/0013164414527449

In a CFA context, you have much more control over the specific effects and restrictions, implying larger amount of empirical content and, if the model fits, provides a stronger support for the assumptions centering around the meaning of the variables and there causal effects (note: level of saturation decreases support).

Second, it is a bit odd, to use an exploratory algorithm (that starts with the data per definition) in a confirmatory manner. But this is (agreed) nitpicking :)

Beyond that, I would refrain from describing any kind of model modification as tinkering. In model modification there are two essential aspects:

a) Whether the modification is theory-based *in face of noting specific kinds of failure of the intial model*. This is called abductive reasoning and goes like "given that I observe this pattern of data (=misfit), what would be the cause / process that would result in this pattern? This is the kind of reasoning, doctors and detectives apply. Note the important difference to inductive reasoning.

If you simply follow the modification indices, this is not abduction. It just treats the symptoms

b) Transparency. The problem with model modification is that it implies a fundamental change in walking the confirmatory road to the exploratory road. This is not problematic as long it is transparent. Tinkering means (in the sense of HARKing) that exploratory considerations are "sold" as if they were confirmatory. However, I would not blame individual researchers that do that but the publication culture that punishes exploration and provides incentives for (faked) success stories (fitting models with nice effects and significance).

And last but not least, neither drawing a new sample nor cross-validation is an optimal test of the modified model as it detects only fitting to idiosyncratic features of the data (e.g., relationships only due to sampling error). Model modification, however, will often change the structure of the model in a systematic, non-random aspect which will hold (despite being misspecified) in a new sample.

This can be easily demonstrated with a simulation by creating a model, drawing data, testing a wrong specification, modifying it until it fits, drawing a new sample, and "testing" the modified, wrong model. This model will fit again. In order to challenge an exploratory generated or modified structure, you have to incorporate new, theoretical hurdles by means of new variables with new predictions. The logic has to be "if the (modified) model is correct and the factors have meaning xy, then which effects should these factors cause (or by which variables should they be caused.

Best,

Holger