Hi Nick,

this may be my personal bias but whenever I plan to measure some entities/attributes which should play a (causal) role in either a formal theory or in my personal view of a field, I would identify or create indicators that as closely match these entities and test them via CFA. An EFA implies too many ambiguities, with indicators loading on several factors, single indicators falling through the grid and extracted factors having to be post-hoc labeled based on their factor loadings. The loadings are often rather moderate in their strength which makes interpretation even harder.

Furthermore, a EFA-based study often starts with many items which has very likely unfortunate effects on the response behavior by individuals. It is often astonishing which view we as researchers have of our respondents. They are no mindless machines that answer item after items without building their own theories what the study is about. Furthermore, administration of many items will surely have unforeseen effects like serial influences of items on each other (either positive, e.g., copy and paste or at least "influencing") or negative (e.g., discriminatory effects, in which respondents try to identify the very discriminatory information in an item after having responded to a similar one). The multiple-indicator-factor-dogma in contrast sees respondents as machines who forget their response to an item right after they have given their response.

My goal is always, to go for 2-3 crystal clear items. This drives proponents of the classical view mad :) but I have higher chance to end with a clearly fitting model, strong loadings, and indicators whose semantics supports the meaning of this unknown latent variable that I strive for. The classical view implies many items, misfitting models, a semantically heterogeneous set of items with low-to-moderated loadings and huge measurement errors which over-corrects effects of the latent variables.

Doing a CFA (or EFA) in any case requires that you know something about response behavior as well as causal structures. In this tread, I gave some advises for beginners with essential references:

https://www.researchgate.net/post/Dear_scholars_how_to_design_a_Structural_Equation_Modeling_surveys

The reason for knowing about SEM/CFA and causal structures is that your CFA will probably not fit which is no disaster and you have to be able to learn/understand how to improve it and how to proceed after the improvement. The practice of a cross-validation or replication as a proper way is still widespread and provide not much support.

With regard to the response behavior, the following book is enlightening:

Sudman, S., Bradburn, N. M., & Schwarz, N. (1996). Thinking about answers. San Fransciso: Jossey-Bass Publishers.

And I would strongly recommend doing an intensive cognitive probing phase. This alone will teach you much more about response behavior, measurement and validity than any post-hoc statistical analysis:

Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage.

Rossiter, J. R. (2002). A c-oar-se procedure for scale development in marketing. International Journal of Research in Marketing, 19, 305-335.

Lee, N., & Cadogan, J. (2016). Welcome to the desert of the real: reality, realism, measurement, and C-OAR-SE. European Journal of Marketing, 50(11), 1959-1968.

Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50, 537-567.

HTH

Holger

Typically both.

Matt

Typically both, too.

Article Psychometric Evaluation of a Patient Safety Competency Self-...

However, EFA only is also possible.

Article Korean Patient-Perceived Satisfaction Scale of Community-Bas...

However, for your research purpose, "determining the existence of relationships between latent variables," I would recommend you to perform both EFA and CFA.

Unless the scales you use have been confirmed over and over again (e.g. The Big Five, Maslach's Burnout Inventory, General Self-Efficacy Scale) I would always (!) try carrying out an EFA followd by a CFA.

Hi Marcel Grieger

actually I don't understand for what reasons you would recommend that. Once one has theoretical expectation about the meaning of latent variable(s) and the measurement structure that follows from this supposed meaning, why not test this expectation and let all all the theoretical restrictions and testable implications of this expectations? How is a blind "throwing indicators theory-blindly into a statistical machinery" with all the ambiguities as well as capitalization-on-chance dangers should be an advantage?

With ambiguities, I mean that a factor model is just one possible explanation of a covariance structure; thus exploring it does not provide much evidence, see Hayduk (2014). Further, if you have some single indicators, these can be modeled theory-based in the CFA but an EFA is not able to identify them. Instead, a single indicator will load (weakly) on the latent variable which is correlated with the actual latent the single indicator reflects. This once happened to me when I mixed some scales -- one of them was goal commitment (3 items) and one was the simple intention (only 1 item). This so important intention item was meshed with the goal commitment items -- with a weak loading. A exploratory approach would have resulted in deleting the item -- and with it the whole latent intention variable. Hence, I regard it better to clearly think about the latent variables, select indicators and rigidly test the whole model.

As a side note, I would strongly doubt that big 5 measures are "confirmed over and over again" -- to the contrary (see Borsboom's "attack", 2006). Those models never clearly fit and given the PCA-history of the big 5 development, it is not even clear if the big 5 should even be conceived as latent variables but not better as aggregates of facets...

Perceiving typical scales especially from psychology as validated or confirmed is only possible by the silent or explicit disregard to misfit (e.g., by relying on the fit indices). Our field would be different if we really challenged (and improve) these scales. But as I said, just a personal note.

Interested what you perceive instead. Appreciate your posts on RG, btw.

Best,

Holger

Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71(3), 425-440. doi:10.1007/s11336-006-1447-6

Hayduk, L. A. (2014). Seeing perfectly fitting factor models that are causally misspecified: Understanding that close-fitting models can be worse. Educational and Psychological Measurement, 1-22. doi:10.1177/0013164414527449

Hi Holger Steinmetz

I always appreciate your in-depth analyses. Allow me to elaborate on my thoughts: As Williams' questionnaire consists of both "new" and "old" items, I would find it difficult to jump to conclusions as far as item distribution and, ultimately, internal consistency are concerned. You are correct when you state that one can test a questionnaire, which is in development, in a confirmatory way. However, I have two objections to that:

1. It is nigh-on impossible to not have at least some theoretical expectation about what you intend to measure, hence, in my humble view that does not justify a CFA. It also depends largely on how well-founded the theory really is (Henson & Roberts, 2006, S. 407-409).

2. If you started with a CFA, modification indices would tell you how to improve you model so that the data fit the specifications. Strictly speaking, you leave the realm of confirmation and return to explorationat at that point.

Lastly, I would claim that most textbooks on this question stick to the order of EFA before CFA (e.g. Brown, 2006; Bühner, 2011; Döhring & Bortz, 2016; Moosbrugger & Kelava, 2012).

As for the Big Five, I stand corrected. I was tought they are somewhat of a golden standard. Thank you for setting that straight.

Kind regards

Marcel

_______________________

Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York NY u.a.: Guilford Press.

Bühner, M. (2011). Einführung in die Test- und Fragebogenkonstruktion (3. Aufl.). München: Pearson.

Döring, N. & Bortz, J. (Hg.). (2016). Forschungsmethoden und Evaluation in den Sozial- und Humanwissenschaften (5. Aufl.). Berlin: Springer.

Moosbrugger, H. & Kelava, A. (Hg.). (2012). Testtheorie und Fragebogenkonstruktion (2. Aufl.). Springer-Lehrbuch. Berlin: Springer.

Great comments guys! thanks

Dear Marcel Grieger ,

I would try to elaborate more. Perhaps you see some merit that going "blindly-exploratory" is neither necessary or can even be harmful. If not you, perhaps this could be helpful for others.

I am sorry that this is so long. BUT: I provided two short paragraphs (each beginning with the phrase "Bottom line") that you may read and skip the rest :) To make it more obvious, I highlighted these paragraphs with dashes.

1) As I have no access to the books you mentioned (only Brown), one overall comment. All of these, of course well-respected, scholars represent classic psychological convictions, mantras, and beliefs--namely to either explicitly regard construct as "operationalizations" of data (which is a consequence of our positivist history) or--if a latent perspective is adopted--to blindly adopt the common factor reflex. Because beliefs are deep and lead quasi-automatically to behavior, both are often mixed and even factor modeling routines rather follow a data reduction procedure than the goal to detect true underlying and existing entities. Psychologists often don't even consider what the "construct" really means in terms of an ontological stance (see Lee & Cadogan, 2016, for an excellent criticism).

In addition--again as a consequence of the positivist history--psychologists have a huge focus on data and statistics and not causality. This is in part due to our respected Sir Karl Pearson who was not only a brilliant statistician but also a positivist and enemy of the causality concept. His invention of the correlation coefficient was not only a result to offer something "statistical" but also to create a sword to smash all causal thinking. In the following decades, he and his disciples went after everyone who published something about causality. Judea Pearl (2018) has devoted a whole chapter in this "book of why" to Pearson and his politics.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

==> Bottom line: These textbook authors are surely brilliant in what they do (I know Agustin Kelava personally and Markus Bühner is a cooperation partner of our department) but their writing reflects a tradition that keeps constructs unclear, avoids causal thinking and thus, assumptions and restrictions (this will become important below) and uses factor analysis as a theory-blind routine while disregarding that other causal structures are possible and likely. The EFA model is blind to such other structures. Therefore, no wonder that these textbooks recommend traditionally exploratory and theory-blind data crunching algorithms.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

2) Let's now look what Brown writes: I was able to locate a short paragraph on p. 106 where he argues that a CFA should be "well grounded by prior empirical evidence and theory" and continues "This is because CFA entails more constraints than other approaches such as EFA (e.g., prespecification of indicator-factor relationships [he means and avoids the term "causal structure"], fixing cross-loadings and error covariances to zero"), thus requiring the researcher to have a firm substantive and empirical basis to guide the model specification".

Perhaps I am missing something but there is no single argument on why this should be the case. The phrase "because CFA entails more constraints" is no argument to go adopt a completely theory-blind approach (EFA) when the basis for these constraints is not "well grounded by prior empirical evidence theory"). I would even go so far that even a weak theoretical basis provides an argument to test this weak basis and learn and improve.

Imagine, you wake up and you had a dream containing some idea about a latent variable, its meaning and some possible way (i.e., indicators) how to measure it. I argue that going to CFA (instead to EFA) even in this extreme case would be advantageous because your CFA-model has at least constraints which can be tested (and this is the point). Your model can be supported/corroborated and this support is stronger (compared to EFA) with regard to your a) dream-based interpretation of the latent and b) the proposed causal structure. You may have a weak theory but a stronger evidence than EFA and can progress from this point (e.g., by imaging further implications of this latent if-it-was-really-the-entity-you-think-it-is, for instance presumed causes, instrumental variables or consequences (e.g., building a nomological net) or creating an experiment to change it. At every step in this process (even within the first step--the first dream-based CFA), you can fail, diagnose failure and learn.

With a EFA, in contrast, you have the worst combination of a weak (e.g., dream-based) foundation AND a weak empirical procedure. The reason for this is the constraints. Lets explain this with a different kind of model--a mediation model.

Imagine, your dream implies a A-B-C effect. Approach #1 would resemble an EFA and use a regression model (which implies a partial mediation and, thus, no restrictions/constraints on the direct effect). You get nice effects but a very low support of your dream-based structure because there are over 40 permutations of this model which all implies the same data. Weak theory AND some nice results with no support for anything. Because this model cannot crash, confirmation bias will play its game and you regard this as a support for your dream.

Approach #2 (~CFA) would test the core of the dream (i.e., A-->B-->C) in a SEM with a fixed direct effect (the weakly/dream-based constraint). As a consequence, the model could crash, leading to some necessary modifications of your dream or even giving it up. This is learning and innovation. Otherwise, the model could fit. This of course does not prove the correctness of your dream but provides a starting point for challenges and improvement which is on a higher level than approach #1 (because you already have some amount of support).

Analogous to this example, EFA is a weak support right because of its low number of constraints. It is an approach that forces a certain model-structure (the common factor model) onto the data without having any flexibility in acknowledging different structures, e.g., serial effects from items on each other (Bollen & Medrano, 1998), single-items latent variables (Hayduk & Littvay, 2012) or network structures (Borsboom & Cramer, 2013). In this regard, other exploratory approaches, such as the TETRAD model (Malinsky & Danks, 2018) are more promising. By routinely applying EFA to data with unknown underlying structure, you start thinking predominantly in factor structures (this is the case in Psych) and if a model does not work, you will not be able to "think out of the box" to come up with improvements (this is why modification indices should not be consulted -- they solve the symptoms (misfit)--e.g., by relaxing constraints--while keeping the disease persisting (the potentially seriously misspecified factor structure).

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

==> Bottom line: The existence of constraints does not logically imply that the basis of these constraints must be strong. It can be anything--a theory or "just an idea". The cost of relying on weak foundation--of course--is a high chance to fail but that can be used as starting point for developing more solid ideas. An EFA always give you something. It is empirically weak because (1) the reduced number of constraints and (2) the criteria to clearly acknowledging a failure (e.g. dubious weak-loading statements such as "below .5" or eigenvalue criteria). I know that the ML EFA provides a chisquare test--that is at least something. The vast majority of EFA applications, in my point of view, however, tap in the confirmation bias trap.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Btw. what does Brown mean with "well grounded in empirical evidence and theory"? A former EFA? Well...no. Repetitions of pure factor models do not lend strong causal support, about which I provided some arguments in this thread

https://www.researchgate.net/post/Can_I_use_EFA_before_SEM_analysis

I also once posted on RG a simulation, in which I had created data not following any factor model. The EFA, however, resulted in a nice one-factor solution and even a CFA could not identify this nonsense. Enlarging this model however, by incorporating further variables (=constraints!) blew it up. I have the R code still and can provide it if you request it (or any other).

The aforementioned thread and the simulation also clarifies how a progression from the weakly-based foundation (i.e., the dream) to a more strongly-based foundation (beyond repetition) could work. By going such a road, dreams can become solid evidence or even theory.

If you disagree, perhaps their may be at least some things you could apply beyond following tradition.

With best regards

Holger

Bollen, K. A., & Medrano, J. D. (1998). Who are the spaniards. Social Forces, 77(2), 587-621.

Borsboom, D., & Cramer, A. O. J. (2013). Network analysis: An integrative approach to the structure of psychopathology. Annual Review of Clinical Psychology, 9(1), 91-121. doi:10.1146/annurev-clinpsy-050212-185608

Hayduk, L. A., & Littvay, L. (2012). Should researchers use single indicators, best indicators, or multiple indicators. BMC Medical Research Methodology, 12(159), 1-17. doi:10.1186/1471-2288-12-159

Malinsky, D., & Danks, D. (2018). Causal discovery algorithms: A practical guide. Philosophy Compass, 13(1), 1-11. doi:10.1111/phc3.12470

Lee, N., & Cadogan, J. (2016). Welcome to the desert of the real: reality, realism, measurement, and C-OAR-SE. European Journal of Marketing, 50(11), 1959-1968.

Pearl, J., & MacKenzie, D. (2018). The book of why. New York: Basic books.

Find attached file for your kind understanding on the subject;

Article An Easy Approach to Exploratory Factor Analysis: Marketing P...

Holger Steinmetz thank you very much for your guidance. i conducted the EFA and CFA at the same time. Although CFA is providing good AVE and CR with good fit, EFA is showing cross loading and i am very confused because i have tried my best to get rid of that cross loading but i could not. can you refer any paper that i can refer for the use of CFA (only) because i have to defend my thesis.

Hello Muhammad,

I would always go for the CFA as this is the more restrictive approach. If the EFA shows *substantial* cross-loadings then I would assume that your CFA shows sign. chisquare which partially may stem from the fixed-to-zero cross-loadings.

"Getting rid" of a cross-loadings (are a parameter) sounds a bit strange as it is (probably) a part of the reality you have to deal with. By the ways: If the model is correctly specified, an estimated cross-loading does not hurt and is not a sign of something bad. A CFA does not have to have a simple structure. Your model can look like a mess unless it is correct (and this is the difficult task).

Best,

Holger