Sounds like the existing scales were measuring different constructs. Depending on the purpose, CFA might be preferred, but you say "I am requires to conduct ..." Whatever is requiring you to do something (which seems a little odd) may require something more specific. Can you say why you are required?

If you have clear theoretical expectations about the factor structure, I would use CFA. You can also compare competing models with CFA (e.g., models with different numbers of factors that correspond to alternative hypotheses).

I would not run an EFA first and then a CFA on the same data set. Running a CFA on the same data (sample) from which the EFA solution was derived does not allow you to adequately “confirm” a factor solution. This would require new or additional data as logically, the same data cannot be used for both exploration (discovery) and confirmation (testing). In other words, a goodness-of-fit test in CFA is essentially meaningless when a factor solution is based on a previous EFA run on the same data as the same information was already used to identify this solution. We cannot “confirm” a factor structure with the same data that was used to “find” the solution.

Replication of the factor solution with new data, possible a sample drawn from a population that differs somewhat from the initial population can be useful. (If you simply draw another random sample from the exact same population, there should not be any meaningful differences in the factor structure besides "trivial" differences caused by random sampling error.)

EFA could be helpful if your CFA model(s) show(s) a bad fit and you have trouble figuring out why the solution(s) do(es) not fit. In other words, an “EFA after CFA” approach can sometimes be useful—whereas the “CFA after EFA” approach is typically not beneficial when conducted on the same set of data.

For further information, you can find extensive discussions of the "CFA vs. EFA" approach on the SEMNET email listserv. You can search the SEMNET archives here: https://listserv.ua.edu/cgi-bin/wa?REPORT=SEMNET&z=4&L=SEMNET&1=SEMNET&X=&Y=

Daniel Wright thanks for your quick reply!

I am "required to" as my master's thesis is just one contribution to a larger research (other master's students are, for instance, running regressions on the data). My task as part of this larger project is to test the validity of the constructs we are researching.

Because I am not in charge of the entire study, I couldn't run a pilot project or gather more data points (the survey has been closed).

I hope this answers your question Daniel, I look forward to your insights!

Dear Christian Geiser thank you for your reply. Unfortunately, when I opened the link you provided, I was not taken to a specific article but rather a search interface.

We have in total 5, 2nd order constructs.

I do have relatively clear theoretical expectations from the scale, BUT it has been adapted on the "1st order construct level". Let me try to illustrate the situation with an example:

One of these 5 higher-level constructs (2nd order) has previously been found to be composed of 6 dimensions (1st order constructs), which were identified via multiple factors during a survey.

Now, we have taken 4 out of the 6 dimensions identified by the original research and added another dimension from another author's study that was shown to relate to the same 2nd-order construct (so in total, we had 4 original and 1 added dimension).

The 2 dimensions from the original study were then incorporated in other 2nd order constructs (that we assume to exist from other research).

What I am trying to showcase is that we have adapted the scale by combining different scales, and some 1st order constructs have also been shifted into different 2nd order constructs.

Would you say that running a CFA is still reasonable with the degree of change in the scale? Or is this too "transformative" and therefore requires an EFA?

Finally, when you mentioned it might be useful to run an EFA after running a CFA, would it, in that case, be reasonable to run the EFA on the same dataset from the CFA?

Thanks again for your support, I look forward to your response!

In construct validation, both exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) serve different purposes. EFA is used in the initial stages of research to explore the underlying factor structure of observed variables, while CFA is employed to test a pre-specified theoretical model. If you have a clear theoretical framework, conducting CFA is appropriate for construct validation.

It is generally recommended to use different samples for EFA and CFA to avoid potential biases. However, in certain circumstances where separate samples are not feasible, conducting EFA and CFA on the same sample can be done cautiously, acknowledging the limitations and potential biases associated with this approach. You could go through the following book for in-depth insights.

Thompson, B. (2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. American Psychological Association. https://doi.org/10.1037/10694-000

Good luck,

Dear Mohialdeen Alotumi thanks for your reply and your insights. I found it very interesting that you mentioned that, under certain conditions, both a CFA and EFA could be conducted on the same data.

I have looked into the book you have recommended but was not able to find anything about conducting the EFA and CFA on the same sample. Could you maybe provide me with the page range and, if possible (if you know any), further books or articles that could also support this?

Thank you very much and have a lovely day!

You could go through the deliberation on p. 131 and p. 162. However, you should read my post above carefully. In particular, you should ponder the part indicating the infeasibility of having different samples. Here is a direct quote from my post above, "in certain circumstances where separate samples are not feasible, conducting EFA and CFA on the same sample can be done cautiously, acknowledging the limitations and potential biases associated with this approach."

To that end, a safer approach is that, if you have a single large sample, you could consider the splitting method of SOLOMON (Lorenzo-Seva, 2022). Here is the full citation.

Lorenzo-Seva, U. (2022). SOLOMON: A method for splitting a sample into equivalent subsamples in factor analysis. Behavior Research Methods, 54(6), 2665–2677. https://doi.org/10.3758/s13428-021-01750-y

Good luck,

The first thing I'd do is a parallel analysis to determine the proper number of factors to extract. Then, I would do an exploratory factor analysis, probably principal axis with oblimin rotation. Then, to determine goodness of fit I'd do a confirmatory factor analysis on the same data, given the limitation of your sample size. From my perspective, the more important consideration is what factors underlie respondents responses rather than what did the theoretician assume they were. If they coincide, congrats to the theoretician(s); if not, re-theorize, based on respondents' responses.

Paul Max Kohn , since I assume you read the other comments before responding, I assume you wrote this because you disagree with Christian Geiser 's answer and the cautions Mohialdeen Alotumi provides. Are you saying they are wrong but just haven't mentioned them by name? I like their responses so I am interested in what parts of their answers you disagree with and why.

It does not make sense to conduct EFA then CFA on the same sample. You cannot confirm something that you derived from the same data source. The argument would be circular/statistically invalid.

Splitting a sample randomly, conducting EFA on the first half of the data, then CFA on the second half also has very limited utility as the two random splits should only differ due to random sampling fluctuations (assuming randomization worked properly). It would not be surprising at all to find the same solution in both parts of the sample. If you find differences, those would have to be due to random sampling error (which is not interesting) or due to the subsamples being too small to provide meaningful results.

CFA requires preconceptions. These need to be based either on theory or empirical data. Theories are not necessarily valid, i.e., they may not correctly identify respondents bases for responding to individual items. Therefore, I'd look to EFA to give me an initial clue as to factor structure, but only after ascertaining the actual number of factors represented by respondents responses. CFA would then provide the luxury of establishing goodness of fit.

I suppose I have less faith in the infallibility of theorists than do Daniel Wright and Christian Geiser.

Paul Max Kohn

Yes, CFA allows you to test theories. And EFA may be useful when you have no theory. But you cannot test a theory with the same data that was used to come up with that theory. CFA goodness-of-fit tests are meaningless/invalid when conducted on the same set of data that was first used for exploration via EFA.

Sometimes, one tests models with mixed psychometric and structural aspects. If the psychometrics don't fit well, the total model probably won't either. In such cases, one may have more confidence in the structural aspects of the model than the measurement aspects. I personally have no problem with using a "cheating model" for the psychometric aspect to provide assurance for the total combined model.