Hello Mollie,

It's interesting how often data forces us to reconceptualize a model.

Yes, many researchers will split a data set and hammer away on the first half to find a solution that appears to work well, then conduct CFA of that final model using the second half of the data set. As you have done.

Since your final model is quite different in form (second order factor structure) and the first order factors are notably different from the originally proposed model (and more task focused for females than for males), you'll have some explaining to do as to how each model (original and final) may be said to have a comparably strong rationale for their form. As well, why a second order structure wasn't originally considered as either an option or a competing model should be addressed.

Good luck with your work.

Hi,

The procedure is perfectly legitimate and is called cross-validation.

In fact, one strategy to address problems connected with multiple model specifications is the cross-validation analysis.

Indeed, with multiple specifications of the model, there is the risk that modifications may be driven exclusively by characteristics of the sample on which you are working on (MacCallum et al. 1992).

As long as the modifications are theoretically driven I do not see any problem.

But you need to take into account that you are leaving in this way a confirmatory approach (testing a specific hypothesis) to employ an exploratory one.

Best,

Gianmaria

Hi Mollie,

the cross-validation ritual was intensively discussed in this thread:

https://www.researchgate.net/post/Can_we_do_exploratory_and_confirmatory_factor_analysis_in_the_same_data_set2

As you will see I strongly argued against it. If you randomly split a sample in two halfs, you get to identical halves. Hence, anything you find in the first part, will be replicated in the second half without any validation evidence.

In addition, I provided a small simulation that illustrates this. In this simulation, I fit a wrong CFA model to the data, drew another sample and the the wrong-albeit-fitting model fitted again.

The issue in model validation is not capitilization on chance, but fitting a wrong causal structure to the stable and thus, replicable, portion of the data.

Better would be to incorporate other variables that should connect with your variables already in the model. These add new requirements for the model to fulfill and impose challenges to the model.

Best,

Holger

I would like to start by saying that I like the answers provided above. First, a confirmatory factor analysis (CFA) is to confirm. To confirm a model, which one refers to as the theoretical model, or the proposed model. (This, in my view, should be presented with a scientific justification). So, I use CFA when there is a theoretical model to test. Second, I would not split the sample for this. I would use my complete sample to confirm or refute the theoretical model and, if the theoretical model shows to be a poor fit to the data, I would inform this and say where the problem is. This will inform and guide the researchers in the particular field of study. I do not see why not to engage in an exploratory factor analysis at that point, to go on to propose another model, stating that such model arose from the exploratory part of my work. This model would be there, as a proposed or theoretical model, to be tested using independent datasets in the future.

According to Hinkin (1995), it is not advisable to conduct EFA and CFA in the same sample by splitting it into two halves because it may lead to biased results. You can follow these references for your work:

Hinkin, T. R. (1995). A review of scale development practices in the study of

organizations. Journal of Management, 21(5), 967-988.

Hinkin, T. R., Tracey, J. B., & Enz, C. A. (1997). Scale construction: Developing reliable and valid measurement instruments [Electronic version]. Retrieved [insert date] from Cornell University, School of Hotel Administration site: http://scholarship.sha.cornell.edu/ articles/613.

It is ok to split your sample with that purpose but always double-check you have enough sample size in both, the training and the confirmation sample.