I made an application for approximately 3000 people for a scale development study. Next, I did exploratory factor analysis (EFA) and it came out one-dimensional with 60% variance. Then, I randomly selected 550 people out of 3000 and performed a confirmatory factor analysis (CFA). However, the variance value of 33% was found and the CFA values were acceptable with 5 modifications. Why can the CFA results of a scale with such good factor structure be so bad?
I do not think you can make the inferences you suggest because it is different sample size in both analyses, if you want to perform a complementary procedure before the validation of a new instrument you can divide into two subsamples of 1500.
Hello Okan,
It wasn't clear whether the 550 cases you used for the CFA came from the set (of 3000) used for the EFA, which convinced you that you had a unifactor structure.
Some germane points here are:
1. Sample to sample differences can and do exist, even if you know you are drawing from the same population.
2. When you run a CFA, you are most likely looking at indices of model-data fit that you did not evaluate with the EFA. It is quite possible that an EFA which "looks good" might have yielded too-high fit indices, had those been evaluated. Therefore, part of what you're seeing (especially since you thought it necessary to introduce a number of model modifications) is perhaps nothing more than more stringent criteria being applied.
3. With respect to the variance accounted for, are you comparing apples to apples? In the EFA, many software packages give variance accounted for among the variables (which you can verify by summing the squared loadings in a one-factor solution). Be sure that the CVA value you're referring to represents the same thing. As well, you didn't indicate what extraction method you used for the EFA. Most CFA software (other than PLS-SEM based routines) would use maximum likelihood estimation. If your EFA used principal axis extraction or worse, principal components extraction, then this could also help account for some of the discrepancy.
4. Finally, did you retain all variables in the EFA? You didn't clarify how many you started with, and how many ended up being retained by you for your EFA structure, or how many ended up being retained in your modified CFA structure.
Good luck with your work.